MIMO SAN NETWORK SETUP

[clayton@mcc911.org]

Hello guys i need some help
i cant wrap my head around this.
i am seting up starwind in a 2 node HA with 3 ESXI boxes
all five boxes have 4 Gb ports for san traffic

i have defined 4 subnets
192.168.100.0 ,101.0 ,102.0 and 103.0

each of the five computers will have 1 port in each subnet

my question is,, i have 2 managed switches in a stack for the SAN
how should i do the cables and vlans
right now there is 1 port in a vlan for switch managment and all the the others in a second vlan
there will only be san traffic on these 2 switches,

should i make 4 vlans for each of the for subnets ? is that really needed?
how would you plug the cables up ? put all off sub net one on switch one and all of sub net 2 one switch two and all of sub net 3 on switch one and all of sub net 4 on switch 2


or all four ports of node one on switch one and all four ports of node 2 on switch 2 and the spread the four ports of the esxi boxes accross both??
and id you did it that way if you used vlans you would have to set vlans accross both swiches....
there just seems to be 6 ways to sunday here please advise
thank you
Clayton

[Anatoly (staff)]

Hello Clayton

I`ve attached recommended scheme for your system

Please note that interfaces between StarWind Servers are used for synchronazation channel and used to be bounded and connected directly. Also switches are connected in order to increase fault-tolerance of your system.

[clayton@mcc911.org]

Anatoly

thank you for the help.
the 4 ports I listed are just for San Traffic
each server has 2 four port cards plus 4 ports on board but your diagram will help

4 port to each switch
4 port to each other for the replication link
and 4 onbord ports for managment ,lan connetions, ip-kvm ect.

but what about the Vlans on the switches do i need them

[anton (staff)]

You don't really need VLAN thing as whole iSCSI network should be run outside DMZ. I would go for dedicated SAN fabric / hardware however. Mostly b/c of performance.

[clayton@mcc911.org]

ok thanks
this is want i set up today before i saw your post
so far i have just loaded server on both and got all the swiches set up
so i dont know if it will work but unless i am missing somethign it should work well.

[Anatoly (staff)]

I think that me and Anton gave you pretty detailed recommendation, but your plan will work too. Please note that you have no need to use four vlans to connect to SANs.
To learn more please read SAN configuration guide from VMware pubs. Here is for ESX 4.0 and ESX 4.1

[mkaishar]

I think 4 VLANs for the SAN environment is overkill and it will be a pain to manage in the future

Our environment is split into a LAN VLAN and a SAN VLAN using the same switches. We don't have Starwind HA at this time, but the settings below should work.

I would change your Sync network to something completely different from your SAN network, maybe it's just me, but using 192.168.xxx for everything just gives me more opportunities for mistakes.

Use a 10.xx or a 172.xx

I like using 10.xx network because they are much easier to type

I know there are recommendations to split physical switches for performance, but that is a waste of ports & money.

Here is what I use through Putty and SSH into on each ESXi server to create the config, change IPs as needed.
I enable everything through console, I guess you can run it through PowerCLI, I just haven't had the time to learn it all yet.

Code: Select all

esxcfg-vswitch -a vSwitch1
esxcfg-vswitch -m 9000 vSwitch1

esxcfg-vswitch -A iSCSI0 vSwitch1
esxcfg-vmknic -a -i 10.0.0.20 -n 255.255.255.0 -m 9000 iSCSI0
esxcfg-vswitch -A iSCSI1 vSwitch1
esxcfg-vmknic -a -i 10.0.1.20 -n 255.255.255.0 -m 9000 iSCSI1
esxcfg-vswitch -A iSCSI2 vSwitch1
esxcfg-vmknic -a -i 10.0.2.20 -n 255.255.255.0 -m 9000 iSCSI2
esxcfg-vswitch -A iSCSI3 vSwitch1
esxcfg-vmknic -a -i 10.0.3.20 -n 255.255.255.0 -m 9000 iSCSI3
esxcfg-vswitch -A iSCSI4 vSwitch1
esxcfg-vmknic -a -i 10.0.0.120 -n 255.255.255.0 -m 9000 iSCSI4
esxcfg-vswitch -A iSCSI5 vSwitch1
esxcfg-vmknic -a -i 10.0.1.120 -n 255.255.255.0 -m 9000 iSCSI5
esxcfg-vswitch -A iSCSI6 vSwitch1
esxcfg-vmknic -a -i 10.0.2.120 -n 255.255.255.0 -m 9000 iSCSI6
esxcfg-vswitch -A iSCSI7 vSwitch1
esxcfg-vmknic -a -i 10.0.3.120 -n 255.255.255.0 -m 9000 iSCSI7

esxcfg-vswitch -L vmnic4 vSwitch1
esxcfg-vswitch -L vmnic5 vSwitch1
esxcfg-vswitch -L vmnic6 vSwitch1
esxcfg-vswitch -L vmnic7 vSwitch1

esxcfg-vswitch -p iSCSI0 -N vmnic5 vSwitch1
esxcfg-vswitch -p iSCSI1 -N vmnic5 vSwitch1
esxcfg-vswitch -p iSCSI0 -N vmnic6 vSwitch1
esxcfg-vswitch -p iSCSI1 -N vmnic6 vSwitch1
esxcfg-vswitch -p iSCSI0 -N vmnic7 vSwitch1
esxcfg-vswitch -p iSCSI1 -N vmnic7 vSwitch1

esxcfg-vswitch -p iSCSI2 -N vmnic4 vSwitch1
esxcfg-vswitch -p iSCSI3 -N vmnic4 vSwitch1
esxcfg-vswitch -p iSCSI2 -N vmnic6 vSwitch1
esxcfg-vswitch -p iSCSI3 -N vmnic6 vSwitch1
esxcfg-vswitch -p iSCSI2 -N vmnic7 vSwitch1
esxcfg-vswitch -p iSCSI3 -N vmnic7 vSwitch1

esxcfg-vswitch -p iSCSI4 -N vmnic4 vSwitch1
esxcfg-vswitch -p iSCSI5 -N vmnic4 vSwitch1
esxcfg-vswitch -p iSCSI4 -N vmnic5 vSwitch1
esxcfg-vswitch -p iSCSI5 -N vmnic5 vSwitch1
esxcfg-vswitch -p iSCSI4 -N vmnic7 vSwitch1
esxcfg-vswitch -p iSCSI5 -N vmnic7 vSwitch1

esxcfg-vswitch -p iSCSI6 -N vmnic4 vSwitch1
esxcfg-vswitch -p iSCSI7 -N vmnic4 vSwitch1
esxcfg-vswitch -p iSCSI6 -N vmnic5 vSwitch1
esxcfg-vswitch -p iSCSI7 -N vmnic5 vSwitch1
esxcfg-vswitch -p iSCSI6 -N vmnic6 vSwitch1
esxcfg-vswitch -p iSCSI7 -N vmnic6 vSwitch1

esxcli swiscsi nic add -n vmk1 -d vmhba38
esxcli swiscsi nic add -n vmk2 -d vmhba38
esxcli swiscsi nic add -n vmk3 -d vmhba38
esxcli swiscsi nic add -n vmk4 -d vmhba38
esxcli swiscsi nic add -n vmk5 -d vmhba38
esxcli swiscsi nic add -n vmk6 -d vmhba38
esxcli swiscsi nic add -n vmk7 -d vmhba38
esxcli swiscsi nic add -n vmk8 -d vmhba38

I put this script in /etc/rc.local for MPIO

Code: Select all

# Fixing iSCSI MPIO througput limited to 1G.

esxcli nmp device list | grep ^eui |
while read device ; do
        esxcli nmp device setpolicy --psp VMW_PSP_RR --device ${device}
        esxcli nmp roundrobin setconfig --type "iops" --iops 3 --device ${device}
done

If your SAN NIC's are Intel's and support VMDQ, you should enable:

http://kb.vmware.com/selfservice/micros ... 0160190099

http://kb.vmware.com/selfservice/micros ... Id=1004593

Code: Select all

esxcfg-module -e igb
esxcfg-module -s "IntMode=2,2,2,2 VMDQ=4,4,4,4" igb

I also use these recommended settings from Starwind:

http://www.starwindsoftware.com/forums/ ... t2296.html

http://www.starwindsoftware.com/forums/ ... t2293.html

[anton (staff)]

1) VLANs are definite overkill. Consensus!

2) "Use a 10.xx or a 172.xx" (c) ... Did you actually mean 192.xx?

[mkaishar]

I meant for sync traffic use 10.xx or 172.16.xx which would be completely different from the OP's 192.168.xxx SAN network

[Anatoly (staff)]

Oh, then yes, sure, then we`ve got consensus at this point too

[clayton@mcc911.org]

thanks guys good tips!!
i use 192.x because its jsut easy for me. my lan is a 172. and all my wan links and hops are 10. so it just easy for me to know all my san suff is 192.168. 1~5.x

[Anatoly (staff)]

You`re very welcome. Please keep us updated.