While these are all great benefits, what about the costs, complexity, and risks? Also, how realistic are these benefits for a VMware environment? Is Boot from SAN really the best way to go for a vSphere host? In this post, we’ll examine these concerns for a vSphere host and see just what makes sense.

Note: I’m specifically leaving out VMware’s Auto Deploy technology in this discussion, as I feel it is worth its own post.

No Free Lunch

Booting from SAN isn’t a check box that you click. As with most anything in the IT world, it requires a decent amount of architecture and design work. Here are some things to consider:

• Is your SAN currently configured for the zoning necessary to provide each host access to the boot LUN?
• Have you come up with a design on how you want boot LUNs to be accessed over your SAN? You want to make sure paths will not be saturated.
• Do you have storage that has the capacity and throughput to be shared for boot LUNs, or available disks to create an isolated storage pool for your boot LUNs?
• Have you designed your storage array to properly present the boot LUNs to each host? Typically, LUN 0 is presented as the boot LUN, so it’s wise not to present any other LUN as 0 unless you are going to pick a different LUN number (assuming your booting HBA supports this).

This isn’t the end-all-be-all of considerations, but it’s a start. Assuming you’re up to the challenge of properly presenting storage to each host, you’re in the clear. The costs (time, effort, equipment) and complexity (additional layer of configuration and management) of the above are all introduced with boot from SAN.

The major risk with booting from SAN is that your entire set of hypervisors into the hands of the array. For you, this can be a good thing (you’re a SAN ninja), or a bad thing (your SAN is a scary place that you don’t like to visit). If the array suffers storage contention on your boot LUNs, or someone goofs up the fabric, you can potentially lose your entire vSphere environment. As a counterpoint, I’m going to assume that your virtual machines are also running on the array, so having your hypervisors crash may be of no consequence if you also have all your VMs crash as well.

The typical SAN ninja. No, wait, this is the wrong photo.

Level of Effort

One of the most tangible results of booting from SAN is the ability to quickly recover from catastrophic hardware failure. You can slap a new host into the environment and it will boot up and appear as the old vSphere host. But then again, we’re talking about a vSphere host – how unique is that when you really look at it? The level of effort to re-install ESXi onto a server, and then give it an IP, is relatively low. With the use of host profiles, distributed switches, and/or kickstart scripts, a lot of the remaining configuration work is already done.

Ask yourself a few questions:

• Is it really worth going through all that zoning, storage provisioning, fabric design, and complexity for every host you use?
• How much time savings are you going to realize when compared to installing and configuring ESXi when a host has a failure?
• In a blade environment, is stateless computing something that you require?

It’s a lot of effort … is it worth it for you?

For other “bare metal” hosts, where you are installing Windows or Linux, there is a greater return on your investment. The installation and configuration requires a lot of time and effort. It’s harder to argue the same for ESXi.

Thoughts

Boot from SAN is something that should be evaluated from all angles, and is not by default a huge benefit to the typical vSphere environment. It adds a decent bit of complexity and time investment while having narrowly defined returns. While there isn’t anything technically wrong with using boot from SAN in a vSphere environment, it’s important to understand that vSphere is already designed to be lightweight and easily replaced. Additionally, modern server technology can easily have a pair of SD cards or SSD drives that are mirrored to protect the data, reducing a reliance on spinning disk.

I’m interested to hear what you think.

Session 1223 is entitled “Design Considerations for Small and Medium Business (SMB)” and features a panel discussion with some really smart guys, many of which have contributed to the VMware community in a variety of ways. It’s a mixed group of engineers with a variety of accolates (VCDX & VCAPs, vExperts), many of which have presented on vBrownBags in the past!

• Harley Stagner, Solutions Engineer, TBL Networks, Inc
• Sean Crookston, Solutions Engineer, TBL Networks, Inc
• Bill Hill, Lead Engineer, OIA Global Logistics
• Chris Wahl, Data Center Engineer, Ahead, LLC

I’ll also give a shout out to Sean Crookston’s VCAP4-DCA study guide (which was a big help to my exam).

There are a lot of other great sessions on SMB, so I hope that what we bring to the table – an open panel discussion on design from folks who are in the trenches – will be worth the time for your vote.

In this video, I created a vSwitch portgroup with the name of “100″. Also, there is a DVPort in use that is also “100″. Watch what happens when I try to delete the vSwitch portgroup “100″ when a vmkernel is in use on the DVPort with the same name.

Can’t argue with Yoda

The Xangati Dashboard has quickly become my go-to tool for figuring out any issues in the lab, or purposely designing issues to better understand how they impact other systems. As a matter of full disclosure: I was provided a handful of socket licenses under NFR, which basically provide the same features as their trial product, but without an expiration date. Point being – anything you read about in my posts can essentially be explored using their free tool (for a single host) or a trial (for a cluster).

Follow The User

So, why am I writing about Xangati today? I was lucky enough to get some hands on lab time with their beta release code, which includes a new feature for profiling VDI end-users across desktop sessions (or even across unique desktops, as with floating pools) without needing an agent. I’d imagine that anyone else who works with virtual desktops can attest to the difficulty of figuring out a performance problem in a non-persistent desktop pool. It usually requires a lot of digging through logs to figure out their session history, and then correlating those sessions (virtual machines) to performance historical data. No fun!

The primary dashboard page showcasing desktops and desktop users

The new release of the Xangati VDI Dashboard uses this new follow-the-user style tracking (that’s my term for it). As you may already be aware, vSphere tracks virtual machines, while View tracks users sesions – there really isn’t a good bridge between the two from a monitoring perspective. The Xangati appliance will poll the View Connection servers for log in data, and make a relationship mapping of user to desktop. Couple that with their knowledge of the underlying environment (from vCenter and NetFlow data) and you’ll get a solid picture of what’s going on.

It also gathers all the running process via WMI queries and PCoIP session statistics from the VMware View Agent on the desktop. This gives you a complete, end-to-end look at the user’s experience across disparate desktops along with the relevant performance data. Eureka!

A replay of PCoIP session details for a desktop user named viewclient2 (highlighted in red)

Another nice thing about this? When a user calls in for help, you don’t need to have them provide an asset tag, desktop pool name, or anything else besides their account name. Xangati already has a list of all the users and what desktop they are on. This saves you some time from digging around the View Administrator to find the user – you can get directly to troubleshooting, and then go to the View Administrator later if the resolution involves some administrative task (refreshing a desktop, for example).

All desktop users are quickly presented and can be clicked on for historical details. Or, you can use the search box.

Response and General “Snappiness”

In addition, I still find the GUI to be extremely responsive and snappy. One gripe I have about a lot of products is their sluggishness as you scale out or begin pulling in a lot of data points. Even over my WAN to the Xangati lab, the Dashboard was still extremely fast to load any data I wanted, and moving from screen to screen (such as from a desktop user to a desktop VM, to the host that the VM was on, to the entire cluster) was nice and fast.

Citrix Support

The VDI Dashboard also offers support for both VMware and Citrix brokers. While not all that prevalent in a smaller environment, I do see mixed end user desktop solutions in larger enterprise deployments. Because Xangati does not use agents, you can have both environments being monitored by their product, and still get PCoIP (VMware) and HDX/ICA (Citrix) session statistics for a user.

Thoughts

This isn’t to say that Xangati is only a VDI monitoring solution. The VI Dashboard has all sorts of server metrics for environments that aren’t doing any virtual desktops. In addition, Xangati has also made mention that their new release will include more capacity management features available, including trend management and waste finding (locating capacity hogs). I’ll be interested to hear of results from users who start using the end-user tracking in the environments – please feel free to comment if you’re using Xangati and want to share an experience.

I have no clue how many turtles you can fit on a rock.

While I’m not writing this as some sort of greater VCAP-DCA study guide, I do think it ties in nicely to a number of VMTN forum questions that I’ve seen and my latest splurge of networking related posts. Also, it gives me a great excuse to do some mini-infographic work, which I really enjoy.

Note: One thing I do want to emphasis is that the Private VLAN concept is not limited to vSphere. It’s a networking technique that has been employed on switches for quite some time. However, it is available within vSphere as an added feature when using a vSphere Distributed Switch.

What Makes A VLAN “Private”

The gist is that you have a logical encapsulation of VLANs within a VLAN. Raise your hand if that made you think of the Xzibit “Yo dawg…” meme.

The infographic below details this a bit more. The blue Promiscuous VLAN is a Primary Private VLAN. It is an externally facing VLAN that is accessible to the external network, and needs to be an available VLAN that can be used by your network.

The Promiscuous VLAN acts as a gateway into the next tier of VLANs called Secondary Private VLANs. These are only reachable through the Promiscuous VLAN. As you may have already guessed, it’s quite common to put a router or multilayer switching virtual machine / appliance in the Promiscuous VLAN so that it can appropriately route traffic into the Secondary Private VLANs. It is also possible to simply have a multi homed virtual machine that uses a vNIC in a Secondary Private VLAN to reach other virtual machines.

Secondary Private VLANs

The Secondary Private VLANs are further broken up into two types: Community and Isolated.

A Community Secondary Private VLAN is one that can talk amongst itself, but can not directly contact any other Secondary Private VLANs. In the diagram above, the white bus connecting the 4 VM cubes denote that each VM can talk to each other, or to the Promiscuous VLAN.

The Isolated Secondary Private VLAN is unique – you can only have one per Private VLAN. VMs inside the Isolated VLAN can not even talk to each other; they can only communicate directly with the Promiscuous Primary VLAN.

Here is a screenshot of the Private VLAN setup on a vDS. I’ve highlighted each VLAN type with a colored box to match the infographic.

Physical Switch Compatibility Required

When you use a Private VLAN with VMs on multiple hosts, you must be aware that your physical switches must also support Private VLANs. Otherwise, the traffic may not be able to reach the other hosts, as the traffic will have to leave one host to reach another and may be discarded at the physical switching layer.

Fortunately, support for PVLANs is rather common.

Thoughts

While I don’t see PVLANs in production that often, I have been assured that folks are using them. Realistically, I can see a some use cases for them where you need to fence off applications (perhaps pre-production?) or wish to have it serve as a method of security to logically segment your virtual machines. This, of course, assumes you trust VLANs as a method of security, which is something I see debated off and on, but mostly tend to trust for my own uses.

So, What is LACP?

LACP, otherwise known as IEEE 802.1ax Link Aggregation Control Protocol, is simply a way to dynamically build an EtherChannel. Essentially, the “active” end of the LACP group sends out special frames advertising the ability and desire to form an EtherChannel. It’s possible, and quite common, that both ends are set to an “active” state (versus a passive state). Additionally, LACP only supports full duplex links (which isn’t a concern for gigabit or faster links). Once these frames are exchanged, and if the ports on both side agree that they support the requirements, LACP will form an EtherChannel.

Note: LACP has a number of other pieces when setting up an EtherChannel, such as calculating system and port priority, configuring an administrative key, and so on. This really isn’t relevant to what you, the VMware admin cares about, but it does exist if you want to go learn the nitty gritty.

LACP Is Not Included in Native vSphere Switches

Simply put, the native vSphere switches are not able to respond to LACP frames. It neither listens nor advertises. If you set up LACP on the remote switch, it will never hear a reply back from the vSphere host, and thus an EtherChannel will not be created.

If you want to form an EtherChannel with a vSphere host, you must create a Static EtherChannel. This is also referred to as being “on” as the commands to set up Static in Cisco is “mode on”. When set to Static, there is no discovery or advertisements – the EtherChannel is immediately created by the physical switch.

How About Load Distribution?

Here’s the kicker:

Both Static and Dynamic (LACP) EtherChannel use the same load distribution methods.

I put that in italics and bold to emphasize the point. Yes, it’s true. Static and LACP use the same load balancing techniques to handle traffic. If anyone tells you otherwise, bet against them for an easy way to make some quick cash.

This could be you!

But Static Requires IP Hash and LACP Doesn’t … Right?

Now we’re at the murky part. The answer is “wrong” but let’s go into the “why it’s wrong” part.

IP Hash is a native vSphere switch requirement. It does not support any other load distribution methods.

Taken from the vSphere Networking guide

Here is the warning from vSphere:

Notice that if I want to use EtherChannel, I have to select IP Hash for my load balancing method, and immediately there is a little information box warning.

Note: The term IP Hash equates to a load distribution policy of src-dst-ip on a Cisco switch.

Static EtherChannel under other circumstances can use any of the available load distribution policies. When talking to a vSphere host, however, it is forced to using an IP Hash because that is what vSphere is able to use.

If you want to use LACP with vSphere, you are required to install a Cisco Nexus 1000V virtual switch. There’s no other way to do LACP with vSphere at the time of this writing. And, since the 1000V is a (nearly) full featured Cisco switch, instead of a native vSphere switch, you can use any load distribution policy you want – you are no longer limited to IP Hash (src-dst-ip)

LACP Advantages over Static

LACP does have a few tricks up it’s sleeve, but none relating to getting traffic from source to destination any more efficiently.

Hot-Standby Ports

If you add more than the supported number of ports to an LACP port channel, it has the ability to place these extra ports into a hot-standby mode. If a failure occurs on an active port, the hot-standby port can replace it.

However, the typical supported number of ports for LACP is 8, so for a vSphere environment this is not a feature we would care about. Realistically, who is making 8 port EtherChannels to a vSphere host?

Failover

If there is a dumb device sitting in between the two end points of an EtherChannel, such as a media converter, and a single link fails, LACP will adapt by no longer sending traffic down this dead link. Static doesn’t monitor this. This is not typically the case for most vSphere environments I’ve seen, but it may be of an advantage in some scenarios.

Configuration Confirmation

LACP won’t form if there is an issue with either end or a problem with configuration. This helps ensure things are working properly. Static will form without any verification, so you have to make sure things are good to go.

Thoughts

Bottom line: I’m not saying LACP (or the Nexus 1000V) is bad. It’s a very popular protocol that I see used all the time. My issue is that I see people wanting LACP thinking that it will better balance traffic, or do something else that it’s absolutely not going to do. Don’t go out and pop a bunch of Cisco 1000Vs into your environment for using LACP unless you have a solid use case.

In some cases, I see what I call a “porcupine” host. One that is covered with NICs to satisfy VLAN requirements, as each uplink (or pair of uplinks) have been connected to access ports on the next hop switch. Yikes!

While there are some use cases for doing this (such as for an air gap security requirement) it’s not necessarily the best way to get the job done. Instead, the use of trunk ports may be appropriate and reduce both cost and complexity.

The infamous “Porcupine Host”!

A Brief Introduction to Trunking

Trunking is the concept of passing multiple VLANs over the same port using tags. The most common tagging is 802.1Q, which is an IEEE standard that nearly all switches support. The tag is there to identify which VLAN the layer 2 frame belongs to. vSphere can both understand these tags (receive them) as well as add them to outbound traffic (send them).

Here are a few good ideas when setting up trunk ports to a vSphere host:

1. Only present the VLANs that are necessary for the virtual machine guests on the vSphere host to the trunk port. This is the “Trunking VLANs Enabled” section of a switchport. It helps keep unnecessary broadcast traffic to a minimum.
2. I typically use both PortFast and BPDU guard on a trunk port destined for a vSphere host. PortFast enables the port to immediately begin forwarding traffic, without the need for spanning tree to ensure that another switch is not connected to the port. BPDU guard listens for BPDUs on the port, which are sent by other switches, to ensure that other switches are not accidentally connected to the port. vSphere standard and distributed switches are an exception to this, as they are unable to form a loop and thus spanning tree will be satisfied.

Configuring a Trunk on a Cisco Switch

From your Cisco switch, identify which port(s) the vSphere host will be connected to. This assumes you have some level of knowledge of how to work with Cisco IOS.

Here are sample commands on my Cisco Catalyst 3550 in the home lab:

User Access Verification
Password:
Lab3550>enable
Password:
Lab3550#config terminal
Enter configuration commands, one per line. End with CNTL/Z.
Lab3550(config)#interface fastEthernet 0/8
Lab3550(config-if)#switchport trunk encapsulation dot1q
Lab3550(config-if)#switchport mode trunk
Lab3550(config-if)#spanning-tree bpduguard enable
Lab3550(config-if)#spanning-tree portfast
%Warning: portfast should only be enabled on ports connected to a single
 host. Connecting hubs, concentrators, switches, bridges, etc... to this
 interface when portfast is enabled, can cause temporary bridging loops.
 Use with CAUTION
Lab3550(config-if)#no shutdown
Lab3550(config-if)#end
Lab3550#

There, I have now have a trunk port.

If I wanted to limit the VLANs allowed on this trunk, the command below can be issued once you enter the (config-if) mode for the interface:

Lab3550(config-if)#switchport trunk allowed vlan [WORD | add | all | except | none| remove]

Setting VLANs on vSphere Portgroups

Now that the switch is configured to pass multiple VLANs over the trunk port, the final step is to create portgroups on the vSphere host. In this scenario, we’re creating portgroups in VST (virtual switch tagging) mode. This means that the virtual switch will read (remove tags) and send (add tags) to traffic, while the guest will be completely unaware of the tags.

The process is simple. From within the settings of a portgroup, navigate to the VLAN section. Change the VLAN type to VLAN, and enter the VLAN ID the portgroup should be receiving. Below is my home lab portgroup that is listening for tags on VLAN 10.

I also find it helpful to name the portgroup something that reflects the VLAN. In my example above, my portgroup is named “VLAN10 – 192.168.10.X” to easily denote both the VLAN and subnet being used.

Thoughts

This one is pretty straight forward, so I will leave you with a dancing bear.

However, there is another option available to those with Enterprise Plus licensing: the “Route by physical NIC load” load balancing policy, otherwise known as load based teaming (LBT). This policy is only available on distributed switches (vDS) which cannot be created if you are not licensed for Enterprise Plus.

Load based teaming is a very powerful technology that monitors vmnics (uplinks) for saturation. When an vmnic reaches 75% utilization for 30 seconds, LBT tries to move workloads to other, non-saturated vmnics. It is my opinion that this was mostly created with the mindset of balancing VM traffic, but it also works well for vmknics carrying NFS traffic. In this post, I’ll go over how this process works, configuration, and a lab test.

Load Based Teaming

The concept of monitoring load and migrating traffic around is nothing new to the world of vSphere. VMware admins are constantly leveraging the ability to vMotion workloads around for maintenance and balance, along with tools such as the Distributed Resource Scheduler (DRS) to assist in automated workload distribution.

Some interesting things about load based teaming and how it works:

• The power of load based teaming exists outside of the portgroup construct. Meaning, you don’t need all of your VMs or vmkernels to exist in a single portgroup to take advantage of load based teaming.
• As long as “Route based on physical NIC load” is selected, any portgroup will proactively monitor the vmnic utilization in their team and shift workloads around, even if another portgroup is responsible for generating the load.
• Ultimately, vmnic utilization triggers moving workloads.
• Turning on LBT is non-invasive and does not impact the active workloads.
• Only active vmnics are considered for movement. Any standby or unused vmnics are not targeted as destinations.
• Saturated 100 MB links do not trigger LBT movement, and I tested this in the lab to confirm – though, is anyone seriously using 100 MB links on their vSphere host?

With that said, let’s cover the configuration of this lab environment to showcase the power of LBT.

Lab Configuration

This time around I’ve reconfigured the lab entirely. The NetApp simulators are incredibly sluggish to configure and test against, so I have switched over to Nexenta’s Community Edition on a virtual machine.

Below is the logical configuration. I’ve created a lab using a single NAS server (Nexenta CE) presenting 4 exports. All traffic is on VLAN 1, 2, 3, and 4 (which is 10.0.X.0/24 in my lab, where the VLAN number equals the third octet) to an ESXi host running 5.0 update 1 (build 623860). The host has 2 uplinks along with 4 vmkernels. In order to consistently create traffic, I have deployed 4 of the VMware IO analyzer appliances – one on each export. This allows me to quickly simulate VM traffic going to all of the exports at the same time.

Lab Screenshots

Rather than using a virtual host, I have rebuilt the lab network to work on on my “production” hosts and switches. This makes it much easier to generate enough traffic to trigger a LBT movement and eliminates the massive amount of duplicate frames received (as seen with virtual hosts on a promiscuous portgroup).

Additionally, the storage has been presented by the Nexenta over NFS. Under the hood are a pair of SSD drives, giving me plenty of IO for this test and the ability to simply mount the same datastore repeatedly using different VLANs.

Lab Test – Triggering Load Based Teaming

Let’s first take a look at the environment and identify the relationships between vmkernels and vmnics (uplinks). vmk1 and vmk4 have been put on vmnic3, while vmk2 and vmk3 are using vmnic0. This was decided by the hypervisor, I had no input in the matter.

Also, note that vmk0 (my management vmkernel) is using vmnic3 and is in an entirely different portgroup. I enabled LBT for that portgroup as well, to prove that LBT doesn’t care about portgroups as a delimiting factor.

Let’s see if we can generate a lot of traffic on vmnic3 and get the other guys to use vmnic0. I’ll fire up the IO Analyzer that is sitting on VLAN1 (vmk1) and see if we can get LBT to shuffle things around. Below is a screenshot showing the results, along with a zoomed image of the ESXTOP data.

The IO Analyzer saturated all of vmnic3, so LBT moved all other vmkernels over to vmnic0, even the management vmk0 on an entirely different portgroup. As you might imagine, this is a very powerful method for load balancing.

For the sake of fun, I’ll generate another big spike of load on the 3 vmkernels sitting on vmnic0 and watch LBT balance them. Below you can see vmk2, vmk3, and vmk4 kick off a large read spike that saturates all of vmnic0.

After trending the traffic for 30 seconds, LBT kicks in and migrates vmk2 and vmk3 to vmnic3. It’s somewhat difficult to balance 3 workloads that are going full speed on 2 uplinks, but LBT does a good job at trying.

Thoughts

It seems that load based teaming is a great way to address dynamic shifts in workload, and is relatively easy to set up. If you’re using Enterprise Plus licensing and are comfortable with distributed switches, this is probably the best way to go. Keep in mind, however, that you will need to oversubscribe your vmnics (uplinks) with a higher ratio of vmkernels. Otherwise, LBT will have nothing to balance. For example, if you had 2 vmkernels for 2 vmnics, each vmkernel has a dedicated uplink – there’s nothing it can move around.

I hope you’ve gained some valuable insight into the world of NFS on vSphere through my deep dive series, and no longer feel that the protocol is only suitable for ISO storage.

NFS on vSphere – Deep Dive Series

The entire series of NFS on vSphere deep dives:

1. NFS on vSphere – A Few Misconceptions
2. NFS on vSphere – Technical Deep Dive on Same Subnet Storage Traffic
3. NFS on vSphere – Technical Deep Dive on Multiple Subnet Storage Traffic
4. NFS on vSphere – Technical Deep Dive on Load Based Teaming

Lab Configuration

The lab looks nearly the same as with the same subnet discussion, but has been adjusted to use a unique VLAN for each vmkernel and storage target.

To demonstrate the way that vSphere routes NFS traffic on multiple subnets, I’ve created a lab using 2 NFS servers (NetApp Simulators) presenting 2 exports each, for a total of 4 exports. All traffic is on VLAN 1, 2, 3, and 4 (which is 10.0.X.0/24 in my lab, where the VLAN number equals the third octet) to an ESXi host running 5.0 update 1 (build 623860). The host has 2 uplinks along with 4 vmkernels. In order to consistently create traffic, I have deployed 4 of the VMware IO analyzer appliances – one on each export. This allows me to quickly simulate VM traffic going to all of the exports at the same time.

Lab Screenshots

First, let’s take a look at the vDS with the new set of vmkernels on multiple subnets. To make things easier to remember, I’ve used the same 4th octet for the set of vmkernel IPs. Each vmkernel is on a unique VLAN (subnet) to a trunked uplink port.

The datastores are also different, with each mapping being presented over a unique VLAN.

Lab Test – NFS Traffic Simulation

There’s really only one point I’m trying to make with using multiple vmkernels on multiple subnets: mounting each export across a unique subnet (VLAN) forces traffic out a vmkernel on that particular subnet (VLAN). The test is simply to simulate load on all four VMware IO Analyzers to see where the traffic was routed to verify my hypothesis. If you recall from above, each appliance is mapped over a different subnet (VLAN).

Here is a screenshot of the load generation in the IO Analyzer. Again, the VM IP Address is for management and is not relevant for the NFS traffic.

Once the simulation begins, the traffic was sent over all four vmkernels and ultimately both vmnics (uplinks).

But wait!

If all four analyzers are sending traffic, how do we know for sure that the traffic for a specific subnet stayed on the vmkernel for that subnet?

Good question. I’ll fire up some traffic on just VLAN 4 (vmkernel 6) and see what happens. Below, the ESXTOP for network shows that all traffic is properly being sent over vmkernel 6 which is on VLAN 4 (10.0.4.0/24).

Lab Test – Conclusions

The test above brings about certain conclusions:

• The hypervisor follows the standard rules of routing. If you have an IP (vmkernel) on a subnet and are trying to reach a destination IP on that same subnet, the hypervisor is just going to send the traffic directly. It will not try to use the default gateway (management vmkernel).
• Using unique subnets to break down the traffic will utilize multiple vmkernels (one for each subnet).
• As long as a vmkernel exists in a subnet, the hypervisor will use it for all traffic destined to a target on the same subnet. Traffic does not cross subnets if it doesn’t have to.
• If you create multiple subnets and present a vmkernel in each subnet, along with a storage interface on each subnet, you can mount exports on the various subnets to do static load balancing.

Pinning Traffic

Additionally, you can opt to pin the traffic to an uplink. If you have 2 uplinks, as with my lab above, you would do the following:

• Create 2 NFS storage subnets
• Create 2 portgroups, 1 for each subnet, and place a vmkernel in each portgroup for that subnet.
• Set the first portgroup uplink team to active / standby, forcing traffic out the first uplink.
• Set the second portgroup uplink team to standby / active, forcing traffic out the second uplink.
• Present exports (datastores) over the unique subnets as needed.

With this design, you could present exports to each subnet based on known load values. If Subnet 1 is filling up with NFS traffic, storage vMotion VMs over to datastores on Subnet 2.

Thoughts

Using multiple subnets should be the de facto method for balancing traffic when using an EtherChannel is not available (or desired). There’s really no other way to get the traffic to route over multiple uplinks and vmkernels without multiple subnets, as otherwise the traffic will continue to route over a single path.

NFS on vSphere – Deep Dive Series

The entire series of NFS on vSphere deep dives:

1. NFS on vSphere – A Few Misconceptions
2. NFS on vSphere – Technical Deep Dive on Same Subnet Storage Traffic
3. NFS on vSphere – Technical Deep Dive on Multiple Subnet Storage Traffic
4. NFS on vSphere – Technical Deep Dive on Load Based Teaming

Lab Configuration

To demonstrate the way that vSphere routes NFS traffic on a single subnet, I’ve created a lab using 2 NFS servers (NetApp Simulators) presenting 2 exports each, for a total of 4 exports. All traffic is on VLAN 5 (which is 10.0.5.0/24 in my lab) to an ESXi host running 5.0 update 1 (build 623860). The host has 2 uplinks along with 4 vmkernels, giving the NFS traffic a lot of options. In order to consistently create traffic, I have deployed 4 of the VMware IO analyzer appliances – one on each export. This allows me to quickly simulate VM traffic going to all of the exports at the same time.

The logical diagram is below. Note that vmk10 has the lowest IP (10.0.5.3) but the highest vmkernel number:

Lab Screenshots

A look at the vDS showing the 4 vmkernels mapped to 2 uplinks. All vmkernels are in the VLAN 5 subnet. No other vmkernel is on VLAN 5; it is a completely isolated subnet.

Next, a picture of the NFS datastores. Note that each datastore is mapped on VLAN 5 to the pair of NetApp Simulators. Mapping was done using a simple PowerShell script.

Lab Test #1 – NFS Traffic Simulation

For the first test, I simply crank up the IO Analyzers and watch ESXTOP. All analyzers were powered on at the same time and storage was presented just prior to loading the analyzers, giving the datastores an opportunity to choose any of the 4 vmkernels on VLAN 5.

The following screenshot shows the configuration of IO Analyzer for my 4 appliances. The IP addresses for each appliance reflect the management IP address and has no relationship to the subnet used for NFS storage.

All NFS traffic chose vmk7, which is using vmnic6. The receive numbers tend to confuse ESXTOP because a virtual ESXi server requires a promiscuous port to operate, and so there are many duplicate frames being received on the other uplinks. Regardless, vmk7 is clearly transmitting the read requests (12,711.04 in this photo).

Lab Test #1 – Conclusions

This test verifies a number of things.

• vmk7 is the lowest vmkernel number and first vmkernel available in the team (vmk8, vmk9, and vmk10 are the other members).
• vmk10 has the lowest IP (10.0.5.3), debunking any thoughts that the IP address is relevant to uplink selection. The host doesn’t seem to care about IP address for selection.
• NFS Datastores do not look at “randomly” picking a vmkernel when they are created, but instead seem to pick the first vmkernel on the list. This doesn’t necessarily mean the lowest number, but typically when creating vmkernels they follow an incremental number pattern (7, then 8, then 9, and so on).

Lab Test #2 – Uplink Removal and Addition

For this test, I am going to run the same IO Analyzer simulation, but then remove the active vmkernel (vmk7), let traffic migrate to another vmkernel, and add vmk7 back into the team. This test will verify if vmk7 is actively sought out, or if a migration of the datastores to another vmkernel is permanent until another failure.

First, traffic is simulated and vmkernel 7 (vmk7) is removed.

The host moves traffic over to vmk8. Note below that vmk7 no longer exists.

I then re-added vmk7 back into the team.

Traffic remains on vmk8. I’ve highlighted vmk7 and vmk8 below.

I’m not going to fill this post with pictures, but I also removed vmk8 and saw traffic move to vmk9. Removing vmk9 moved traffic to vmk10, and then removing vmk10 put the traffic back on the original vmk7.

Lab Test #2 – Conclusions

From this test, a few other conclusions can be made:

• The next vmkernel port on the list is chosen in the event of vmkernel removal. This test reinforces the idea that the list order is how selection is determined, not the vmkernel number. Notice that vmk7 appears at the bottom of the list after being removed and added, but was not chosen again until all other vmkernels were removed.
• Adding a vmkernel with a lower number to the list does not seem to influence the next vmkernel choice for NFS traffic.

Additionally, I unplugged the uplink and saw that the vmkernel simply moved over to the other available uplink.

Thoughts

I hope this clears up some confusion on same subnet vmkernel selection for NFS storage (and unbound iSCSI storage). The main takeaway here is that using a single subnet has no chance of load balancing NFS storage traffic in a default configuration.

If you’re using EtherChannel on a single subnet, the best you can hope for is a single vmkernel on the host and multiple IPs on the storage target, in which the switch’s Load Distribution policy will do an IP hash. But, you will need a different target for each uplink with a unique least significant bit (or bits, depending on uplink count). This means that if you have 2 uplinks, you’ll need 2 storage targets. Subsequently, 4 uplinks will require 4 storage targets. Using multiple vmkernel ports on the same subnet will have no influence on your EtherChannel Load Distribution because the host will only pick a single vmkernel to route traffic!

NFS on vSphere – Deep Dive Series

The entire series of NFS on vSphere deep dives:

1. NFS on vSphere – A Few Misconceptions
2. NFS on vSphere – Technical Deep Dive on Same Subnet Storage Traffic
3. NFS on vSphere – Technical Deep Dive on Multiple Subnet Storage Traffic
4. NFS on vSphere – Technical Deep Dive on Load Based Teaming