NFS on vSphere Part 3 – Technical Deep Dive on Multiple Subnet Storage Traffic

Chris Wahl · Posted on

Now that I’ve gone over misconceptions of NFS on vSphere, as well as a deep dive on same subnet storage traffic, the next discussion will be around leveraging multiple subnets and using VMware’s load based teaming. Again, this article is focused on NFS traffic in regards to vSphere, but can also apply to iSCSI traffic when no binding has occurred.

Lab Configuration

The lab looks nearly the same as with the same subnet discussion, but has been adjusted to use a unique subnet for each vmkernel and storage target.

To demonstrate the way that vSphere routes NFS traffic on multiple subnets, I’ve created a lab using 2 NFS servers (NetApp Simulators) presenting 2 exports each, for a total of 4 exports. All traffic is on subnet 1, 2, 3, and 4 (which is 10.0.X.0/24 in my lab, where the subnet number equals the third octet) to an ESXi host running 5.0 update 1 (build 623860). The host has 2 uplinks along with 4 vmkernels. In order to consistently create traffic, I have deployed 4 of the VMware IO analyzer appliances – one on each export. This allows me to quickly simulate VM traffic going to all of the exports at the same time.

Lab Screenshots

First, let’s take a look at the vDS with the new set of vmkernels on multiple subnets. To make things easier to remember, I’ve used the same 4th octet for the set of vmkernel IPs. Each vmkernel is on a unique subnet to a trunked uplink port.

In practice, I’d use a unique VLAN for each subnet. However in this lab case I’m presenting all four subnets over a single VLAN.

The datastores are also different, with each mapping being presented over a unique subnet.

Lab Test – NFS Traffic Simulation

There’s really only one point I’m trying to make with using multiple vmkernels on multiple subnets: mounting each export across a unique subnet forces traffic out a vmkernel on that particular subnet. The test is simply to simulate load on all four VMware IO Analyzers to see where the traffic was routed to verify my hypothesis. If you recall from above, each appliance is mapped over a different subnet.

Here is a screenshot of the load generation in the IO Analyzer. Again, the VM IP Address is for management and is not relevant for the NFS traffic.

Once the simulation begins, the traffic was sent over all four vmkernels and ultimately both vmnics (uplinks).

But wait!

If all four analyzers are sending traffic, how do we know for sure that the traffic for a specific subnet stayed on the vmkernel for that subnet?

Good question. I’ll fire up some traffic on just subnet 4 (vmkernel 6) and see what happens. Below, the ESXTOP for network shows that all traffic is properly being sent over vmkernel 6 which is on subnet 4 (10.0.4.0/24).

Lab Test – Conclusions

The test above brings about certain conclusions:

  • The hypervisor follows the standard rules of routing. If you have an IP (vmkernel) on a subnet and are trying to reach a destination IP on that same subnet, the hypervisor is just going to send the traffic directly. It will not try to use the default gateway (management vmkernel).
  • Using unique subnets to break down the traffic will utilize multiple vmkernels (one for each subnet).
  • As long as a vmkernel exists in a subnet, the hypervisor will use it for all traffic destined to a target on the same subnet. Traffic does not cross subnets if it doesn’t have to.
  • If you create multiple subnets and present a vmkernel in each subnet, along with a storage interface on each subnet, you can mount exports on the various subnets to do static load balancing.

Pinning Traffic

Additionally, you can opt to pin the traffic to an uplink. If you have 2 uplinks, as with my lab above, you would do the following:

  • Create 2 NFS storage subnets
  • Create 2 portgroups, 1 for each subnet, and place a vmkernel in each portgroup for that subnet.
  • Set the first portgroup uplink team to active / standby, forcing traffic out the first uplink.
  • Set the second portgroup uplink team to standby / active, forcing traffic out the second uplink.
  • Present exports (datastores) over the unique subnets as needed.

With this design, you could present exports to each subnet based on known load values. If Subnet 1 is filling up with NFS traffic, storage vMotion VMs over to datastores on Subnet 2.

Thoughts

Using multiple subnets should be the de facto method for balancing traffic when using an EtherChannel is not available (or desired). There’s really no other way to get the traffic to route over multiple uplinks and vmkernels without multiple subnets, as otherwise the traffic will continue to route over a single path.