While the title is meant to be a bit of a dramatized attention grabber, I do run into a vast majority of situations where someone has configured a set of port channels, also known as an EtherChannel or the more recently popular static-LAG, to a vSphere host for storage traffic. More often than not this simply boosts the complexity of the architecture with no clear advantages, or causes issues in that the configuration is not appropriate for a vSphere host in the first place.

The main reason for this stems from the fact that a vSphere host contains virtual switches and most physical network designs call for setting up port channels between switches. Physical switches use these as a sort of “fat pipe” to transport traffic throughout the network, can be handy with peer links for virtual port channels (vPCs), and help avoid network loops. This doesn’t translate well into the virtual world: a virtual switch is not able to form a loop. There’s no way to link a vSwitch to another vSwitch without intentionally multi homing a virtual routing appliance and plugging it in the middle.

Let’s dive right in to some real world implementations of port channels for both iSCSI and NFS, and I’ll do my best to explain why I think it’s best to avoid port channels.

Port Channels for iSCSI Storage

This one’s easy: don’t do it.

The best way to set up an iSCSI network for vSphere is to use vmkernel port binding. This makes a 1:1 relationship between a vmkernel port and a physical uplink and allows you to use various path selection policies (Most Recently Used, Fixed, or Round Robin). Using a port channel is an inferior way to aggregate uplinks for iSCSI storage traffic. Port binding for iSCSI has been available since the vSphere 4 days, but is exposed via the GUI in vSphere 5.

[symple_box color=”yellow” text_align=”left” width=”100%” float=”none”]Need details on how to configure iSCSI vmkernel port binding in vSphere 5? Check out my post “Configuring Proper vSphere iSCSI Multipathing via Binding VMkernel Ports [Video]” [/symple_box]

Port Channels for NFS Storage

This configuration is what I commonly encounter in the field. You have a goal of making a “large pipe” for your NFS storage traffic because vSphere uses NFS v3 which doesn’t do multipathing, also known as session trunking. So, you create a port channel to accomplish this and think all is good – right?

Unfortunately, no. Let’s start with the good news: it is possible to successfully use a port channel for NFS traffic, and even if done incorrectly, traffic will usually still flow. But the bad news is worse: an incorrect configuration results in no realized benefits, or the negative possibility of pinning all traffic to a single uplink, and you will not be able to use more than one uplink’s bandwidth per NFS export no matter how you configure a port channel. That “large pipe” dream is just that – a dream.

Incorrect Design

Here is an example incorrect design.

Here’s how traffic flows in this scenario:

1. NFS I/O requests are generated
2. The vSphere host finds the first vmkernel port that exists on the same subnet as the NFS export IP
3. The port group that the vmkernel lives on is set to “IP Hash” which is required for a port channel
4. The host generates a hash of the source IP (the vmkernel port) and the destination IP (the NFS export) and uses the results to choose an uplink
5. NFS traffic flows over that uplink

Now here’s the kicker. Every time storage flows to that NFS export, the same IPs will be used, and the IP Hash results will be the same. It doesn’t matter how many VMs are talking to storage, nor how many vmkernel ports are created. You have essentially pinned traffic to this uplink because all Vol1 datastore traffic will use dvUplink1, and dvUplink2 will only be used in the event that dvUplink1 fails. There’s no value add here to having a port channel.

Working Design

So here’s another design that would work with a port channel. The advantage with this design is that you can force a single subnet of NFS traffic down two different dvUplinks.

Here’s how traffic flows in this scenario:

1. NFS I/O or read requests are generated
2. The vSphere host finds the first vmkernel port that exists on the same subnet as the NFS export IP
3. The port group that the vmkernel lives on is set to “IP Hash” which is required for a port channel
4. The host generates a hash of the source IP (the vmkernel port) and the destination IP (the NFS export) and uses the results to choose an uplink
5. NFS traffic flows over that uplink

It’s the same traffic flow as before, except that now the hash values are different because you are using multiple exports.

In this design, the IP Hash has sent Vol1 traffic to dvUplink1 and Vol2 traffic to dvUplink2 because of the different Hash values. Keep in mind that the bit value of “20” and “21” must be unique at the least significant bit – that is, the last bit. Binary 20 ends with a 0 and Binary 21 ends with a 1. Those are unique values. If I had used “20” and “22” they would not be unique – both of them end with a Binary 0.

You can imagine how complex this gets when more uplinks or NFS exports are introduced. Unless you are limited to a single subnet or suffering VLAN exhaustion (not common for most private enterprises), using additional subnets is often easier to both visualize and troubleshoot.

Thoughts

Port channels bring unnecessary complexity to almost every design I’ve ever seen for very little (if any) benefit. Make sure to have a solid use case and really understand exactly how IP Hashing works for a port channel before you go implementing one in your environment.

You can achieve superior results compared to the aforementioned “NFS Working Design” by using VLANs with unique subnets, which also gives you the added bonus of being able to invoke LBT (Load Based Teaming), known as “route based on physical NIC load” in the vSphere client. If you’d like to see how to do that, head on over to my “NFS on vSphere Part 3 – Technical Deep Dive on Multiple Subnet Storage Traffic” post. What I like about the multiple subnet design is that LBT is actively monitoring and looking to balance load, where as port channels do not.

Also, keep in mind that more modern deployments that utilize 10 GbE connections have little to worry about. If you’re maxing out a 10 GbE connection with NFS traffic you’re in a different situation than most. This article is generically focused on those who are limited to 1GbE connections for rack mount servers.