In a traditional rack mount vSphere design, a host would have several NICs. A pair of them would often be dedicated to vMotion in one way or another – either paired with management traffic in a flip flop manner, or isolated completely. It was assumed that the northbound switching infrastructure would handle transporting the traffic to another host. Often, a TOR switch (top of rack) would be the only hop necessary for transporting the vMotion Ethernet frames because the cabinet contained all of the vSphere hosts inside. In larger designs, where servers spanned multiple cabinets, the data may have to ride up into the aggregation (formerly known as distribution) layer – or sometimes all the way up to the core.

Now, enter converged infrastructure and the use of either chassis switches (as with an HP c7000 BladeSystem) or domain switches (as with Cisco UCS Fabric Interconnects). In these situations, there exists the ability to keep the vMotion traffic from ever egressing the chassis (for intra-chassis blade communication) or domain (for intra-domain blade communication). The design varies slightly for the two use cases, so I will focus on the simple fact of pinning traffic for now.

Network Design Visualization

It can be difficult to picture all the pieces and parts, so I have created a visualization of the common design used for Cisco UCS. Take a look below.

The vMotion port group contains two uplinks – vmnic0 and vmnic1. It doesn’t matter if these are the only two uplinks presented, such as with a 2 vNIC design, or if there are many vNICs. The point is that each vNIC will be bound to a different chassis or domain switch. Because this graphic is based on Cisco UCS, I’m using the Interconnect terminology. Since you would never want to use a single interconnect, nor have vmnics on the same interconnect, this is a rather universal and generic design.

The points of interest revolve around the red X‘s that I drew.

1. Interconnect X: First, the Interconnects do not carry traffic from one to another. The cables for L1 and L2 are used only to facilitate the Interconnects acting as a cluster, doing duties such as heartbeating.
2. TOR X: You should not assume that the TORs are connected to one another – make sure to verify this. In many cases, such as using Nexus 5Ks for TOR, you will indeed find a VPC spanning across the switches to handle the LACP port channel spanned across them. Make sure you understand the northbound switching architecture before making assumptions.

Based on the graphic above, you should have a inkling as to why pinning will boost performance.

vMotion Is Layer 2

Routing your vMotion traffic is not only unwise, but also not supported. The idea is to use a single subnet on an isolated VLAN to carry vMotion traffic. This eliminates any unwanted broadcast traffic, can increase security, and also provides easier troubleshooting. Additionally, VMware only supports a configuration in which vMotion traffic is switched (layer 2) and not routed (layer 3). Thus, you should use a single subnet across all hosts in the same vMotion domain.

vMotion and IP-based storage traffic should not be routed, as this may cause latency issues. Any internal/private subnet can work as long as it is unique and dedicated exclusively to that specific type of traffic. (source)

When using the converged infrastructure model shown above, pinning provides an opportunity for the vMotion traffic to stay within the chassis (HP) or domain (UCS). If all vMotion vmkernel ports share the same layer 2 switch, it never needs to go any further northbound than the Virtual Connect (HP) or Interconnect (UCS). While there are some other ways to do this with HP’s horizontally placed virtual connect modules (see Eric Shanks’ article on how to build a SUS), let’s just keep this simple.

How To Pin vMotion

Pinning is a method to force traffic to use a specific uplink. I borrow the term from the commands used to configure a Nexus 1000v switch (static pinning). In the case of vSphere switches it is the act of setting a specific uplink to active, while putting the rest to standby or even unused. You have then pinned traffic to use a specific path into (and out of) the host.

Here is an example of a pinned port group named “vMotion-A-253” in which dvUplink1 is set to transmit all traffic in my dvSwitch. Look familiar?

By setting this globally on the port group in a dvSwitch, you can then make sure that every host is using a specific side of the network fabric (either a specific Interconnect or Chassis Switch).

I also invite you to visit Josh Odgers’ blog post entitled “Example Architectural Decision – vMotion configuration for Cisco UCS” that covers this in a more decision tree type format.