Windows 2003 and vSphere Refuse To Handshake

Upgrading any environment can be a bear of a task, especially when making huge leaps across major releases. Such is the case for those still escaping the clutches of vSphere 4.x, which is past End of General Support in its lifecycle. There is, however, an interesting little caveat beyond the upgrade relating to those also running Windows 2003 or Windows XP – both rather long in the tooth – and trying to connect to a vSphere 5.1 or greater environment.

A Firm Handshake

After going through a long and tiresome upgrade from vSphere 4.x to vSphere 5.1 (or higher), you might find that certain clients are unable to connect to the vCenter Server by way of the vSphere Client or PowerCLI scripts. The vSphere Client simply fails to connect after a short period of time, with no reason given, and has all the symptoms of a firewall or network issue at first blush.

Pretty much always
Pretty much always

However, when I ran into this issue, it was on clients that could formerly connect to the vSphere Client. The issue did not present itself until after the vCenter Server was upgraded from 4.1 to 5.1. Therefore, we had some reasonable assurances that it was not a network or firewall issue, especially after validating that other folks were able to connect just fine. See: Sphere Elimination.

In this case, it turns out that the server, being vCenter, is expecting the client to use a strong cipher. The operating systems I mentioned earlier, being Windows 2003 and XP, do not possess these strong ciphers and are unable to complete a handshake with the vCenter server.  In fact, VMware KB 1018510 does a great job at showing exactly what changed between various vCenter versions, shown below with yellow highlights that I added to pinpoint the ciphers that are no longer supported:

vCenter Server cipher support
vCenter Server cipher support

Three of the formerly supported ciphers have been removed in vCenter 5.x.

Patching Windows 2003

Are older operating systems just out of luck? Actually, no, there are a few ways around the issue outside of an upgrade to a newer version of Windows. Here are two workarounds:

  • Patch your Windows 2003 operating systems, or
  • Allow weak ciphers on your vCenter Server

As per VMware KB 2049143, there are a handful of Microsoft hotfixes that address the issue on Windows 2003 (See Microsoft KB 948963). Essentially, these two ciphers are added:

  • TLS_RSA_WITH_AES_128_CBC_SHA AES128-SHA
  • TLS_RSA_WITH_AES_256_CBC_SHA AES256-SHA

You can download a hotfix for your flavor of Windows 2003 below:

If you’re stuck on Windows XP, your choices are much more limited.

There is no hotfix available for Windows XP (32-bit). Microsoft currently only provides limited support for Windows XP, and as a result the hotfix has not been released for it. To resolve this issue, you must upgrade your host operating system to Windows Vista or later, which support the use of high cipher strengths. (source)

Editing vpxd.cfg

Another alternative is to edit the vpxd.cfg file to allow weak ciphers. To start, head over to C:\ProgramData\VMware\VMware VirtualCenter\ and locate the vpxd.cfg file.

This is where vpxd.cfg lives
This is where vpxd.cfg lives

Create a copy as a backup. Then, open the file and add the following line just after the <ssl> tags:

<cipherList>ALL</cipherList>

Here’s an example:

Allowing all types if ciphers to connect
Allowing all types if ciphers to connect

Restart the vCenter Server service and the Windows 2003 and XP clients should be able to connect. VMware KB 2049143 also contains instructions for those using the vCenter Server Appliance (vCSA).

Thoughts

The added security from a strong cipher requirement provided by vCenter 5.1 and higher are something that I would hesitate to remove. If you need to get around it as a temporary workaround, by all means, but I would prioritize removing those older, end of life operating systems from the environment as quickly as possible.