Hyper-Threading (HT) is sometimes viewed as the mystical voodoo that enables you to have twice as many logical cores on your physical CPU. Occasionally, the voodoo is abused when sizing the quantity of virtual machine vCPUs for a workload. After all – if the vSphere host says I have 8 logical processors available to me, can’t I use all of them for a single VM and its vCPUs? Hyper-Threading definitely has its advantages, but to properly use the technology in your vSphere host, you must understand a little of the basics.
Hyper-Threading in a Nutshell
Without going too deep into the weeds, the main point of HT is to allow two different compute tasks (threads) to schedule execution resources on the same underlying physical core. There are often periods of time where a physical core is underutilized because a compute task is waiting on a device. Other tasks would be blocked from executing on the physical core until the running task completed. HT allows the scheduler to assign two tasks to a physical core, effectively sharing the execution resource.

Each physical core contains a single execution resource. HT allows for greater scheduling efficiency, but cannot speed up a single task.
Over Sizing the vCPUs in a Virtual Machine
A vSphere host is smart enough to identify an Intel CPU that can perform Hyper-Threading. It will display the quantity of physical sockets, physical cores per physical socket, and logical processors. If your server does not have HT, the quantity of physical cores and quantity of logical processors will be equal. When HT is enabled, the quantity of logical processors will be double that of the physical cores.
You may have a desire to create a virtual machine that can consume all of your logical processors. You will not encounter an error if you try to do this, but it is ill advised. Let’s look at why.
In this example, I’ll use one of my home lab servers. Each server has an Intel Xeon E3-1230 processor with four physical cores and HT enabled. This gives me 8 logical processors. If I create a virtual machine with 8 vCPUs, it will look a bit like this:

Putting aside co-stop and other scheduling concerns with having a single VM consume my entire CPU, there are also issues with running tasks inside of my virtual machine. The VM believes that it has access to 8 different CPU cores. In reality, the vCPUs are sharing sibling Hyper-Threads (two Hyper-Threads on the same underlying core), which is something the CPU Scheduler tries to avoid.
What if the VM guest tries to run 8 tasks that require 100% of the execution resources from the cores? There are only 4 execution resources available. Thus, this would cause a great deal of CPU Ready because the VM would be ready to consume CPU time on 8 cores but only 4 are actually available. If you looked at ESXTOP, you would most likely notice that each HT was at 50% UTIL, and each PCPU (physical core) would be at 100% UTIL.
If I had two physical sockets of the configuration shown above, the CPU Scheduler would likely place 4 of the vCPUs on CPU 0 and the other 4 vCPUs on CPU 1. This would avoid any need to schedule vCPUs on sibling Hyper-Threads, but may cause other headaches elsewhere with NUMA node locality, co-stop, and workload contention.
Avoid creating a single VM with more vCPUs than physical cores. Having multiple VMs that, when combined, exceed the number of physical cores is typically OK – this is called overcommitment.
Thoughts
Hyper-Threading offers a great deal of scheduling flexibility in almost all virtualization scenarios. It’s perfect for sharing the physical execution resources available with a large quantity of virtual machines. Just be careful that you don’t abuse the logical processors by creating a virtual machine that has more vCPUs than there are physical cores available. Always try to keep the quantity of vCPUs set to a size that is needed by the virtual machine, and if you must grow to a size that is larger than your physical core count, it’s time to either buy a bigger host or split up the roles serviced by the virtual machine.

Interesting piece Chris!
Citrix in the past also has done some writing on this and if I compare your key takeaway:
Avoid creating a VM with more vCPUs than PHYSICAL cores.
With theirs:
It is advisable not to allocate more vCPU than there are LOGICAL cores within the given hardware.
I’m getting even more confused..maybe I’m crossing logical with physical but confusing it remains (as Yoda would say).
I’m referring to these articles:
http://blogs.citrix.com/2011/06/15/virtualization-best-practices-for-xenapp/
http://blogs.citrix.com/2013/01/07/whats-the-optimal-xenapp-6-5-vm-configuration/
Any thoughts?
I agree – I would never configure more vCPUs on a VM than available logical cores because that would offer zero value. But I also advise avoiding the physical core boundary as well because there is a great potential for CPU scheduling woes and abuse. Mark has a good corner case comment below that might shine light on a corner case to use more vCPUs in a VM than physical cores on the host.
Consider the advantage of parallelism.
[corner case]
I agree with Chris that I often size to the physical core count to be conservative and ensure a solid performance profile. That said, sometimes there is an advantage of offering more vCPUs, and therefore more parallel threads, to an application – IF – you can be reasonably assured the threads don’t saturate the underlying physical execution resource. This type of configuration allows the advantage of many threads to run in a short and/or bursty type pattern which can benefit some workloads. In this case I use the word “evaluate” to see if you can improve application KPI’s by using more logical CPUs than physical CPUs.
This is an excellent point. It would definitely require a very close eye on the workload and environment size, and if the risk outweighed the performance loss, may be worth moving forward with. Thanks, Mark!
I often have to answer questions with “it depends” in IT.
As to scheduling more vCore than logical Core, I can see when this would be appropriate; e.g. if you have two server running incompatible software that work in batch mode virtualised on a physical server then along as they don’t both wake up at the same time then you’re good.
The issue is very much a one of “it depends”. Again you might want to over provison the total number of vCpus vs pCPUs per vm by 50%? So that when a vm needs a lot of power there is the possibility of it being able to use it.
As you say, you need to keep a very careful watch as you can plan for X expect worst case to be Y and your users deliver ZZ!
@Alister
This post is specifically focused on the size of a single VM, not about overcommit of the total vCPUs to pCPUs on a host. It’s definitely quite common to see overcommit on vCPUs:pCPUs on a host, sometimes as high as 8:1, depending on the workload.
I have noticed since moving to Intel i7-4770 (Quad Core w/ HT) from an AMD FX-8320 (8 core), my CPU is at like 70% for all 4 cores. Does the % take into affect HT?
The host CPU bar is for the physical execution resources (cores) in GHz. If you open the summary page, you’ll see that CPU usage says “Capacity [# of physical cores] x [clock speed] GHz”
Check out VMware KB 2055995 (http://kb.vmware.com/kb/2055995): Difference between cpu.usage and cpu.utilization counters for HostSystem object.
“The counters for the HostSystem object provide this information:
cpu.utilization – Provides statistics for physical CPUs.
cpu.usage – Provides statistics for logical CPUs. This is based on CPU Hyperthreading.”
[…] Hyper-Threading Gotcha with Virtual Machine vCPU Sizing (Chris Wahl) VMware vSphere 4: The CPU Scheduler in VMware ESX 4 (VMware Tech Paper) The CPU Scheduler in VMware ESX 4.1 (VMware Tech Paper) The CPU Scheduler in VMware vSphere 5.1 (VMware Tech Paper) Virtual Machine Monitor Execution Modes: in VMware vSphere 4.0 (VMware) Does corespersocket Affect Performance? (VMware vSphere Blog) Four Things You Should Know About ESX 4’s Scheduler (Virtual Performance) […]
Is this “best practice” per Socket or per core(s)? For instance, I get that in a 1 socket 6 core processor a VM should not have more that 6 vCPUs. That makes sense to me. But what about a host with 2 socket, 6 core processor (12 cores total). Does this mean that my “monster VM” can have 12 vCPUs? Does the scheduler span processes across cores?
Or will it schedule 6 threads…then wait for the other 6 threads from the other 6 vCPUs on the other socket?
I wrote:
“Does the scheduler span processes across cores?”
I meant:
Does the scheduler span processes across sockets? lol
Thanks!
Yes, the CPU scheduler can span physical sockets. If I had two physical sockets of the configuration shown above, the CPU Scheduler would likely place 4 of the vCPUs on CPU 0 and the other 4 vCPUs on CPU 1.
Hey Chris,
Thanks! I know you wrote “Avoid creating a VM with more vCPUs than physical cores”. I just needed the clarification of that was per socket or the entire cores of the host.
Thanks for the reply!
I have a ESX 5.1 cluster running a windows 2008 SQL cluster and a Windows 2012 SQL cluster. We will have 2 (2 node 2008) and 2 (2 node 2012) clusters. Should I turn on hyper-threading?
Thanks in advance for any responses.
[…] deliver that big an advantage, especially in my lab environment. Nice piece on this subject: http://wahlnetwork.com/2013/09/30/hyper-threading-gotcha-virtual-machine-vcpu-sizing/ I decided to buy the Intel i5 4440 processor. A common CPU, on-cpu graphics, well […]
Hi Chris,
I am a newbie and your article is a relief
Central Idea is hyper threading is helpful only if I create 4 vms and assign a core each.
AMD FX 8350 says that they have 8 cores (some post suggests 4 strong and 4 weak cores)
i5 4670k says they 4 cores with hyper threading option.
Which one should I go for if I need a strong processor to run multiple VMs..
Can I go AMD and run more than 4 VMs simultaneously?
Amritanshu – It’s perfectly fine to overcommit the total number of vCPUs across multiple VMs. Environments typically go anywhere from 4:1 to 8:1 overcommit. This article only addresses the creation of any single VM with more vCPUs than pCPUs.
Hi Chris. Thanks for the info.
I am actually having the same situation as Amritanshu, I hope he doesnt mind me jumping in and ask my question here too as I see it quite related.
Like I said, I would like to build a rig that will support multiple VM, and each VM will be running some sort of software for data mining (web). Will I have a higher advantage going with 8-cores, or HyperThreading?
When you said its fine to overcommit, suppose I have 8 cores, I can setup 4 VM and assign 2 cores to each? So what’s left on my host then?
And if I go with the i7 (hyperthread – quad core), I will be assigning 1 core to each of my VM?
Sorry, my question might sound confusing because I am confused myself! I hope it makes sense. I have no idea what is vCPU and stuff. I just want to make sure I buy the right CPU before the weekly discount ends!
Thanks!
@Andy –
Think of it this way. A CPU socket with 4 physical cores but 8 HyperThreads still ultimately has 4 execution resources, one per physical core. A HyperThread is just a clever way to share out a physical core for multiple threads / VMs. If you had a VM that needed to use 100% of all 4 physical cores, your host would have no more CPU resources available for other workloads – it is using ALL of it. This is a bit rare for most use cases – usually a VM will burst to 100% for short periods of time, allowing other VMs to use the cores and share the execution resource. The ESXi scheduler doesn’t “assign” a core to a VM – it can shift around bursty VMs in real time.
tl;dr – I would snag a CPU that can HyperThread because it is really good at handling a mixture of VMs. I often see clients who have 8 or 12 HyperThreads create several VMs with a total 16 ~ 32 vCPUs, which is overcommiting (more total vCPUs than HyperThreads). And it runs fine for 99% of folks. This post is focusing on making ONE GIANT VM with more vCPUs than physical cores, which I do not agree with.
@Chris,
Thanks for the prompt response.
Yeah.. I believe I will be better off going with intel hyper-threading feature over AMD in this case. I read that it also consume less power and in long run, I guess it will break-even with the AMD in terms of price.
I was doing a bit more research today and found that i7-4770 and E3-1240/1245 actually have very similar features and perhaps even better than the i7 itself when it is used for virtualization. What your opinion on this?
And I am really just an amateur trying to get into all these VM stuff, and have no idea what ESXi setup is. I was planning to download VM player and setup XP on it etc. Would that be fine?
[…] PM #25 Hyper-Threading Gotcha with Virtual Machine vCPU Sizing | Wahl Network Originally Posted by ChooseLife Congratulations, you have passed the necessary […]
This article made me doubt that it is a good idea to allocate 12 cores to a virtual machine on a 6-core processor with hyper-threading. Thankfully I did not trust it blindly and made some tests.
Use case: I have 3 VMs, 1 is router/firewall (pfSense), 1 is NAS storage (Solaris), 1 is Win7 x64 sometimes doing CPU-intensive tasks, such as video encoding. I allocate 12 cores to the Win7 VM and 1 core to the other VMs. I measured how fast I can encode videos and here are my test results:
a) bare-metal Win7 x64 installation: 7.93 FPS
b) VM under ESXi with 12 cores allocated: 7.71 FPS
c) VM under ESXi with 6 cores allocated: 6.28 FPS
This is on Xeon 1650v3 CPU (6 cores with hyper-threading). Higher FPS is better, of course.
Conclusion: best performance is achieved when I allocate all logical cores to the CPU-demanding VM.
@TtG – I’m glad you tested it, too. It’s actually somewhat common to see a 10-15% increase in performance when using HyperThreads in the manner you described, but nothing near the 2x increase you might expect from having double the logical cores. Keep in mind, too, that it’s relatively trivial for the hypervisor to provide resources to 3 VMs in which 2 of them have but 1 vCPU. Mark Achtemichuk wrote a fantastic blog post describing your results, which is linked in the article towards the top.
Wonderful insight, just what i needed.
[…] In this example, our 2 socket, 8 core system with 16 cores total now presents 32 cores to VMware. Chris Wahl has an excellent post on this topic which I strongly encourage you to read, but for now I’m just going to “borrow” […]
In case of Xen, I normally pin VCPUs to DomUs to avoid internal CPU scheduling as much as possbile. In real production environments, isn’t it better to just pin each VCPU to a physical core ? I know this might reduce the number of VMs that can be run on a single machine but I think its more stable.
For vSphere, pinning should be avoided under most circumstances. The scheduler is designed to handle the relationship between virtual CPUs and physical cores. I’ve only encountered pinning one time, and it was to circumvent an older ESXi 4.x issue with NUMA (fixed in 5.x and later).
Hi Chris,
Would you recommend Intel Xeon E5-2660 v3, which has 20 HT, or would you go with something like Intel Xeon E5-4627 v3 for a virtual environment? Each host would have about 15 Servers running 2008/2012 R2, and about 8GB of ram. I am in the process of upgrading to a Cisco UCS B420 host, and trying to decide on CPU’s.
I’m not sure I can make specific processor recommendations, because this is just a high level discussion on how Hyper-Threading works. But I tend to use HT enabled processors for vSphere because it’s typically rare to have a workload consume 100% of a single physical core, so having HT helps schedule multiple workloads.
we are migrating from 2 * 6 core, total 24 processor(Hyper-thread enabled) server to VMs; ow many VCPUs we should request for?
I got a host that has a 12 cores, so according to you i have a total of 24 VCPU. I have 6 VM running on that server which requires a total of 19 cores but after allocating required cores to the VM i ran a prerequisite tool and i got i the following message “Action: The number of CPU cores configured for the virtual machines in the inventory exceeds the number of CPU cores available to the host. Configure virtual machines with appropriate capacity”. This is a Hyper V environment
[…] deliver that big an advantage, especially in my lab environment. Nice piece on this subject: http://wahlnetwork.com/2013/09/30/hyper-threading-gotcha-virtual-machine-vcpu-sizing/ I decided to buy the Intel i5 4440 processor. A common CPU, on-cpu graphics, well […]
[…] Explanation:A total of 24 vCPUs can be assigned to a virtual machine if hyperthreading is enabled.Reference: http://wahlnetwork.com/2013/09/30/hyper-threading-gotcha-virtual-machine-vcpu-sizing/ […]