Review: Supermicro ESXi 5 Whitebox Home Lab Servers
Now that my new vSphere lab is up and running, I wanted to write up a brief review on the Supermicro servers running ESXi 5. For details on the design and links to parts, review my previous post on the home lab.
Motherboard Bells and Whistles
This is my first build with Supermicro, and after building a server with their motherboard, it won’t be my last. The boards are well documented through both a CD and booklet (I always prefer to use the booklet) with pages of detailed diagrams on each feature that the board offers. The writing is easy to read and not broken English like you get with some manufacturers. Many explanations also include screenshots from an actual system. The board arrived with all of the latest firmware and bios revisions applied, so I did not need to update anything.
The IPMI feature is extremely valuable, and I refuse to build any system without some sort of out of band management. You can opt to either use the dedicated port for IPMI, or ride on top of the first NIC port. The web interface is very clean and simple to use and includes the ability to perform a number of maintenance tasks (firmware updates, system configuration, etc.) with a few clicks. I was also impressed with the amount of configurable options available: LDAP / Radius / AD integration, Dynamic DNS hooks, fan speeds, etc.
This is the main screen of the IPMI interface on the server, including a preview of the screen which opens the console when clicked.
The java based remote console is very peppy and feature rich, with the ability to pass along remote media to the server. I used this to install ESXi 5.0 update 1 on the internally mounted USB drive.
Here I have the console open, along with the virtual storage interface mounting the ESXi 5.0 update 1 ISO from my PC.
A number of configuration items available from the IPMI
vSphere Perspective
The server provides every feature I can imagine in vSphere, from Fault Tolerance to DirectPath I/O, along with having 2x 1GbE NICs.
A look at the server from vSphere
Passthrough device selection.
Caveats
A few points I’ll cover when setting up this server configuration.
Fan Connections
Make sure to plug in the fans directly to the motherboard, don’t use the PSU 3to4 pin converters. The board will alert with a siren-like noise if it doesn’t detect any fans running.
Power Supply Fit (Lian Li PC-V351 with Seasonic)
I found that the power plug receptacle on the Seasonic slightly bumped against the rear housing of the Lian LI PC-V351. The housing has a curved shape while the Seasonic has a square guard around the plug. If you mount the Seasonic properly (open vents facing up) this will prevent mounting the PSU. To fix this, I cut a small notch into the housing with a pair of tin snips (just a few millimeters into the case).
Thoughts
I’m very pleased with how this build turned out, and feel that the extra features offered by the Supermicro board made the price tag extremely palatable. Realistically, these are easier to manage and have more features (both software and hardware) than my Dell T110, ending my habit of buying servers from a vendor for the home lab.
For those using Supermicro, I’d be interested to hear your thoughts on the board (both good and bad) in the comments below. I can also try to answer any questions you may have on this build.
Nice build,I actually just built two of these with pretty much the same exact hardware last night! I went with micro 4Gb USB drives right on the handy on-board USB slot. I think I’m going to do two nested ESXi hosts per box and treat them as separate “datacenters”. Thanks for posting the NIC driver in one of your other posts!
That sounds like a cool idea (nested data centers), I’ve been using my Dell T110 as a management server so that I can blow up the Supermicro boxes at will, but also created some nested ESXi servers for making multi site configs.
I’ve been running the SuperMicro X8SIL-F for a year now w/ a Xeon 3450 & just bumped it up to 32GB RDIMMs not UDIMMs thank god (& v5.0.1). After nesting 4 hosts for vCD along side the standard infra VMs, my original 16GB was pretty much tapped. I love the board & think my next addition will be a pair of GB NICs to isolate the traffic to/from my handy-dandy Synology DS1010+ (also totally wicked). Good luck with the new lab! I’m sure you’ll get more than you expected from it in the long run.
I would love to have 32 GB of RAM, but am afraid to even try RDIMMs as they are not supported per the manufacturer’s page. I just figured I’d get some small SSDs for host cache if RAM becomes an issue.
I should have gone with something like your DS1010 for the dual GbE ports, the 411 only has one port which I am disappointed with. Perhaps next revision of the lab.
Thanks for sharing your config!
I found the secret for 32GB memory in the X9SCM-F. CT102472BA160B from Crucial. It’s what I have running, granted I so far have only powered the hosts up enough to install ESXi on them, but it “sees” all 32GB.
Thanks for the part number. Unfortunately $160+ per stick is a bit pricey for my taste, compared to just using an SSD cache.
So 32 GB RDIMMS is working with X8SIL-F ?????
Wondering how this translates to a production vSphere + View 5 box? When i planned my View deployment, I was thinking I would use my older (3 years) servers for two branch offices but it seems i will have to put some cash into these and it seems like a bad investment for old servers. So basically what I presented to the board in terms of cash is now going to go up.
For a low number of desktops, this solution may be able to fulfill your needs. I only run a handful of desktops as part of the lab, along with a wide variety of other applications and services.
The host RAM seems a bit low (only 7-14 desktops per host depending on vRAM allocated) for any heavy consolidation ratios.
Only 5 desktops on each server build. The problem i am running into is the 16GB limit on ram because i have to run a file server, view server, panologic server (vdi clients) and then the 5 desktops with 1GB each (windows 7). By my calculations that is 14GB of ram. That is cutting it close! I would feel more comfortable with 24GB. My plan is to use local SSD for storage but the cost may prevent me from doing that. (for 1 TB it’s $3k)
I appreciate the responses. As a one man IT shop and not a huge budget, its hard to test everything before buying.
make sure you plan for the M$ desktop access licenses as well if you don’t have software assurance. Found out about that after purchasing our view licenses so still haven’t deployed. Also a one man shop for a non-profit and want to setup a small business offering using a supermicro platform and spla licensing and can’t due to the DAL requirement.
Thanks for the update!
Thanks for putting this together. You’ve saved me a bunch of work
Did you have to do anything special to get Directpath I/O to work? Mine says it is unsupported.
No problem, glad you found use from the post. Nothing special to get Direct I/O working, it was available after installing ESXi 5.0u1. I tried to keep the build as stock as possible from the perspective of settings – I didn’t make any modifications to the BIOS or other settings beyond IPMI IPs for out of band management.
Jeff, did you ever resolve this? I am having the same problem. I believe I am running ESXi 5.0u1 this is the iso image that I used: VMware-VMvisor-Installer-5.0.0.update01-623860.x86_64.iso
OK, finally resolved this DirectIO problem, you need to make sure that VT-d is enabled in BIOS, mine wasn’t set. I am liking this build the more I have been playing with it. Haven’t run any performance tests yet. Just wanted to get it working. Anybody have any suggestions for a good low cost gigabit switch?
I also like this build and it realy help me to select the right hardware to plan my whitebox.
In case of the not directly supported second network on the X9SCM-F card I like to move to the new avaiable MB X9SCM-IIF which comes with the Intel® 82574L, 2x Gigabit LAN ports.
Does someone have any experience with this new board?