Skip to content

Building ESXi 5 Whitebox Home Lab Servers

by Chris Wahl on Mar 13th, 2012 | 37,371 views
whitebox

I recently decided it was time to graduate into a more robust home lab environment, as I’ve been pushing the boundaries of what a single Dell T110 running ESXi 5 can do. I’m no longer satisfied with nesting virtual ESXi 5 servers like a set of Russian nesting dolls, although we all have to start somewhere. To that end, I have decided to go forth with some whitebox builds to upgrade the Wahl Network vSphere 5 home lab.

For those interested in doing the same, this post outlines the additions I am making along with the overall design that I am working towards.

I just had to use this nesting doll photo, do you blame me?

Selecting ESXi 5 Host Hardware

The biggest conflict when picking a platform for a host is memory. Server memory is expensive, desktop memory is cheap, and pretty much anything you buy from a vendor will charge a mint for memory. If it were just about buying sticks of RAM, I’d go with a desktop build. However, I noticed that desktop builds just lose out on so many features that would make life easier. Things like IPMI, VT-x / VT-d (variable support), ECC, internal ports, network interfaces, and so on. So, I went with a build that utilizes server parts.

Let me briefly state that I don’t feel there is a wrong or right answer to what you ultimately choose to build with. As with any design, identify your functional requirements, the nice-to-haves, the budget, and then go forth.

I met the infamous @RootWyrm (Phillip Jaenke) at an HP Cloud Tech Day event last year, who runs a tech website that contains the build list for a whitebox server called the Baby Dragon. One thing I learned about Phil is that he’s very passionate about server builds and really hates noise and heat. He updated the Baby Dragon build to version 2, which is where I based my parts list. I’ve made a few changes to suit my tastes, with the end result being (per server):

  • CPU: Intel Xeon E3-1230 “Sandy Bridge” – 3.2GHz, 4 Cores, 8 Threads, 8MB (Newegg)
  • Motherboard: Supermicro X9SCM-F – Intel C204, Dual GigE, IPMI w/Virtual Media, 2x SATA-3, 4x SATA-2 (Newegg)
  • RAM: 16GB (4 x 4GB) Kingston 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 (PC3 10600) Server Memory Model (Newegg)
  • Updated! RAM: 32GB (4 x 8GB) Kingston 240 PIN DDR3 SDRAM ECC Unbuffered 1600 (PC3 12800) Server Memory Model (Amazon)
  • Disk: Lexar Echo ZX 16GB (Newegg)
  • Case: LIAN LI PC-V351B Black Aluminum MicroATX Desktop Computer Case (Newegg)
  • Fans: 2 x Scythe SY1225SL12L 120mm “Slipstream” Case Fan (Newegg)
  • Power: Seasonic 400W 80 Plus Gold Fanless ATX12V/EPS12V Power Supply (Amazon)

Hey there, sexy, want to run some ESXi 5?

Cost per server (at time of writing) is about $850.

The end result is a small form factor box that will produce nearly no noise (the case fans are only 10.7 dBA @ 41CFM), has no spinning disk (again, less heat and power), and has a dedicated out of band management port with the ability to do virtual media. Each box also has a pair of GbE NICs.

Down side? It only has 16 GB of RAM – there is no financially viable option for 8GB sticks of ECC UDIMMs at this time. I could have bought SSD drives for local swap cache, and may do so in the future when the price of SSDs fall further.

Update 3/20/2012: Intel 82579LM Drivers

There does not seem to be a driver for the Intel 82579LM card at this time for ESXi 5.X. In the meantime, use the other port, which is an Intel 82574L, to install the hypervisor. You can then add a custom driver to enable the other port by following the instructions found in this thread:

Install your machine(s) with the vanilla ESXi 5.0 ISO.

Log on to the console (or via ssh) of one of the machines and install the vib file by using the following commands:

esxcli software acceptance set --level=CommunitySupported
esxcli software vib install -v http://files.v-front.de/net-e1001e-1.0.0.x86_64.vib

Reboot and configure all NICs.

Update 4/12/2012: VMware Site Survey (Fault Tolerance)

Per a request in the comments, I’ve run the VMware Site Survey report to verify that it results in compatibility with fault tolerance.

How About Some NAS?

I also decided to retire an old model whitebox tower that contained 6 x 300 GB SATA drives in favor of a Synology DS411. Ultimately, I heard a lot of good feedback on the Synology, and it supports SSD! Not much else to say beyond that. ;)

  • Enclosure: Synology DS411 Diskless System 4-bay NAS Server (Newegg)
  • SSD: 2 x Intel 320 Series SSDSA2CW120G310 2.5″ 120GB SATA II MLC Internal Solid State Drive (Newegg)
  • SATA: 2 x Seagate Barracuda Green ST2000DL003 2TB 5900 RPM 64MB Cache SATA 6.0Gb/s 3.5″ Internal Hard Drive (Newegg)

Cost for the NAS with drives (at time of writing) is about $950.

Because I can tap into SSDs, I went with the green 5900 RPM model to save heat and power. I analyzed my lab environment using Xangati and found that I peak out at about 50 IOPS in most cases.

An IOPS report from Xangati

Thoughts

I waffled for a long time on the storage, but ultimately the ability to use SSD won me over. I am confident that in about a year, SSD technology will be at a price point where putting 4 x 250 GB drives in a NAS box will be budget friendly. It would be neat to see a small form factor storage appliance come out from Tintri or Pure Storage as I enjoy both of their approaches to making flash sing. Wishful thinking? :)

You can also get another perspective on building with a whitebox from Robert Novak, who has a very in depth post on his build process using a Shuttle box.

98 Comments
  1. Sean Duffy permalink - Mar 13th, 2012

    Nice post Chris! I love reading about different Whitebox builds. I will also take this route in the near future as I am tired of running 2-3 x ESXi nested Hosts in Workstation with only 16GB of main PC host RAM to play with! I have been using SSD in my system for VM Datastores and must say it makes such a difference compared to my older SATA drives that were running datastores from various FreeNAS/OpenFiler/HP VSA VMs!

    • Chris permalink - Mar 13th, 2012

      Good to hear. :)

      If I didn’t need some space for my media server movies and such, I might have just gone with 4 SSD drives. It is a bit pricey at this juncture.

  2. @dboftlp permalink - Mar 13th, 2012

    Hey, nice write up. I built out my initial vLab on his initial Baby Dragon v1. and just purchased a memory upgrade to 32GB (and I’m impatiently waiting on Provantage to get it here!). At any rate, you mentioned the cost of RAM and I just have to put in my 2 cents. My initial build was for 16GB and when I went back and looked, the 32GB upgrade cost me about $60 less! I jumped on it and I’m thinking the single 32GB host is going to be my Inception box for some time (vCloud design in hand!). I’m also rocking a Synology DS1010+ I picked up open box from Newegg for an absolute steal and have been so impressed with it, it’s seriously a killer NAS. You’ll love the SuperMicro boards, they are awesome!
    Enjoy!

    @dboftlp

    • Chris permalink - Mar 13th, 2012

      Thanks, I’m pretty jazzed about the new upgrades.

      This particular board states it only supports ECC UDIMMs. There really aren’t many choices online for unregistered /unbuffered (UDIMM) sticks that are also 8GB and ECC. The only few sticks I could find online were $130+ a stick. :(

  3. Keith permalink - Mar 13th, 2012

    Nice setup! I’m building out 2 x Shuttle SH67H3 systems w/ Core i7-2600, 16GB, USB Boot but still need to choose a NAS appliance. I also want to go back and upgrade them to 32GB… I’m currently running FREENAS in a VM via workstation. Synology, QNAP, DROBO…choices, choices..

    • Chris permalink - Mar 19th, 2012

      For what it’s worth, I have been very impressed with the features and performance of the Synology DS411. It’s very easy to set up and even has hardware acceleration when I mounted some iSCSI LUNs to the hosts.

  4. rsts11 permalink - Mar 18th, 2012

    I ended up going with the Sh67H3 because of the 32GB RAM (which ended up being fairly cheap, $60/8gb stick)… but it’s not ECC, and the remote management card is going to cost me an extra $100 probably. So yeah, server grade and desktop grade can have their handoffs. :)

    Looks like Komputerbay doesn’t have DDR3 ECC non-registered RAM, or I’d suggest giving them a try through Amazon. And best price I could find was $130ish too, a hardforum link.

    • Chris permalink - Mar 19th, 2012

      Yeah, the IPMI is absolutely essential. I hate having to set up a physical KVM or “crash cart” style connections. I assume that the 8GB ECC UDIMMs will eventually become popular and cheap, and I can upgrade then. Until then, a smallish SSD drive is very price competitive for using with Host Cache.

  5. benoit permalink - Mar 24th, 2012

    hi

    does the config work with FT and vt-d etc.

    thanks

    • Chris permalink - Mar 24th, 2012

      Indeed it does. Detailed screenshot on the review page.

      http://wahlnetwork.com/2012/03/24/review-supermicro-esxi-5-whitebox-home-lab-servers/

      • benoit permalink - Apr 12th, 2012

        hi again
        Sorry to distrub
        I asked many user to try this to see if there setting is working.

        Download the VMware SiteSurvey (http://www.vmware.com/download/shared_utilities.html ) utility to check if your configuration can run VMware FT.

        I knows yours is working , but i just want to knows if this utility is really working.

        thanks

      • Chris permalink - Apr 12th, 2012

        Sure. I ran the utility and everything responded with a green check mark. I’ll update the post with a picture.

      • benoit permalink - Apr 12th, 2012

        thanks

        you are relly helpful

        Just one last question , for the memory if i buy 8g stick is it possible to buy only three for 24gb or the only work in pair

        thanks again

      • Chris permalink - Apr 12th, 2012

        I’m not sure, the memory configurations should be available in the manual – http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCM-F.cfm

      • benoit permalink - Apr 17th, 2012

        Hi

        I finally decided , i will buy the same config as your.
        I am going to do it tomorrow, are there any thing i should know before buying or for the installation.

        Thanks a lot for your help

        Benoit

      • benoit permalink - Apr 17th, 2012

        Hi
        i have two other questions.
        What are the 2 fans for (2 x Scythe SY1225SL12L) ? there are already 3 in the Case.
        And i want to have hard drive to boot on it and put some vms , will the controler see the drive ??

        Thanks

      • Chris permalink - Apr 17th, 2012

        I bought the fans to replace the stock fans that come with the Lian Li case. They are very quiet and of higher quality. Optional.

        Your hard drive will be detected as long as you do not try to perform any sort of software RAID on the host.

  6. Arturo permalink - Mar 30th, 2012

    is there a particular reason why you didn’t buy 1X http://www.newegg.com/Product/Product.aspx?Item=N82E16820139077 and 1X of http://www.newegg.com/Product/Product.aspx?Item=N82E16820139268 to save $6 or is it just not recommended to mix and match.

    • Chris permalink - Mar 31st, 2012

      No reason that I can think of, I just typically buy memory in pairs or sets to ensure I get identical performance out of all sticks.

  7. jasemccarty permalink - Apr 5th, 2012

    Chris,

    Good setup. It is very similar to my setup: http://www.jasemccarty.com/blog/?p=1516.

    I chose the TYAN S5510GM3NR to overcome the multi-nic driver situation… Other than the nic difference, I would have likely picked the Super Micro board.

    Also have to agree with you that the IPMI/iKVM are essential to have, especially if you have to get into it remotely. I keep my lab at a Co-Lo site.

    Great info,
    Jase

    • Chris permalink - Apr 6th, 2012

      Hi Jase -

      Thanks for your comment! I actually recall reading your article way back when. I waffled on the motherboard for a while until I saw that driver post in the Hardforum on the Intel 82579LM, and then I went with it.

      To that end, I’ve also got a set of dual port NICs coming soon from a friend who doesn’t need them, which will bump my port count to 4 per host. Otherwise I was thinking of getting some more single port Intel 82574L cards (which I use in my Dell T110).

  8. Aaron permalink - Apr 16th, 2012

    Hello Chris,

    I am also doing a Baby Dragon II system, everything is exactly what you have used except that I went for this USB stick (http://www.newegg.com/Product/Product.aspx?Item=N82E16820235062) because my OCD just couldn’t handle the cap on the Lexar lying around indefinitely for the rest of its life. My build wont be as much for vmware experimentation, but more for home use. As such, I will only be building one server and it will be running a 2008R2 and WHS 2011 and a few other vms for media services. I included two Intel E1G42ETBLK cards (http://www.newegg.com/Product/Product.aspx?Item=N82E16833106034). One dedicated to iSCSI MPIO and the other for vm traffic. The onboard NIC is the WAN connection for my virtual pfSense machine. I will be booting the server up for the first time this evening.

    I am using a Synology DS1512+ which is amazing so far, very excited to see its iSCSI performance.

    Thanks again for the great write up!

    • Chris permalink - Apr 16th, 2012

      Hah, I hear you on the USB sticks. I ended up just pitching the caps. :) Sounds like a cool build, I’m jealous of the DS1512+

      I did later receive some HP NC360T cards (dual port Gb NIC) for all 3 hosts, but they were freebies.
      http://h18004.www1.hp.com/products/servers/networking/nc360t/index.html

      • rsts11 permalink - Apr 16th, 2012

        I used the Store-n-Stay in my Proliant Microserver… very similar low profile drive in my Shuttle. I have a NC380T that I wanted to use but the Shuttle wouldn’t boot with it. Not sure if it’s an x16 vs x4 thing I don’t quite understand… gonna try it in a different machine someday soon.

    • jasemccarty permalink - Apr 16th, 2012

      I looked at those cards… But couldn’t stomach $150 each for them. I went with 2 of these: http://www.newegg.com/Product/Product.aspx?Item=N82E16833106033 at $29 each. They work just as well.

      Cheers,
      Jase

      • Aaron permalink - Apr 16th, 2012

        I looked at those exact cards but a colleague told me he wasn’t able to get vmware to recognize them! I guess I’d better check more sources next time…

      • Chris permalink - Apr 16th, 2012

        They definitely work and appear as a “Intel Corporation 82574L” adapter. I have a pair of them in my Dell T110 lab server. I would have snagged a pair for each Supermicro server if I didn’t already have those HP cards sitting around.

      • Aaron permalink - Apr 16th, 2012

        Those HP cards were some nice freebies!

  9. Chris de Groot permalink - Apr 20th, 2012

    Thanks for all your posts and all included information. I finnally got the right information to buy a VMware Whitebox Home Lab. The update on “VMware Site Survey” is pretty nice. Can’t wait till all the parts arrive. I haven’t decided yet on the NAS/SAN to use. My main optie is a I3-530/8GB FreeNAS ZFS-based array of three 1,5TB 5400rpm disks. Got some advice?

    Cheers!

    Chris

    The Netherlands

  10. ITforMe permalink - May 8th, 2012

    FYI for those using this SuperMicro board. It seems to be pretty common that many add-in PCI NICs are not recognized at all by ESXi. This can be fixed by downgrading the BIOS from 1.1a to 1.0.c. I had this occur with a couple of dual port cards on this motherboard, and after performing the step that worked for others, I can confirm that it worked for me as well.

  11. Norbs permalink - Jun 6th, 2012

    Great job, I just set up an Ivy Bridge i7 system using a MSI-Z77MA-G45 board. So far everything is working great and VT-D is supported.

  12. voom permalink - Jun 22nd, 2012

    @Norbs. Can you describe the parts you used for your Ivy Bridge I7 system? Please list.

    Great write up Chris.

  13. Sate permalink - Jun 27th, 2012

    Hi Guys,

    Help please: I’m planning to make a dual boot server (Win2008R2 + ESXi 5.0).

    I would like to have the 2tb internal drive as a data store for VMs.
    QUESTION:Can I open those VMs with the VMWare Workstation 8 when booting with the Windows 2008 server?

    If not, what other solution can I use to have access to those VM from ESXi and VMW Workstation 8 without having to convert them.

    Cheers in advance!!!
    scan I use the biggest disk to store VM and open them from the ESXi and from VMWare workstation?

    • Chris permalink - Jul 10th, 2012

      Sorry Sate. I’ve not worked with a dual boot server in this manner.

  14. ESX_Student permalink - Jul 26th, 2012

    Anyone know if all the PCI slots work properly with add-in cards with the latest 2.0a BIOS update? Looking at building a lab with this board and while I’d like to use Ivy bridge since it’s out also want the most compatibility and stability. Thanks.

  15. Mel permalink - Jul 28th, 2012

    Chris,
    Have you found a financially viable option for 8GB sticks of ECC UDIMMs yet?

    • Chris permalink - Jul 28th, 2012

      Kingston has sticks from about $85 online now (So about $350 to fill up a server). 4GB are still cruising at around $25-30 a stick.

  16. Angelo permalink - Aug 2nd, 2012

    Interesting enough I purchased everything you listed Chris (x2) and neither system will POST, no beeps nothing. (Looked around and I have seen that if you have an older BIOS version it may cause this issue).

    • Chris permalink - Aug 2nd, 2012

      That sucks. I didn’t do anything special to get mine working, just assembled and powered on. I do know one guy had issues when he plugged the fans into the 3to4 pin molex converters – make sure to plug the fans directly into the motherboard.

      • rsts11 permalink - Aug 2nd, 2012

        I would doublecheck the motherboard attachment to the case for shorts, and try memory one stick at a time and/or testing the memory somewhere else. I had an issue with my Shuttle when one of my dimms was bad… it was silent and I feared I had a $900 paperweight on my desk. :( Tried one at a time and found the issue, and replaced the bad RAM.

        Be sure you’re building it initially with just board, cpu, memory. pop any NICs or disks just in case.

      • Angelo permalink - Aug 2nd, 2012

        rsts11. I tried it with no memory, one, both. still no POST. I also get no video single as if the board is short. So I bench built it and still same thing. Given that I just dropped 2k on all this I am NOT a happy camper right now.

        Only thing i noticed is the attachment screws for the case to mother board are some shiny metal. I always have used copper ones.

      • rsts11 permalink - Aug 2nd, 2012

        Did any of the pieces (probably the case) come with little silicone washers? It’s possible, not sure how probable, that you’re getting a short between the case/mount screws and the board. I’d try that first if possible.

        Disclaimer: I built a different config, but I have had either loose screws or improper touching :) cause this sort of issue in the past on other DIY systems.

      • ESX_Student permalink - Aug 2nd, 2012

        Sorry if this is overlooking the obvious as I saw that you said you ordered all the same parts but you did get Sandy Bridge and not Ivy Bridge CPU right? I think Ivy Bridge need the 2.0 BIOS or above. I’ve been wanting to build this setup but am hesitant because I would like to try to get the latest Ivy bridge CPU but have been seeing on other forums people having issues with the PCI cards with add-in NICs and having to downgrade to 1.0c which won’t work with Ivy bridge. Not a huge difference between Ivy and Sandy but would be nice to get the latest. Anyway, hope you get your setup working! :-)

      • Angelo permalink - Aug 2nd, 2012

        I got Sandy. Literary Chris has posted I clicked and said out to cart and check out. ONLY difference I did was I went with this ram. But still would not stop video or post beeps.
        http://www.newegg.com/Product/Product.aspx?Item=N82E16820139931

      • Chris permalink - Aug 2nd, 2012

        Actually that’s the problem there. That’s Registered memory (RDIMMs) and this board only takes Unregisterred memory (UDIMMs).

      • rsts11 permalink - Aug 2nd, 2012

        Yeah, what Chris said. Registered memory isn’t the same as ECC. :( Send it back and get unregistered ECC.

  17. Angelo permalink - Aug 2nd, 2012

    LOL Didn’t even catch that but the SOB should still post unless this board requires mem to boot.

    • ESX_Student permalink - Aug 2nd, 2012

      Nice, so looks like once you get the correct memory you should be all set. It will be interesting to see if yours shipped with the newer BIOS already on there and if you have any issues withe ESX seeing add-on NICs in certain PCI slots. Looks like from Newegg the last few people have gotten the newer Ivy supported BIOS shipped.

    • Paul T. permalink - Aug 23rd, 2012

      Angelo, I am having the same problem, maybe for the same reason. I am waiting for my UDIMM’s to replace the RDIMM’s that I ordered. Were you able to get this to work/

      • Chris permalink - Aug 23rd, 2012

        I can confirm that RDIMMs will not work with this setup; the system will fail to power on properly. UDIMMs must be used as per the vendor’s support matrix.

      • Paul T. permalink - Aug 24th, 2012

        That is my experience as well, hopefully I get the UDIMM’s today. However, shouldn’t I get a bunch of beeps, 5 short and 1 long if there is no memory installed on this MBD?

      • Paul T. permalink - Aug 24th, 2012

        Yup, it was RAM, and I guess it doesn’t beep with no RAM installed, but that’s ok. Got the RAM today: 32Gb Kingston KST KVR13E9K2/16I R. Booted right up, ESXi installed without a hitch and I have my first VM running now. Great little box so far.

      • Markus permalink - Aug 30th, 2012

        Had the exact same issue, I was baffled since there was no beeps or blinking LEDs. I was about to RMA the motherboard when it struck me that I had ordered Registered DIMMs. It’s been way to long since I built a box and didn’t order a Dell :)

  18. Mattia permalink - Aug 8th, 2012

    Very nice build! Does anyone know if the motherboard raid controller is supported in esxi?

    • Chris permalink - Aug 8th, 2012

      Software RAID controllers are not supported by the hypervisor. You’d need to pick up a hardware controller card.

      • Mattia permalink - Aug 8th, 2012

        Mh, that’s unfortunate. Thank you anyway :)

  19. Alex permalink - Aug 25th, 2012

    Hi, congratulation, your whitebox mech is very interesting, i’m ready to build the same system, but i have a question for you.

    I’ve downloaded the mainboard X9SCM-F user-manual, and it says about networking:

    “One (1) Intel 82574L Gigabit Ethernet controller for
    LAN 1, and one (1) Intel 82579 PHY for LAN 2″ , is then the lan 2 supported from esxi5?

    Thank you!

  20. ESX_Student permalink - Sep 6th, 2012

    Thought I would share this. Looks like Supermicro has a updated X9SCM-IIF Micro-ATX board that comes with the two out of the box compatible Intel® 82574L NICs and should come with the newer Ivy CPU supported BIOS.

  21. Richard Robbins permalink - Sep 8th, 2012

    Do you recommend connecting the Scythe fans to fan connectors on the motherboard or do you prefer to wire them to directly to the power supply? The fans are three pins fans so they won’t be controlled by the motherboard in any event.

    • Chris Wahl permalink - Sep 8th, 2012

      I’d connect them directly to the motherboard. I recall someone else in the comments had issues when they used the PSU. My guess is that the board expects to see draw on the CPU and/or MB1 connectors.

      • Richard Robbins permalink - Sep 8th, 2012

        Thanks. My incarnation of your system came together really nicely. Has there been any update on getting the second NIC to show up beyond your March 20, 2012 update? I’m ready to tackle that problem.

      • Chris Wahl permalink - Sep 8th, 2012

        The other NIC works if you use the driver linked.

      • Richard Robbins permalink - Sep 8th, 2012

        I’m new to vmware and am a bit lost. Is this a driver patch that I make to my host or is it one that I need to apply to each machine I create? How do I get to the command line interface?

      • Chris Wahl permalink - Sep 8th, 2012

        The step by step instructions are included in my post and in the link provided. You will need to follow the steps for each physical ESXi host. To reach the CLI, use the DCUI or SSH – this post should assist with that.

        http://wahlnetwork.com/2012/08/01/becoming-an-esxi-shell-ninja/

  22. Richard Robbins permalink - Sep 16th, 2012

    I’ve configured the ESXi host and coupled it with a ReadyNAS device that I have which supports iSCSI for the datastore — so my box has no internal storage and no optical drive. I’ve configured a virtual machine on the datastore but can’t figure out how to install an OS on the virtual machine. I’ve tried associating an iso file to the VM from vSphere client and I’ve tried associating an iso file with the virtual CD device on the ESXi host. I’m sure I’m missing something obvious but I’m stumped. I don’t want to break down an purchase an optical drive for the host just because I’m stuck. Can you point me in the right direction?

  23. bw permalink - Sep 21st, 2012

    hey Chris,

    really useful information here. i’m about to embark on building a “whitebox” . a bit new to this “whitebox” area. been using VMWare server 2.0 on top of win 2003 for years. time to update my homebased technology.

    i’m wondering as a general setup situation, these ESXi boxes contain the disk storage where the VM’s will reside or is storage external like on a qnap or synology box?

    if some one could enlighten me on this aspect that would be great. thx

    • Chris Wahl permalink - Sep 21st, 2012

      If you are only building one physical host for your lab, either one is fine. Once you start introducing more physical hosts, shared storage becomes more important (such as a Synology NAS) so that you can migrate the VMs around. This isn’t to say that you can’t use a virtual storage appliance, such as Nexenta, to share out the storage on one of your hosts. But, I tend to find that the performance of such a solution can be prohibitive due to the nature of the consumer hardware being used.

      If you plan to design a lab with multiple hosts, I’d encourage a NAS if budget allows.

      • bw permalink - Sep 21st, 2012

        i like that Synology solution. seems flexible for home use. appreciate your reply and help.

  24. keamas permalink - Feb 20th, 2013

    Hi great setup I think I will build kind of the same box.
    But how about a internal Raid Controller?
    Is there a good and cheap one which you can recommend for the setup?

    I don’t want to use a external NAS.

  25. sea pro permalink - Apr 11th, 2013

    Hello.
    I hope this post isn’t dead and others are still participating. I’ve been looking over Brian’s post and followed a link there to yours. Your configured system ‘fits’ into the smaller type of systems that I’d like to build and I’d like to thank everyone for their additional posts and tips. Now that this post is a few months old – can you update it with any additional ‘gotachas or patches’ that you’ve had to make. I have a synology 1812+ @ home that I’ll be using and wanted to build a couple of smaller case boxes to pound it with. I’m mostly wanting to have this as a lab, but also (to get work to help offset some costs) have the ability to test some DR for them. I was thinking of upping or maxing out the memory and need several more nic’s to get everything working and ‘similiar’ to my work enviroment. I’m also going to purchase the vmware essentials plus package and manage the systems that way. Before I use these parts, are there any ‘tweaks’ out there with newer boards or faster procs that I might could squeeze into this build – staying in a small for factor case..maybe for a few more $$ ?

    Any thoughts from anyone – and thanks again for all the great info.

    mark

    • Chris Wahl permalink - Apr 11th, 2013

      This post is actually over a year old now :) Not much else to say beyond a RAM upgrade to 32GB (which is noted in the hardware list). Still going strong in my lab: http://wahlnetwork.com/home-lab/my-home-lab/

      For more posts on the home lab, you can check out my “vSphere Lab” category here: http://wahlnetwork.com/category/tech-talks/vsphere-lab/

      I also have a Lab Resources page with a ton of other builds you can get ideas from: http://wahlnetwork.com/home-lab/resources/

      • sea pro permalink - Apr 13th, 2013

        I’ve started putting the pieces together and suddenly find that the fans aren’t available anymore. Any suggestions from anyone on some quiet fans for this box?
        Also, are you running the vmware o/s off of the ssd card, or a usb drive? I was kind of suprised that you hadn’t slipped in another pci-x nic already – but probably don’t need it depending on how your building/testing.

        thanks again -

      • Chris Wahl permalink - Apr 14th, 2013

        Any fan of similar size and dBa should be fine.

        The hosts are booting off a local USB stick. I’ve picked up NICs over the year and added them to the hosts – you can view the most current state of my lab on the “My Home Lab” section:

        http://wahlnetwork.com/home-lab/my-home-lab/

  26. sea pro permalink - Apr 17th, 2013

    Wow. Only took 2 days to get the approval for the balance of $$ to get the equipment. I’ll be ordering the the equipment over the next week or so – and can’t wait to get started. I’m going ahead with the 32Meg ram chips, and a few different fans (scythe SY1225SL12M). the reviews were good on them and even though they are a little more powerful, 1200cfm vs 850cfm – it should only help in keeping things cool, while staying pretty quiet. I like these nics, (Intel PWLA8492MT PRO/1000 MT PCI/PCI-X Dual Port Server) but as they are ‘full’ heigth – but low profile. As you’ve put the boxes together, will I have enough room for them? I’ve decided to get the vmware essentials package to use to play with too. :)

    mark

    • Chris Wahl permalink - Apr 18th, 2013

      Plenty of room in the Lian Li box for a full height card.

      • mark permalink - Apr 18th, 2013

        Thanks – I’m gonna order the equipment this week. I was worried about bios, nic’s, space etc. I haven’t ‘built’ something in about 6 years – but all indications seem to be pretty much plug and play. :) I like the way you are able to measure your power consumption. Do you mind sharing how you do that? Is it just some type of ‘inline’ device you bought – and took a cool cam pic of?

        mark

  27. mark permalink - Apr 18th, 2013

    dolt !!!!
    should have followed the link for the ups – as the pic is there..sorry :)

    • Chris Wahl permalink - Apr 20th, 2013

      No worries – glad you found it. It’s a great feature of the UPS.

  28. Kuntal Patel permalink - Apr 21st, 2013

    Hi Chris,
    Are you using the stock Intel CPU coolers? I am about to place the order for the hardware so I just wanted to confirm how the stock cooler was working out for you, assuming you are using it. Thanks

    • Chris Wahl permalink - Apr 21st, 2013

      Yup, used the stock cooler. No complaints.

  29. Dave permalink - Apr 29th, 2013

    Hey Chris does your SuperMicro boxes make siren noises when booting up? I noticed my 2 boxes make a siren noise when booting up one in ESXI its fine. I checked the temperatures and they are 32C and all fans are working. Was just curious if you had the same issue.

    • Chris Wahl permalink - Apr 30th, 2013

      Mine beep a few times on boot as it POSTs but no sirens. Make sure to connect the fans to the motherboard so that it use the RPM sensor; don’t use a molex connector to the PSU. I recall someone having issues that did that.

      • Dave permalink - Apr 30th, 2013

        Yeah I already did that. It looks the the sirens have stopped after leaving the system on for a few hours while doing a memory test. I think the CPU Thermal paste needed to warm up/ spread out or something.

        PS: On a side note today I picked up a QNAP TS-859 for $500! Now need to look into hard drives for it!

        Chris thanks again!

  30. Alex permalink - May 18th, 2013

    Hi Chris how it is going?

    Finally i’ve buy one of your system, with 16Gb of ram.
    The Seasonic is awesome! and also no NOISE

    Anyway I’m testing some datastore with NFS on FreeNAS, I’m build a Home-made-NAS based on a cool supermicro board.
    Ceck my blog for more info about the board.
    I’m very busy at the moment, cause i’m setting up my home-lab inside a ex-data center rack (it’s a gift ;-) a tremendous Sun-Rack 900-38.
    When done, picture on my blog ;-)

    PS: I’ve see that you have put some network power with 10Gb FCOE cards! good job (how much does it will cost? )
    see you soon

    • Chris Wahl permalink - May 18th, 2013

      Thanks, and yes it is super quiet – one of my favorite things about the PSU!

      I don’t know the Emulex card cost. I acquired them through a good friend who wanted to help me do some 10 GbE testing in my lab.

Trackbacks & Pingbacks

  1. rsts11: Building my compact VMware server at home « rsts11 – Robert Novak on system administration
  2. Review: Supermicro ESXi 5 Whitebox Home Lab Servers « Wahl Network
  3. Bulding an ESXi Home Server
  4. Three Labs for three reasons « A glimpse into the life of IT
  5. The All Flash Synology DS2411+ Array | Wahl Network
  6. Home labs - a scalable vSphere whitebox | www.vExperienced.co.uk
  7. Building new whitebox servers for VMware home lab - Virtualization Tips
  8. Vote For The Top Virtualization Blogs, Win A Cookie! via @ChrisWahl | Wahl Network
  9. WebbosWorld» Blog Archive » ESXi 5.1 Whitebox Build
  10. Maxing Out SuperMicro Whitebox Servers With 32 GB of RAM via @ChrisWahl | Wahl Network
  11. Building new whitebox servers for VMware home lab - Virtualization Tips | VirtualizeTips

Leave a Reply

Note: XHTML is allowed. Your email address will never be published.

Subscribe to this comment feed via RSS