Synology DS411 vSphere Home Lab Storage – Protocol Bakeoff
If you’re an owner of a Synology NAS array, you might find some value in this brief posting. I recently decided to do a bakeoff on my own DS411 array (a 4 bay enclosure) to find out which protocol reigned supreme when it came to presenting storage to my vSphere home lab: iSCSI or NFS?
So, I wiped off all data on the array completely and formatted it in two ways: as an iSCSI Single LUN (Block-Level) and as a single Volume with an NFS export. The Synology NAS box was populated with 4x 120 GB SSD drives in a RAID-5 configuration.
Benchmark Testing
In order to get some valid statistics out of the array, I fired up a VMware IO Analyzer and ran a series of random workloads. I’m not really interested in sequential workloads, as they are going to be rare in the grand scheme of things in my lab (or really with most any VMware workload out there, many VMs with IO != sequential).
- 4KB 100% Random 100% Read
- 4KB 100% Random 0% Read
- 512KB 0% Random 100% Read (Throughput Test)
Bring on the IOPS!
Performance
Here is a breakdown of results by transport protocol. Each test was ran for 5 minutes and repeated for accuracy.
iSCSI Statistics
I was honestly surprised with how badly the Synology handled the write workload over iSCSI. It appears that LIO_iblock process is the bottleneck, as it consumes about 70% CPU (with the remainder taken up by raid-5 process and other activities).
- 4KB 100% Random 100% Read – 5122 IOPS, 4 ms average latency
- 4KB 100% Random 0% Read - 681 IOPS, 28 ms average latency (Note: Appears to be a CPU bottleneck on array, high latency)
- 512KB 0% Random 100% Read (Throughput Test) – 98.14 MBps
Even the read IOPS weren’t that spectacular, considering there are 3 SSD drives that can feed read information down the pipe.
NFS Statistics
NFS ran like a champ. I saw 3 to 4 nfsd process threads spawn while doing the tests. Here’s a chart of the latency during a 100% random write test!
All of the tests did really well and made the SSD shine:
- 4KB 100% Random 100% Read – 8750 IOPS, 3 ms average latency
- 4KB 100% Random 0% Read - 4604 IOPS, 1 ms average latency
- 512KB 0% Random 100% Read (Throughput Test) – 109.23 MBps
Here’s a look at the IOmeter and Synology CPU Usage during the 100% random write test.
A healthy amount of IOPS for a 100% random write test!
The DS411 divides and conquers the NFS workload
Thoughts
I’ve always used NFS on the Synology DS411, and decided to give iSCSI a whirl. Because I have a rather low end model, I am limited on array resources (CPU and memory). While Memory is never an issue (it usually tickles down in the 10-20% range of used RAM) the CPU is quite taxed with iSCSI. NFS is the clear choice for presenting storage for VMware workloads.
Low-end NAS boxes are almost always CPU bound. That’s why I don’t fool around with SSD or RAID1/0 on them: not enough value for the cost uptick. If you’ve got some 7200RPM disk to try, that would make an interesting counterpoint, especially on the iSCSI test.
My original starting point was 4x 1TB 7.2K drives in RAID 1+0 over NFS. They performed rather miserably; I could only get about 4 or 5 VMs on them before I’d begin to notice a sluggishness. I now have 14 VMs running on the arrangement above and still have plenty of performance headroom left.
Such iops for a “lab” are quit nice
And the latency make me drewl .. Over here around 2-25 ms, with spikes upto 220 ms. Make me wanna upgrade my nexenta vsa (raidz1 3x 2TB 6G sata) with a ‘tier1′ pool of dedicated ssd’s
I went with the DS1512+ because it’s only slightly more expensive, yet has much higher build quality (metal rather than plastic enclosure and integrated power supply), with a faster dual-core CPU and upgradable memory. Plus you can expand it to 60TB! The three-year warranty doesn’t hurt either
I’m running iSCSi but will try NFS. Thanks for the research, Chris!
BTW, Chis, I followed your Lab recipe to the letter. Works a treat! I love the Supermicro’s built-in KVM, since I am running my lab in a colo. You’re super for sharing your design!
That’s great to hear. I have made it a point to have IPMI and remote console on all builds for the lab. Much easier to manage vs an IP KVM or KVM dongle switch.
Good data. I’m using a DS1512+ with iSCSI primarily. I may look at trying an NFS datastore.
I have resently bought a synology 812+ device to hold my VMs. I cant decide weather to use NFS or ISCSI. according to your write out i should obiously use NFS.
would you advice that i use NFS or would it be a different case for me as im using a different device.
thanks in advance
Ajay Paul
You’d have to perform a similar test on that model, as it has a newer and more powerful CPU than my DS411. I will comment that with a single NIC NAS model the multipathing advantages of iSCSI are not applicable, making NFS a bit more attractive for the lab. I use iSCSI on my DS2411+ to take full advantage of both NICs (it also has a powerful CPU).
many thanks for the reply, i will be conducting the similar tests this week some time.
thanks again.
Did you happen to try this on iSCSI (Regular files)? Just curious if that helps since you can enable VAAI with that option.
Not in this particular experiment, as I don’t believe that was available at the time (and possibly not for this specific model).
Hi again,
i dont get it, i just ran I/O tests on my Synology RS812+ NFS drive and i got the following results:
1.4KB 100% Random 100% Read – 1228 IOPS
2.4KB 100% Random 0% Read – 994 IOPS
3.512KB 0% Random 100% Read (Throughput Test) – 36 MBps
Considering i have a newer Synology then yours should i be experiencing at least similar results to yours?
I have my P.C attached to the synology RS812 via a 1GB network.
any help would be much appreciated. Thanks
This was on DSM 4.0 with 4 SSD drives. I suppose if you had a different make/model of SSD drives it may impact things, but your IOPS do sound pretty low if you are using SSDs. Make sure you have a single volume in RAID 5 with no other workloads going on, and are using the latest IO analyzer from VMware. Outside of that, watch the Resource Monitor in DSM and ESXTOP on the ESXi host to see where your bottleneck might be.
Many thanks for your reply, very helpfull.
I do indeed have SSD but am only using 2 drives JBOD.