Welcome to my first Labworks post in which I test, build & validate a ZFS storage solution for my home Hyper-V & VMware lab.
Be sure to check out the followup lab posts on this same topic in the table below!
[table]
Labworks Chapter, Section, Subject, Title & URL
Labworks 1:, 1, Storage, Building a Durable and Performance-Oriented ZFS Box for Hyper-V & VMware
,2-3, Storage, I Heart the ARC & Let’s Pull Some Drives!
[/table]
Labworks #1: Building a durable, performance-oriented ZFS box for Hyper-V, VMware
Primary Goal: To build a durable and performance-oriented storage array using Sun’s fantastic, 128 bit, high-integrity Zetabyte File System for use with Lab Hyper-V CSVs & Windows clusters, VMware ESXi 5.5, other hypervisors,
Secondary Goal: Leverage consumer-grade SSDs to increase/multiply performance by using them as ZFS Intent Log (ZIL) write-cache and L2ARC read cache
Bonus: The Windows 7 PC in the living room that’s running Windows Media Center with CableCARD & HD Home Run was running out of DVR disk space and can’t record to SMB shares but can record to iSCSI LUNs.
Technologies used: iSCSI, MPIO, LACP, Jumbo Frames, IOMETER, SQLIO, ATTO, Robocopy, CrystalDiskMark, FreeBSD, NAS4Free, Windows Server 2012 R2, Hyper-V 3.0, Converged switch, VMware, standard switch, Cisco SG300
Schematic:
Hardware Notes:
[table]
System, Motherboard, Class, CPU, RAM, NIC, Hypervisor
Node-1, Asus Z87-K, Consumer, Haswell i-5, 24GB, 2x1GbE Intel I305, Hyper-V
Node-2, Biostar HZZMU3, Consumer, Ivy Bridge i-7, 24GB, 2x1GbE Broadcom BC5709C, Hyper-V
Node-3, MSI 760GM-P23, Consumer, AMD FX-6300, 16GB, 2x1GbE Intel i305, ESXi 5.5
san2, Gigabyte GA-F2A88XM-D3H, Consumer, AMD A8-5500, 24GB, 4x1GbE Broadcom BC5709C, NAS4Free
sw01, Cisco SG300-10 Port, Small Busines, n/a, n/a, 10x1GbE, n/a
[/table]
Array Setup:
I picked the Gigabyte board above because it’s got an outstanding eight SATA 6Gbit ports, all running on the native AMD A88x Bolton-D4 chipset, which, it turns out, isn’t supported well in Illumos (see Lab Notes below).
I added to that a cheap $20 Marve 9128se two port SATA 6gbit PCIe card, which hosts the boot volume & the SanDisk SSD.
[table]
Disk Type, Quantity, Size, Format, Speed, Function
WD Red 2.5″ with NASWARE, 6, 1TB, 4KB AF, SATA 3 5400RPM, Zpool Members
Samsung 840 EVO SSD, 1, 128GB, 512byte, 250MB/read, L2ARC Read Cache
SanDisk Ultra Plus II SSD, 1, 128GB, 512byte, 250MB/read & 250MB/write?, ZIL
Seagate 2.5″ Momentus, 1, 500GB, 512byte, 80MB/r/w, Boot/swap/system
[/table]
Performance Tests:
I’m not finished with all the benchmarking, which is notoriously difficult to get right, but here’s a taste. Expect a followup soon.
All shots below involved lzp2 compression on SAN2
SQLIO Short Test:
ATTO standard run:
NFS Tests:
None so far. From a VMware perspective, I want to rebuild the Standard switch as a distributed switch now that I’ve got a VCenter appliance running. But that’s not my priority at the moment.
Durability Tests:
Pulled two drives -the limit on RAIDZ2- under normal conditions. Put them back in, saw some alerts about the “administrator pulling drives” and the Zpool being in a degraded state. My CSVs remained online, however. Following a short zpool online command, both drives rejoined the pool and the degraded error went away.
Fun shots:
Because it’s not all about repeatable lab experiments. Here’s a Gifcam shot from Node-1 as it completely saturates both 2x1GbE Intel NICs:
and some pretty blinking lights from the six 2.5″ drives:
Lab notes & Lessons Learned:
First off, I’d like to buy a beer for the unknown technology enthusiast/lab guy who uttered these sage words of wisdom, which I failed to heed:
You buy cheap, you buy twice
Listen to that man, would you? Because going consumer, while tempting, is not smart. Learn from my mistakes: if you have to buy, buy server boards.
Secondly, I prefer NexentaStor to NAS4Free with ZFS, but like others, I worry about and have been stung by Open Solaris/Illumos hardware support. Most of that is my own fault, cf the note above, but still: does Illumos have a future? I’m hopeful, NextentaStor is going to appear at next month’s Storage Field Day 5, so that’s a good sign, and version 4.0 is due out anytime.
The Illumos/Nexenta command structure is much more intuitive to me than FreeBSD. In place of your favorite *nix commands, Nexenta employs some great, verb-noun show commands, and dtrace, the excellent diagnostic/performance tool included in Solaris is baked right into Nexenta. In NAS4Free/FreeBSD 9.1, you’ve got to add a few packages to get the equivalent stats for the ARC, L2ARC and ZFS, and adding dtrace involves a make & kernel modification, something I haven’t been brave enough to try yet.
Next: Jumbo Frames for the win. From Node-1, the desktop in my office, my Core i5-4670k CPU would regularly hit 35-50% utilization during my standard SQLIO benchmark before I configured jumbo frames from end-to-end. Now, after enabling Jumbo frames on the Intel NICs, the Hyper-V converged switch, the SG-300 and the ZFS box, utilization peaks at 15-20% during the same SQLIO test, and the benchmarks have show an increase as well. Unfortunately in FreeBSD world, adding jumbo frames is something you have to do on the interface & routing table, and it doesn’t persist across reboots for me, though that may be due to a driver issue on the Broadcom card.
The Western Digital 2.5″ drives aren’t stellar performers and they aren’t cheap, but boy are they quiet, well-built, and run cool, asking politely for only 1 watt under load. I’ve returned the hot, loud & failure prone HGST 3.5″ 2 TB drives I borrowed from work; it’s too hard to put them in a chassis that’s short-depth.
Lastly, ZFS’ adaptive replacement cache, which I’ve enthused over a lot in recent weeks, is quite the value & performance-multiplier. I’ve tested Windows Server 2012 R2 Storage Appliance’s tiered storage model, and while I was impressed with it’s responsiveness, ReFS, and ability to pool storage in interesting ways, nothing can compete with ZFS’ ARC model. It’s simply awesome; deceptively-simple, but awesome.
Lesson is that if you’re going to lose an entire box to storage in your lab, your chosen storage system better use every last ounce of that box, including its RAM, to serve storage up to you. 2012 R2 doesn’t, but I’m hopeful soon that it may (Update 1 perhaps?)
Here’s a cool screenshot from Nexenta, my last build before I re-did everything, showing ARC-hits following a cold boot of the array (top), and a few days later, when things are really cooking for my Hyper-V VMs stored, which are getting tagged with ZFS’ “Most Frequently Used” category and thus getting the benefit of fast RAM & L2ARC:
Next Steps:
- Find out why my writes suck so bad.
- Test Nas4Free’s NFS performance
- Test SMB 3.0 from a virtual machine inside the ZFS box
- Sell some stuff so I can buy a proper SLC SSD drive for the ZIL
- Re-build the rookie Standard Switch into a true Distributed Switch in ESXi
Links/Knowledge/Required Reading Used in this Post:
[table]
Resource, Author, Summary
Three Example Home Lab Storage Designs using SSDs and Spinning Disk, Chris Wahl, Good piece on different lab storage models
ZFS, Wikipedia, Great overview of ZFS history and features
Activity of the ZFS Arc, Brendan Gregg, Excellent overview of ZFS’ RAM-as-cache
Hybrid Storage Pool Performance, Brendan Gregg, Details ZFS performance
FreeBSD Jumbo Frames, NixCraft, Applying MTU correctly
Hyper-V vEthernet Jumbo Frames, Darryl Van der Peijl, Great little powershell script to keep you out of regedit
Nexenta Community Edition 3.1.5, NexentaStor, My personal preference for a Solaris-derived ZFS box
Nas4Free, Nas4Free.org, FreeBSD-based ZFS; works with more hardware
[/table]
Wilson.tech 
Hi Michael, thanks for the comment.
So, this post you commented on in 2015 is over a year old. Whatever mistakes there are in the post were done by a younger, more foolish and ignorant me. 🙂
That said:
For example, on your NAS4Free setup, did you use a LAGG for the two interfaces?
That array is not around anymore, but no, I did not. I never use a LAG or team the adapters on a storage system. What I had in that box was a 4x1GbE broadcom 5709C PCIE 2 card. 1xGbE was management network, the remaining three were separate /24s on discrete VLANs (switchport was access mode for those three)
Also, what is the purpose of the two iSCSI VLANs? It appears that you have two NAS devices serving up LUNs and two iSCSI consumers (Hyper-V and ESX). Is segmentation of storage providers / consumers part of the answer?
So here’s the thing. I like symmetry and I hate provisioning IP addresses. I prefer to automate and regulate IP provisioning in service of VM provisioning. So here’s how it works out in practice:
– Storage Array with 3 or 4 discrete IPs. Each subnet a /24 or smaller (separate broadcast domains)
– Virtual server host (VMware or Hyper-V, but the latter is preferred) with 2-6 teamed adapters & 3-4 virtual ethernet adapters on the host, tagged to appropriate VLAN.
I can automate that type of setup like no one’s business Michael. Host vServer on 10.92.250.0/24 for MGMT, 172.16.10.0/24, 172.16.11.0/24, 172.16.12.0/24 etc for storage, and the VMs just plug into a virtual switch that accepts their 802.1q tags.
Also, you mentioned getting a “proper” SSD for your ZIL drive. Is that a reference to using “low-capacity, low-latency devices capable of high IOPS”. How does your existing SSD not fulfill this?
Consumer-grade SSDs (MLC or TLC) aren’t the best for sequential or random writes. SSDs in general suck as write-cache targets, which is why so many SSD manufacturers have settled on wear-leveling algorithims to extend the life of their SSDs.
What you want for a ZIL is something purpose-built for acknowledging writes back to the OS. You want NVRAM, non-volatile memory, that’s backed up with 1) a fat capacitor, or 2) a battery.
Think of it this way: the ZIL records intent…the app/OS intends to write this data to disk, ZIL acknowledges OS intent, says ACK! and OS goes on writing more things, even if, in reality, the ZIL is acting more like a buffer than a disk.
In that scenario, SSDs as I was using in my lab at home, aren’t good for write-caching.
Cheers!
LikeLike
Great write-up Jeff. However, there appears to be some fundamental information about your setup missing. For example, on your NAS4Free setup, did you use a LAGG for the two interfaces? Also, what is the purpose of the two iSCSI VLANs? It appears that you have two NAS devices serving up LUNs and two iSCSI consumers (Hyper-V and ESX). Is segmentation of storage providers / consumers part of the answer?
Also, you mentioned getting a “proper” SSD for your ZIL drive. Is that a reference to using “low-capacity, low-latency devices capable of high IOPS”. How does your existing SSD not fulfill this?
Again, I appreciate your well-written post.
LikeLike