Just as I was wrapping up my time at my last employer, Nimble Storage delivered a great big Christmas gift, seemingly prepared just for me. It was a gift that brought a bit of joy to my blackened, wounded heart, which has suffered so much at the hands of storage vendors in years gone by.
What was this amazing gift that warmed my soul in the bleak, cold Southern California winter? Something called SMI-S, or Smizz as I think of it. SMI-S is an open standard management framework for storage. But before I get into that, some background.
You may recall Nimble Storage from such posts as “#StorageGlory at 30,000 IOPS,” and “Nimble Storage Review: 30 Days at Ludicrous Speed.” It’s fair to say I’m a fan of Nimble, having deployed two of their mid-level arrays this year into separate production datacenter environments I was responsible for as an employee, not as a consultant. From designing the storage network & virtualization components, to racking & stacking the Nimble, to entrusting it with my VMs, my SQL volumes, and Exchange, I got to see and experience the whole product, warts and all, and came away damned impressed with its time-to-deploy, its flexibility, snapshotting, and speed.
But one of the warts really stood out, festered, itched and nagged at me. While there has been support for VMware infrastructure inside a Nimble array since day one, there was no integration or support for Microsoft’s System Center Virtual Machine Manager, or VMM as us ‘softies call it. What’s a Hyper-V & System Center fanboy to do?
Enter SMI-S, the Storage Management Initiative – Specification,
a somewhat awkwardly-named but comprehensive storage management spec allowing you to provision/destroy volumes, create snapshots or clones, and classify your tiers via 3rd party tools, just the way $Deity intended it.
SMI-S is a product of the Storage Networking Industry Association and there’s a ton of in-depth, technical PDFs up on their site, but what you need to know is the specification has been maturing for a decade or longer, and it’s been adopted by a modest but growing number of storage vendors. The big blue N has it, for instance, as does HP and Hitachi Data Systems.
The neat thing about SMI-S is that it’s built atop yet another open management model, the Common Information Model, which, as MS engineers know, is baked right into Windows Server (both as a listener and provider).
And that has made all the difference.
I love SMI-S and CIM (as well as WBEM) because it’s a great example of agnostic computing theory working out to my benefit in practice. SMI-S and CIM are open-standards that save time, money & complexity, abstracting (in this case) the particulars of your storage array and giving you the freedom to purchase & manage multiple different arrays from one software interface, System Center via that other great agnostic system, https.
Or, to put it another way, SMI-S and CIM help keep your butt where it should be, in your chair, doing great IT engineering work, not in the CIO’s office meekly asking, “Please sir, may I have another storage system API license?”
Fantastic. No proprietary or secret or expensive API here, no extra licensing costs on the compute side, no new SKUs, no gotchas.*
And now Nimble Storage has it.
Nimble’s implementation of SMI-S is based on the Open Pegasus project**, the Linux/Unix world’s implementation of CIM/WBEM. All Nimble had to do to make me feel happy & warm inside was download the tarball, make it, and stuff it into NimbleOS version 2.2, which is the release candidate OS posted last week.
For IT organizations looking to reduce complexity & consolidate vendors, a Nimble Array that can be managed via System Center is a good play. For Nimble, that may only be a small slice of the market, but in that slice and among IT pros who focus on value-engineering just as much as they focus on convergence, System Center support enhances the Nimble story and puts them in league with the bigger, more established players, like the big blue N.
Which is just where they want to be, it appears.
Nimble’s on a roll and closing out 2014 strong, with fiber channel support, new all-flash shelves, faster models, a more mature OS (in fact, I believe it’s mostly re-written from the 1.4x days), stable DSMs for my Microsoft servers, and now, like icing on the cake, an agnostic standards-based management layer that plugs right into my System Center.
* Well, one gotcha. As the release notes say: “Note: SCVMM can only discover volumes that have the agent_type smis attribute.When logical units are created using SCVMM, the SMI-S provider ensures the agent_type smis attribute is added to the volumes. However, volumes created from the array do not automatically have the attribute.You must add the attribute when you create the volume; otherwise, SCVMM will not be able to discover it. For more information about the agent_type smis attribute, see Create a Starter Volume.” So existing volumes won’t show in your VMM but’s not too big of a headache as you can storage live migrate your VMs to volumes you’ve provisioned via VMM.
Also, as a footnote, I believe NetApp charges for SMI-S support.
** Open Pegasus is itself affiliated with the Open Group, an unsexy but in my view exciting & important IT standards organization that 1) is legit as the official certifying body of the UNIX trademark, 2) is not ITIL-affiliated as best I can tell and 3) aligns very well with Microsoft’s servers & systems. SMI-S is Ajust one piece of the puzzle; another is instrumentation & other infrastructure items. To that end, the Open Group oversees work being done on Open Management Infrastructure, which Microsoft supports and can utilize via WSMAN and wmi. Cisco, Arista and others are on board with this, and though I haven’t yet programmed a Nexus switch with Powershell yet, it is a real option and offers a compelling vision for infrastructurists like me: best-in-class storage, network, compute hardware, all managed & instrumented via System Center or whatever https front-end is suitable. Jeff Snover detailed the relationship over two years ago in this blog. *** Incidentally,without SMI-S & CIM, there’d be no way for me to build a simulation SAN in the Daisetta Lab (#StorageGlory Achieved : 30 Days on a Windows SAN) and management via VMM, but as I detailed earlier this summer, you can: stand up a Windows file server box, turn on the feature “Standards Based Storage Management,” point VMM at it and provision
My idea was to combine all types of disks -rotational 3.5″ & 2.5″ drives, SSDs, mSATAs, hell, I considered USB- into one tight, well-built storage box for my lab and home data needs. A sort of Storage Ark, if you will; all media types were welcome, but only if they came in twos (for mirroring & Parity sake, of course) and only if they rotated at exactly 7200 RPM and/or leveled their wears evenly across the silica.
And onto this unholy motley crue of hard disks I slapped a software architecture that promised to abstract all the typical storage driver, interface, and controller nonsense away, far, far away in fact, to a land where the storage can be mixed, the controllers diverse, and by virtue of the software-definition bits, network & hypervisor agnostic. In short, I wanted to build an agnostic #StorageGlory box in the Daisetta Lab.
Right. So what did I use to achieve this? ZFS and Zpools?
Hell no, that’s so January.
VSAN? Ha! I’m no Chris Wahl.
I used Windows, naturally.
That’s right. Windows. Server 2012 R2 to be specific, running Core + Infrastructure GUI with 8GB of RAM, and some 17TB of raw disk space available to it. And a little technique developed by the ace Microsoft server team called Tiered Storage Spaces.
Was a #StorageGlory Achievement Unlocked, or was it a dud?
Here’s my review after 30 days on my Windows SAN: san.daisettalabs.net.
It doesn’t make you pick a side in either storage or storage-networking: Do you like abstracted pools of storage, managed entirely by software? Put another way, do you hate your RAID controller and crush on your old-school NetApp filer, which seemingly could do everything but object storage?
When I say block, do you instinctively say file? Or vice-versa?
Well then my friend, have I got a storage system for your lab (and maybe production!) environment: Windows Storage Spaces (now with Tiering!) offers just about everything guys like you or me need in a storage system for lab & home media environments. I love it not just because it’s Microsoft, but also because it doesn’t make me choose between storage & storage-networking paradigms. It’s perhaps the ultimate agnostic storage technology, and I say that as someone who thinks about agnosticism and storage.
You know what I’m talking about. Maybe today, you’ll need some block storage for this VM or that particular job. Maybe you’re in a *nix state of mind and want to fiddle with NFS. Or perhaps you’re feeling bold & courageous and decide to try out VMware again, building some datastores on both iSCSI LUNs and NFS shares. Then again, maybe you want to see what SMB 3.0 3.0 is all about, the MS fanboys sure seem to be talking it up.
The point is this: I don’t care what your storage fancy is, but for lab-work (which makes for excellence in work-work) you need a storage platform that’s flexible and supportive of as many technologies as possible and is, hopefully, software-defined.
And that storage system is -hard to believe I’ll grant you- Windows Server 2012 R2.
I love storage and I can’t think of one other storage system -save for maybe NetApp- that let’s me do crazy things like store .vmdks inside of .vhdxs (oh the vIrony!), use SMB 3 multichannel over the same NICs I’m using for iSCSI traffic, create snapshots & clones just like big filers all while giving me the performance-multiplier benefits of SSDs and caching and a reasonable level of resiliency.
File this one under WackWackStorageGloryAchievedWindows boys and girls.
I can do it all with Storage Spaces in 2012 R2.
As I was thinking about how to write about Storage Spaces, I decided to make a chart, if only to help me keep it straight. It’s rough but maybe you’ll find it useful as you think about storage abstraction/virtualization tech:
It’s easy to build and supports your disks & controllers: This is a Microsoft product. Which means it’s easy to deploy & build for your average server guy. Mine’s running on a very skinny, re-re-purposed SanDisk Ready Cache SSD. With Windows 2012 R2 server running the Infrastructure Management GUI (no explorer.exe, just Server Manager + your favorite snap-ins), it’s using about 6GB of space on the boot drive.
And drivers for the Intel C226 SATA controller, the LSI 9218si SAS card, and the extra ASMedia 1061 controller were all installed automagically by Windows during the build.
The only other system that came close to being this easy to install -as a server product- was Oracle Solaris 11.2 Beta. It found, installed drivers for, and exposed all controllers & disks, so I was well on my way to going the ZFS route again, but figured I’d give Windows a chance this time around.
Nexenta 4, in contrast, never loaded past the Install Community Edition screen.
It’s improved a lot over 2012: Storage Spaces almost two years ago now, and I remember playing with it at work a bit. I found it to be a mind-f*** as it was a radically different approach to storage within the Windows server context.
I also found it to be slow, dreadfully slow even, and not very survivable. Though it did accept any disk disk I gave it, it didn’t exactly like it when I removed a USB drive during an extended write test. And it didn’t take the disk back at the conclusion of the test either.
Like everything else in Microsoft’s current generation, Storage Spaces in 2012 R2 is much better, more configurable, easier to monitor, and more tolerant of disk failures.
It also has something for the IOPS speedfreak inside all of us.
Tiered Storage Spaces & Adjustable write cache: Coming from ZFS & the Adaptive Replacement Cache, the ZFS Intent Log, the SLOG, and L2ARC, I was kind of hooked on the idea of using massive amounts of my ECC RAM to function as a sort of poor-mans NVRAM.
Windows can’t do that, but with Tiered Storage Spaces, you can at least drop a few SSDs in your array (in my case three x 256GB 840 EVO & one 128GB Samsung 830), mix them into your disk pool, and voila! Fast read-cache, with a Microsoft-flavored MRU/LFU algorithm of some type keeping your hottest data on the fastest disks and your old data on the cheep ‘n deep rotationals.
Which i naturally did while building this guy out. I mean, who wouldn’t want more write-cache?
But there’s a huge gotcha buried in the Technet and blogposts I found about this. I wanted to pool all my disks together into as large of a single virtual disk as possible, then pack iSCSI-connected .vhdxs, SMB 3 shares, and more inside that single, durable & tiered virtual disk.What I didn’t want was several virtual disks (it helped me to think of virtual disks as a sort of Aggregate) with SMB 3 shares and vhdx files stored haphazardly between them.
Which is what you get when you adjust the write-cache size. Recall that I have a capacity of about 17TB raw among all my disks. Building a storage pool, then a virtual disk with a 10GB write cache gave me a tiered virtual disk with a maximum size of about 965GB. More on that below.
It can be wicked fast, but so is RAID 0: Check out my standard SQLIO benchmark routine, which I run against all storage technologies that come my way. The 1.5 hour test is by no means comprehensive -and I’m not saying the IOPS counter is accurate at all (showing max values across all tests by the way)- but I like this test because it lets me kick the tires on my array, take her out for a spin, and see how she handles.
And with a “Simple” layout (no redundancy, probably equivalent to RAID 0), she handles pretty damn well, but even I’m not crazy enough to run tiered storage spaces in a simple layout config:
What’s odd is how poorly the array performed with 10GB of “Write Cache.” Not sure what happened here, but as you can see, latency spiked higher during the 10GB write cache write phase of the test than just about every other test segment.
Something to do with parity no doubt.
For my lab & home storage needs, I settled on a Mirror 2-way parity setup that gives me moderate performance with durability in mind, though not much as you’ll see below.
Making the most of my lab/home network and my NICs: Recall that I have six GbE NICs on this box. Two are built into the Supermicro board itself (Intel), and the other four come by way of a quad-port Intel I350-T4 server NIC.
Anytime you’re planning to do a Microsoft cluster in the 1GbE world, you need lots of NICs. It’s a bit of a crutch in some respects, especially in iSCSI. Typically you VLAN off each iSCSI NIC for your Hyper-V hosts and those NICs do one thing and one thing only: iSCSI, or Live Migration, or CSV etc. Feels wasteful.
But on my new storage box at home, I can use them for double-duty: iSCSI (or LM/CSV) as well as SMB 3. Yes!
Usually I turn off Client for Microsoft Networks (the SMB file sharing toggle in NIC properties) on each dedicated NIC (or vEthernet), but since I want my file cake & my block cake at the same time, I decided to turn SMB on on all iSCSI vEthernet adapters (from the physical & virtual hosts) and leave SMB on the iSCSI NICs on san.daisettalabs.net as well.
The end result? This:
[table caption=”Storage Networking-All of the Above Approach” width=”500″ colwidth=”20|100|50″ colalign=”left|left|center|left|right”]
1,MGMT,100,192.168.100.15,MGMT & SMB3
2,CLNT,102,192.168.102.15,Home net & SMB3
3,iSCSI-10,10,172.16.10.x,iSCSI & SMB3
4,iSCSI-11,10,172.16.11.x,iSCSI & SMB3
5,iSCSI-12,10,172.16.12.x,iSCSI & SMB3
That’s five, count ’em five NICs (or discrete channels, more specifically) I can use to fully soak in the goodness that is SMB 3 multichannel, with the cost of only a slightly unsettling epistemological question about whether iSCSI NICs are truly iSCSI if they’re doing file storage protocols.
Now SMB 3 is so transparent (on by default) you almost forget that you can configure it, but there’s quite a few ways to adjust file share performance. Aidan Finn argues for constraining SMB 3 to certain NICs, while Jose Barreto details how multichannel works on standalone physical NICs, a pair in a team, and multiple teams of NICs.
I haven’t decided which model to follow (though on san.daisettalabs.net, I’m not going to change anything or use Converged switching…it’s just storage), but SMB 3 is really exciting and it’s great that with Storage Spaces, you can have high performance file & block storage. I’ve hit 420MB/sec on synchronous file copies from san to host and back again. Outstanding!
I Finally got iSNS to work and it’s…meh: One nice thing about san.daisettalabs.net is that that’s all you need to know…the FQDN is now the resident iSCSI Name Server, meaning it’s all I need to set on an MS iSCSI Initiator. It’s a nice feature to have, but probably wasn’t worth the 30 minutes I spent getting it to work (hint: run set-wmiinstance before you run iSNS cmdlets in powershell!) as iSNS isn’t so great when you have…
SMI-S, which is awesome for Virtual Machine Manager fans: SMI-S, you’re thinking, what the the hell is that? Well, it’s a standardized framework for communicating block storage information between your storage array and whatever interface you use to manage & deploy resources on your array. Developed by no less an august body than the Storage Networking Industry Association (SNIA), it’s one of those “standards” that seem like a good idea, but you can’t find it much in the wild as it were. I’ve used SMI-S against a NetApp Filer (in the Classic DoT days, not sure if it works against cDoT) but your Nimbles, your Pures, and other new players in the market get the same funny look on their face when you ask them if they support SMI-S.
“Is that a vCenter thing?” they ask.
Microsoft, to its credit, does. Right on Windows Server. It’s a simple feature you install and two or three powershell commands later, you can point Virtual Machine Manager at it and voila! Provision, delete, resize, and classify iSCSI LUNS on your Windows SAN, just like the big boys do (probably) in Azure, only here, we’re totally enjoying the use of our corpulent.vhdx drives, whereas in Azure, for some reason, they’re still stuck on .vhds like rookies. Haha!
It’s a very stable storage platform for Microsoft Clustering: I’ve built a lot of Microsoft Hyper-V clusters. A lot. More than half a dozen in production, and probably three times that in dev or lab environments, so it’s like second nature to me. Stable storage & networking are not just important factors in Microsoft clusters, they are the only factors.
So how is it building out a Hyper-V cluster atop a Windows SAN? It’s the same, and different at the same time, but, unlike so many other cluster builds, I passed the validation test on the first attempt with green check marks everywhere. And weeks have gone by without a single error in the Failover Clustering snap-in; it’s great.
It’s expensive and seemingly not as redundant as other storage tech: When you build your storage pool out of offlined disks, your first choice is going to involve (just like other storage abstraction platforms) disk redundancy. Microsoft makes it simple, but doesn’t really tell you the cost of that redundancy until later in the process.
Recall that I have 17TB of raw storage on san.daisettalabs.net, organized as follows:
Disk Type, Quantity, Size, Format, Speed, Function
WD Red 2.5″ with NASWARE, 6, 1TB, 4KB AF, SATA 3 5400RPM, Cheep ‘n deep
Samsung 840 EVO SSD, 3, 256GB, 512byte, 250MB/read, Tiers not fears
Samsung 830 SSD, 1, 128GB, 512byte, 250MB/read, Tiers not fears
HGST 3.5″ Momentus, 6, 2TB, 512byte, 105MB/r/w, Cheep ‘n deep
Now, according to my trusty IOPS Excel calculator, if I were to use traditional RAID 5 or RAID 6 on that set of spinners, I’d get about 16.5TB usable in the former, 15TB usable in the latter (assuming RAID penalty of 5 & 6, respectively)
For much of the last year, I’ve been using ZFS & RAIDZ2 on the set of six WD Red 2.5″ drives. Those have a raw capacity of 6TB. In RAIDZ2 (roughly analogous to RAID 6), I recall getting about 4.2TB usable.
All in all, traditional RAID & ZFS’ RAIDZ cost me between 12% and 35% of my capacity respectively.
So how much does Windows Storage Spaces resiliency model (Mirrored, 2-way parity) cost me? A lot. We’re in RAID-DP territory here people:
Ack! With 17TB of raw storage, I get about 5.7TB usable, a cost of about 66%!
And for that, what kind of resieliency do I get?
I sure as hell can’t pull two disks simultaneously, as I did live during prod in my ZFS box. I can suffer the loss of only a single disk. And even then, other Windows bloggers point to some pain as the array tries to adjust.
Now, I’m not the brightest on RAID & parity and such, so perhaps there’s a more resilient, less costly way to use Storage Spaces with Tiering, but wow…this strikes me as a lot of wasted disk.
Not as easy to de-abstract the storage: When a disk array is under load, one of my favorite things to do is watch how the IO hits the physical elements in the array. Modern disk arrays make what your disks are doing abstract, almost invisible, but to truly understand how these things work, sometimes you just want the modern equivalent of lun stats.
In ZFS, I loved just letting gstat run, which showed me the load my IO was placing on the ARC, the L2ARC and finally, the disks. Awesome stuff:
As best as I can tell, there’s no live powershell equivalent to gstat for Storage Spaces. There are teases though; you can query your disks, get their SMART vitals, and more, but peeling away the onion layers and actually watching how Windows handles your IO would make Storage Spaces the total package.
So that’s about it: this is the best storage box I’ve built in the Daisetta Lab. No regrets going with Windows. The platform is mature, stable, offers very good performance, and decent resiliency, if at a high disk cost.
I’m so impressed I’ve checked my Windows SAN skepticism at the door and would run this in a production environment at a small/medium business (clustered, in the Scaled Out File Server role). Cost-wise, it’s a bargain. Check out this array: it’s the same exact Hardware a certain upstart Storage vendor I like (that rhymes with Gymbal Porridge) sells, but for a lot less!