Month: July 2014

Meet my new Storage Array

So the three of you who read this blog might be wondering why I haven’t been posting much lately.

Where’s Jeff, the cloud praxis guy & Hyper-V fanboy, who says IT pros should practice their cloud skills? you might have asked.

Well, I’ll tell you where I’ve been. One, I’ve been working my tail off at my new job where Cloud Praxis is Cloud Game Time, and two, the Child Partition, as adorable and fun as he is, is now 19 months old, and when he’s not gone down for a maintenance cycle in the crib, he’s running Parent Partition and Supervisor Module spouse ragged, consuming all CPU resources in the cluster. Wow that kid has some energy!

Yet despite that (or perhaps because of that), I found some time to re-think my storage strategy for the Daisetta Lab.

Recall that for months I’ve been running a ZFS array atop a simple NAS4Free instance, using the AMD-powered box as a multi-path iSCSI target for Cluster Shared Volumes. But continuing kernel-on-iscsi-target-service homicides, a desire to combine all my spare drives & resources into a new array, and a vacation-time cash-infusion following my exit from the last job lead me to build this for only about $600 all-in:

Software-defined. x86. File and block. Multipath. Intel. And some Supermicro. Storage utopia up in the Daisetta Lab
Software-defined. x86. File and block. Multipath. Intel. And some Supermicro. There’s some serious storage utopia up in the Daisetta Lab

Here are some superlatives and other interesting curios about this new box:

  • WP_20140705_01_19_31_ProIt was born on the 4th of July, just like ‘Merica and is as big, loud, ostentatious and overbearing as ‘Merica itself
  • I would name it ‘Merica.daisettalabs.net if the OS would accept it
  • It’s a real server. With a real Supermicro X10SAT server/workstation board. No more hacking Intel .inf files to get server-quality drivers
  • It has a real server SAS card, an LSI 9218i something or other with SAS-SATA breakout cables
  • It doesn’t make me choose between file or block storage, and is object-storage curious. It can even do NFS or SMB 3…at the same time.
  • It does ex post facto dedupe -the old model- rather than the new hot model of inline dedupe and/or compression, which makes me resent it, but only a little bit
  • It’s combining three storage chipsets -the LSI card, the Supermicro’s Intel C226, and ASMedia 1061- into one software-defined logical system. It’s abstracting all that hardware away using pools, similar to ZFS, but in a different, more sublime & elegant way.
  • It doesn’t have the ARC –ie RAM AS STORAGE– which makes me really resent it, but on the plus side, I’m only giving it 12GB of RAM and now have 16GB left for other uses.
  • It has 16 Disks : 12 rotational drives (6x1TB 5400 RPM & 6x2TB 7200RPM) and four SSDs (3x256GB Samsung 840 EVO & 1x128GB Samsung 830) and one boot drive (1x32GB SanDisk ReadyCache drive re-purposed as general SSD)
  • Total capacity RAW: nearly 19TB. Usable? I’ll let you know. Asking
    “Do I need that much?” is like asking “Does ‘Merica need to stretch from Sea to Shining Sea?” No I don’t, but yes ‘Merica does. But I had these drives in stock, as it were, so why not?
  • It uses so much energy & power that it has, in just a few days, erased any greenhouse gas savings I’ve made driving a hybrid for one year. Sorry Mother Earth, looks like I’m in your debt again
  • But seriously, under load, it’s hitting about 310 watts. At idle, 150w. Not bad all things considered. Haswell + full C states & PCIe power management work.
  • It’s built as veritable wind-tunnel as it lives the garage. In Southern California. And it’s summer. Under load, the CPU is hitting about 65C and the south-bridge flirts with 80c, but it’s stable.
  • It has six, yes, six, 1GbE Intel NICs. Two are on the motherboard, and I’m using a 4 port PCIe 2 card. And of course, I’ve enabled Jumbo Frames. I mean do you have to even ask at this point?
  • It uses virtual disks. Into which you can put other virtual disks. And even more virtual disks inside those virtual disks. It’s like Christopher Nolan designed this storage archetype while he wrote Inception…virtual disk within virtual disk within virtual disk. Sounds dangerous, but in the Daisetta Lab, Who Dares Wins!

So yeah. That’s what I’ve been up to. Geeking out a little bit like a gamer, but simultaneously taking the next step in my understanding, mastery & skilled manipulation of a critical next-gen storage technology I’ll be using at work soon.

Can you guess what that is?

Stay tuned. Full reveal & some benchmarks/thoughts tomorrow.

 

 

All the WANs are a stage

All the WANs are a Stage,

and all the packets and flows are players. 

They have their ingress and egress

from a vm here, through an F5 there, out the traffic shaper and then to the next hop

The Great Unknown, the Slash 8

Truly one packet in its time plays many routes

alas,  aggregate, balance or seek diverse routes

the packets do not

Into oblivion go the flows

when the WAN LED no longer glows

Let’s take a step together into a place unfamiliar and dark. A place that is, by all rights, strange and bewildering. A little place I like to think of as just one order of magnitude less rational than the Twilight Zone…a place few understand, and even fewer have mastered. A place just beyond my gateway, a place I really don’t care about except when I do, a place I like to call, the Wide Area Network.

That’s right. Let’s talk about the next hop. The land of BGP and OSPF and NAT and VPNs and QoS and CoS and DSCP and the “Goddamn ASA” and static routes and the “Goddamn firewall” all these words, phrases and acronyms you heard once, but dismissed as just so much babble out of the networking guy’s mouth, the one guy on your team who seems to age faster than all the others.

lacpHell, if it were up to you, Mr. Storage Networking Engineer, you’d do some LACP trunks or hook up MPIO up to that WAN and call it a day, amiright?  I mean what’s so complicated here? Of course links go down, that’s why teams (and virtual teams-of-teams!) are so cool!

But alas, all the world’s not a storage array, and all links to it are not teamed GigE interfaces with sub-millisecond latency.

And your business WAN, particularly the links to/from remote sites that comprise the RFC-1918d, encapsulated, virtual private wide-area network your typical mid-sized business with a large footprint depend on, fail far too often.

Or at least they have for me when I look back and survey the glories & wreckage of my 15 year IT career.

Verily I say unto you, the WAN is my White Whale, and I am an IT Ahab.

Here are some of the tools & techniques networking firms, engineers, architects and people way smarter than I have come up with to deal with the multiple pains of the WAN, followed by my snarky, yet honest, hurt, yet hopeful, lust-filled yet realistic view of them:

  • Multiprotocol layer switching (MPLS): The go-to solution for WAN pain, particularly for businesses that can’t/won’t employ a networking wonk equal to Mr. Ivan Pelpjnak. MPLS is a god-send for some firms, but it’s very costly. To really get value out of an MPLS strategy, you almost have to couple it with a session vritualization or in-datacenter-computing model (XenApp, RDS, VDI etc). Why? While MPLS makes the WAN as reliable and as accessible as your LAN, it doesn’t defeat latency. And latency is a hard thing to explain. Go on. Try it. On your spouse or significant other.
  • MPLS part two: And just so that I can get it off my chest…when the primary link at a branch site does go down, why do MPLS providers have such a hard time failing over to a secondary? I mean for real guys? Just keep the secondary WAN/VPN link up, or do something fancy with VRRP or VARP or something. Without a failover link, a downed-MPLS is worth less than a regular commodity internet circuit.
  • MPLS part three: In previous roles, I worried that maintenance of the MPLS became an end unto itself. I can see how this would happen, and I’ve been guilty of it myself; sometimes IT guys think in IP addresses, when they should have an eye to the future and think in FQDN, as the former is and shall forever be not routable, while the latter is the future. Underlining this point is the argument (well-supported in 2014, I think) that MPLS is, at best, a transitional technology. Build your business on it if you have to, but don’t tie anything to it, in other words. Sure it’s cloud-compatible, but so is dial up.
  • Inline Compression/dedupe: As a storage networking nerd, I Heart me some Riverbed and SilverPeak. But those are tools on the WAN that, in my experience, are just one CapEx ask too much. I’ve never actually used one of them. Love the idea, can never justify the cost. Open source alternatives? There’s really none (Except for this brave guy), speaking, perhaps, to how sophisticated and well-engineered these devices are, which justifies their cost but also makes them unobtainable for SMB shops.
  • Pertino and the like: I’ve been a fan of Pertino since I first started using this “Cloud VPN” product, which I likened more to a Layer 2 switch in the sky than a traditional VPN service. It’s some great tech; not clear that it can scale to 100s and 100s of users though. But very promising nonetheless, especially for really small but geographically-diverse environments.

  • It's just like Least Queue depth, you see, only ON YOUR WAN
    It’s just like Least Queue depth, you see, only ON YOUR WAN
  • Link aggregation + VPN all in one device: If you’re going to go hub & spoke because MPLS costs too much, or you can’t quite do full-cloud yet, this is a promising strategy, and one I’ll soon be testing out. I know I’ m not alone in the WAN-is-my-white-whale meme because companies like Peplink, Talari Networks, and even Cisco are still building products that address WAN problems. I have used Peplink before; was impressed, would use again, want one in my home with a second internet line, A+++++. The only thing that scuttled wider adoption in my last role was voice, a particularly difficult problem to sort out when you slap some good ol’ LACP-style magic onto your WAN ills. These devices, ranging from a few hundred bucks to several thousand, are almost too good to be true, as they tell the IT Pro that yes, he can have his cheap but rapidly-deploy-able commodity internet circuits aggregate into one, high speed, fault-tolerant link, and yes, that “unbreakable VPN” (as Peplink dubs it), can connect back to the HQ. Doesn’t defeat latency, true, but it sure makes the ASA look old-hat doesn’t it?
  • Cloud: The default winner, of course. But OpEx is hard to quantify. Sure, I guess I could up and move my datacenter assets to a CDN and let the network take care of the rest, or I could stand up a VM in a datacenter close to my users. But replication to on-prem assets/sources can be difficult, and, in some ways, in a really wide WAN, don’t we start worrying about version control, that what the New York branch is looking at is the same as the Seattle branch? Even so, I’m down with it, just need to fully comprehend it first.

What’s worked for you?