Practical SDN with Pertino

I don’t know if Software Defined Networking is a legitimate thing I should pursue, or just another mine in the IT requisition battlefield I need to be aware of. If it’s something I should pursue, what is the scope, budget, risks, and payoff? And what do I need to buy exactly? With x86 virtualization, it was clear (powerful pizza boxes! Lots of RAM!), with network virtualization…not so much.

I do know this much: the traditional, monolithic & inflexible business WAN causes me pain and suffering.

You know the type, or maybe in your career, you’ve moved on from such environments. But many of us haven’t. Remember this thing? The Business WAN:

You built this, now put it on the Internet where your users are.
You built this, now put it on the Internet where your users are.

Yeah baby. It’s still kicking after all these years…you get yourself some T1s for the branches, 10MegE/100MegE for the tentpole sites, some Cisco routers, OSPF & maybe MPLS to tie it all together with a safe, predictable RFC-1918 ipv4 address scheme and NAT on the ASA edge device. Active Directory is usually built on top with a replication topology matching the sites’ /24s.

And on the seventh day, the young engineer stood back and beheld what he had built, and said, Go forth IT department, and let thy services & value multiply: Exchange, Sharepoint, SMB shares, SANs, QOS policies, print servers, a Squid caching server here, a Peplink there, oh my!

This model is straight out of the 1990s playbook, but it’s still in wide-use. In the meantime, a crazy thing happened, the internet came along and for some inscrutable reason, it’s really popular, accessible and useful, and people like it. Your thought your advanced Business WAN was hot stuff, but your users feel it’s positively archaic because they have 20 megabits of bandwidth on their tiny 4G LTE phone, limitless storage & bandwidth via the Dropboxes & Googles of the world, and an internet that’s never under maintenance and never tells them no.

This then, is the problem I want SDN to solve: take the stuff my users need that’s on the business WAN and put it where my users are: on the internet. 443 doesn’t work for everything and while cloud is the ultimate home, I’m looking for baby-steps to the cloud, things I can do today with SDN that are low-risk and potentially high-reward. 

What do you do hotshot? 

Once upon a time in the Microsoft world, there was a thing called Direct Access. This was a software-defined solution that eased access to corporate resources for users on the internet. Without initiating a VPN connection, your C-level could access that stubborn decade-old UNC path from his laptop anywhere on the internet. IPV6 to the rescue!

But it was somewhat painful to install, especially in multi-domain scenarios, and sadly, only worked on Windows, which was great 10 years ago, but we’re not in a world where the PC is increasing in relevance; we’re in a world where the PC is less relevant by the day.

timthumbEnter Pertino, which to the cynical is yet another SDN startup from the Valley, but to me, is the among the first vendors wearing the badge SDN that actually knows my WAN pain and is building something SDN-related that is imminently practical and immediately applicable.

Pertino bills itself as a Cloud VPN provider, which, I think, doesn’t do it justice. VPN calls to mind dial-up…remote users connecting to your LAN. Pertino is sort of the opposite: this bit of tech allows you to extend your WAN/LAN into the cloud effortlessly.

I’m pretty jazzed on it because I think Pertino, or something like it, could be the next evolution in business WAN networking, especially if your organization is cloud-cautious. 

So What is it?

Pertino is essentially an ipv4 & ipv6 overlay technology employing NVGRE-like encapsulation dubbed “Overpass” that works with your existing on-prem equipment and extends securely your Layer 2/Layer 3 LAN assets to the places where your users are: the internet.

It’s so simple too. All you need is a modest 16 megabyte application that, to your users, will remain largely invisible. Once installed, Pertino sits quietly in the Windows system tray or in the background on Android and just generally stays out of the way, which makes it about 10x better than dial-up style VPNs of yesteryear.

While that application is low drama, what’s happening behind the scenes is some serious high stakes vKung-Fu, involving on-demand VMs, virtual switches, control & data planes and encapsulation.

On the Windows side, Pertino creates a simple virtual network interface, hooks onto your existing internet connection and begins a session with a Pertino virtual machine in a datacenter somewhere, in theory, close to your device.

All traffic on that vif is encapsulated via the NVGRE-like Overpass and your Windows client or Android handset is assigned both an ipv4 & ipv6 address. And just like that, you have what is in effect a fully switched LAN on the internet, to the point where an arp lookup sees other MAC addresses of other Pertino-enabled devices wherever they are.

Wait a sec...that MAC is at my house, 30 miles away.
Wait a sec…that MAC is at my house, 30 miles away.

Just think about that for a second. In years past, you’d have to call up your provider and order an exotic Virtual Private Wire Service to extend Layer 2 across a Layer 3 link if you wanted to expose your MAC addresses to the remote end.

Now I’m doing in effect the same thing with a simple Windows application. And I didn’t have to hire a consultant or mess around with NAT or the ASA, which is both comforting in that I like my security blanket, yet terrifying at the same time because it’s ipv6ing its way over my firewall. Pertino is essentially giving me a layer 2 switch….in the cloud.

In the Layer 3 space, your ipv4 address isn’t RFC-1918, but it’s not routable either. You can have any ipv4 address you like as long as it’s 50.203.x.x. Pertino is using Overpass encapsulation here to isolate tenants & customers from each other, reminding me of the way Microsoft uses NVGRE in Hyper-V & Azure.

Always one hop to your Pertino devices on your cloud LAN. Crazy but practical SDN!
Always one hop to your Pertino devices on your cloud LAN. Crazy but practical SDN!

After installing Pertino, you’re going to look at the output of ipconfig or ifconfig and say, “Wait. They gave me an entire /24?” Indeed, it seems you have a /24 but it’s not routable or unique. Another Pertino customer is probably getting the same /24. That’s what’s cool about encapsulation.

On your ipv6 Pertino network, things are a little more hazy. I think ipv6 is also NVGRE-encapsulated and perhaps all customers share the same /64, but while I’m a willing conscript in Tom Hollingsworth’s army of ipv6 boosters, I’m not smart enough to speak to how it works with Pertino. I know this: I can’t ping that ipv6 address from a non-Pertino client, yet the address differs substantially from the one my Windows 8.1 & 2012 R2 clients assign my NIC, which I can’t ping either.

So whatever man. It’s ipv6 and it’s mysterious.

What can you do with this? 

Last fall I was hot on Pertino because I envisioned using it at scale in a modern business WAN: imagine being able to kill your expensive, continent-hopping MPLS network without having to revamp your entire infrastructure.

I’m not sure Pertino could do that, but still: the mind races.

Pertino supports this:

  • Active Directory joins 
  • AD replication
  • Bind your DNS server to it
  • As much as I hate to see it because I think it encourages bad behavior (printing), you can do print servers over this
My DC in AWS 3000 miles away
My DC in AWS 3000 miles away

I’ve been using Pertino in the Daisetta Lab and quietly at work for about five months. With it, I’ve done this:

  • Built an Domain Controller in AWS somewhere in Virginia, joined & promoted it as a DC with the DC in my Daisetta Lab
  • SMB (CIFS to the non Microsoft crowd) shares via common UNC paths
  • Remote desktop, ssh
  • Mounted LUN in remote datacenter via iSCSI and MS iSCSI Initiator and ran IOMETER over Pertino
    • Just kidding

Performance

So you’ve got your fancy Pertino adapters deployed to laptops, mobile phones, iPads, and certain strategic servers, you’re living the SDN dream with only a modest OpEx spend and no rip & replace and your users can finally access CIFS, Sharepoint, and other internal resources from whatever internet connection they have.

How’s this baby perform?

Couple of measurements:

[table]

Test Type, Details, Time, Subjective Feeling

SMB File Copy, Copied 104MB file from remote site, 10 minutes 3 seconds, Felt slow but was evening hours at home

SMB File copy, Copied 95MB of random files from remote site, 3 minutes 46 seconds, Felt much faster speed varied 400k to 1mb/s

Latency tests, Simple pings to remote pertino  clients, 90ms minimum/300ms max, Was variable but mostly similar to standard VPN perofrmance

RDP, 2560×1440 remote desktop session, Session connected in 10s or so, Better that expected artifacting and compression minor

[/table]

Not bad
Not bad

There’s room for improvement here, but I’m on the free tier of Pertino service. The company offers some enhancements for paying customers, but Pertino’s not something you deploy as Tier 1 infrastructure; this is better used to fill gaps between your infrastructure and your users, and as such, I think the performance is acceptable.

It’s at least as fast as what my users call the “stupid VPN,” so what’s not to like?

Reliability

I’ve been using Pertino now for almost five months. I’d give them an A- for reliability.

I’ve been trying to push this review out for months, but it’s so easy to forget Pertino’s there. 99.9% of hte time, it’s invisible and just running in the background, connecting me seamlessly to whatever remote device I need to access.

There have been only two times the network failed me. Once, briefly in January I couldn’t RDP into the home network, and then, last week, there was an hours-long outage affecting many Pertino customers.

Credit to Pertino here: the same day they blogged about the outage, its cause and promised to make it better. Essentially a Pertino datacenter went offline, which they can recover from, but the resulting failover process snowballed and a widespread outage resulted:

On the afternoon of April 1st, there was a network outage between a cluster of data plane v-switches and the control plane, which was located in a different cloud datacenter. The disruption was brief, but lasted long enough for the control plane to consider those v-switches at risk. So, the control plane began migrating customers to other v-switches.

However, due to a new bug in the data plane, too many messages were sent from the data plane v-switches to the control plane, increasingly loading it with requests.

 

The Pertino client on my Nexus 5
The Pertino client on my Nexus 5

It’s been so reliable that after carefully considering the options, I had no problem recommending Pertino to my own parent partition, dad, a radio engineer by trade & reluctant IT consultant, as he often needs to connect small, distant radio stations to each other over IP. Usually he purchases a Zywall appliance, connects the sites together via VPN and with LogMeIn, he can remotely support these small operations. 

Pertino is an obvious fit for scenarios like that as well, and it’s probably cheaper.

Cost

Back when I first was testing Pertino, they allowed you to install the package on up to three devices for free. It looks like that plan is gone now,

Pertino still offers a free account for IT pros to sink their teeth into: with it, you can add Peritno on up to three devices to see how this works.

After that, the base level pricing for Pertino is $29/month for up to 10 devices. It scales from there, but only modestly: enterprise packaging starts at 40+ devices and you have to contact them to get pricing.

One gets the feeling that perhaps this is aimed at really small SMB markets, but I’m not so sure. If you have 1500+ objects in Active Directory, you surely don’t need Pertino on all of them. Just certain strategic, edge-focused & secured ones: a Domain Controller & SMB server here, a few executive or important mobile user laptops there. You get the idea.

Up to 40 devices can be connected through Pertino for about $90 per month.

Conclusion

All in all, I’ve been pretty impressed with this kit. It’s at once a practical way to get your on-prem services out to your users, dip your toes into some ipv6 waters even if you’re not engineering it yourself, and leverage some real software defined networking (insert definition of that here) in a safe, low-risk way.

In fact, I think you should help me test it a bit more. If you’re a fellow tech blogger with a lab at home and you’re interested in this or suffer from WAN pains too, let’s link up: The Supervisor Module spouse isn’t interested in becoming a test user on my Pertino network, but if you are, shoot me an email . I can add you as a user on my Pertino network, you can join a VM to my Pertino switch, and we can have a sort of home lab Apollo-Soyuz moment and learn something together.

Epilogue

Crazy timing, but within minutes of me posting my review of Pertino’s CloudVPN tech yesterday, Scott Lowe, a well-known rockstar virtualization blogger weighed in with his views of Pertino on Twitter:

I hope I don’t appear to be a Pertino shill and no disrespect to Scott intended, but I don’t think there’s anything “simplistic” about what is in effect a layer 2 switch in the cloud.

Okay, let’s suppose it’s a dumb switch.

Still. A dumb switch…in the cloud is something much more than a dumb switch in your rack.

Maybe I’m just easily impressed and Scott should show me how simplistic it is by joining my Pertino network. 😀

Disclosure

When I first signed up for the 3 device free Pertino service in the fall, a customer agent reached out to me to see if I had good experience. I relayed my experiences, linked to this blog and Pertino executives took notice. They offered me the use of up to 9 devices fro free on my Pertino network if I would post an unedited & unreviewed blog about my experiences with the product. No other compensation was given and no Pertino employees viewd the content of this post prior to its publication. 

You can read my disclosure policy here.

Part 2: Wargaming Mass Storage migration with a 6509e & Hyper-V

As you’ll recall from part 1, much of my time at work lately has been consumed by planning, testing and executing mass Storage Live Migration of 65+ .vhdx files from our old filer (built by a company that rhymes with PetTap) & its end-of-life 7200 RPM shelves to our new hotness, a Nimble CS260.

Essentially I’ve been a sort of IT Moses, planning the Exodus of my .vhdxs out of harsh, intolerably slow conditions, through some hazards & risks (Storage Migration during production), and into the promised land of speed, 1-5ms latency (rather than 20ms+!!), and user happiness.

Now VMware guys have a ton of really awesome tools to vMotion their .vmdks around their vCenters & vSpheres and their ESXis and now their VSANS too. They can tune their NFS for ludicrous speed and their Distributed Switches, now with Extra Power LACP support, can speak CDP & even tell you what physical port the SFP+ 10GbE adapter is plugged into.

Do I sound envious? Green with it even? Well I am. 

In comparison,  I got me some System Center Virtual Machine Manager (2012 SP1), Microsoft Failover Cluster mmc snap-in, a 6509e with two awesome x6748 performance blades (February 2008’s centerpiece switch mod in the popular “Hot Switch Racks” pin-up calendar), Hyper-Vs converged fabric design, 8x1GbE LACP teams, Manage Engine’s NetFlow, boat-loads of enthusiasm and a git ‘r done attitude.

And this is what I’ve got to git done:

untitled
Filer Exodus, Nimble Promised Land

And it has to be done with zero downtime because we have a  24/6 operational tempo, and I like my Saturdays.

One of my main worries as I’ve tried to quarterback this transition has been the switch. Recall from part 1 how I’m oversubscribed to hell & back on my two 6748s:

6748fabrichyper-v

 

I fear the harsh judgment of my networking peers (You’re doing that with that?!?!) so let me just get it out there: yes, I’m essentially using my 6509 & these two blades as a bus for storage IO. In fact, iSCSI traffic accounts for about 90% of all traffic on the switch in any given 24 hour period:

You can choose any storage paradigm you like as long as it's iSCSI
You can choose any storage paradigm you like as long as it’s iSCSI

Perhaps I’m doing things with this switch & with iSCSI that no sane engineer would, but I have to say, this has proven to be pretty durable and adequate as far as performance goes. Would I like some refreshing multi-channel SMB 3 file storage, some relief from the block IO blues & Microsoft clustering headaches? Yes of course, but I’ve got to shepherd the VMs to their new home first.

And to do that, I’ve got to master what this switch is doing on an hour by hour basis as my users log in around the clock.

So I pulled some Netflow data together, cracked my knuckles, and got busy with Excel and Pivot tables.

I’m glad I went through this exercise because it changed the .vhdx parade route & timing. What I thought was the busiest part of my infrastructure’s day was wrong, by a large factor. Here’s 8 days worth of Netflow traffic on the iSCSI & Live Migration VLANs, averaged out. Few live migrations were made during this period:

Image 1
Some of the numbers aren’t scaling properly because it’s a pain in the ass to get Excel to properly display bytes, bits, and millions/billions of packets

What you see here are the three login storms (Times on the graph are MST, they start early down under) as my EU, North America & Australia/New Zealand users login to their session virtualization VMs or hit the production SQL databases or run their reports.

I always thought EU punched my stack the hardest; our offices there have as many employees as North America, but only one or two time zones rather than three in North America.

But EU & North America and Australia combined don’t hit my switch fabric as hard as I do. Yes, the monkey on my back is…me. Well, me & the DBA and his incurable devotion to SQL backups in the evening. My crutch is DPM.

I won’t go into too much detail here but this was pretty surprising. At times over the eight days, Netflow would record more than 1 billion packets traversing the switch in one evening hour; the peak “payload” was north of 1 terabyte of iSCSI bytes/hour on some days!

Now I’m not a networking guy (though I do love Wifi experimenting), but what I saw here concerned me, gave me pause. Between the switch blades, I’ve supposedly got a 40 Gigabit/s backplane, to my Supervisor 720 modules, but is that real 40Gbit/s or marketing 40Gbit/s?

The key question: Am I stressing this 6509e, or does it have more to give?

Show fabric utilization detail said I was consuming only 20% of the switch fabric during my exploratory storage migrations, and that was at peak. 4 gigabit/second per port group.

Is that all you got? the 6509e seemingly taunted me.

But oh my stars, better check the buffers:

buffers

 

ACK! One dropped buffer call or whatever you call it, 7 weeks ago, way before I had the Nimble in place. Still….that’s one drop too many for me.

Stop pushing me so hard, the 6509e pleaded with me.

So I did what any self-respecting & fearful admin would do: call TAC. Show me the way home TAC, get me out of the fix I’m in or at least sooth my worry. Tell me I have nothing to worry about, or tell me I need to buy a Supe 2T to do what I want to do, just give me some certainty!

A few show tech supports, one webex session and one telephonic call with a nice engineer from Costa Rica later, I had some certainty. The config was sound, the single buffer drop was concerning but wasn’t repeating, even when I thought I was stressing the switch.

And I didn’t need to buy a Supe 2T. 

On to the Exodus/.vhdx parade.

In all my fretting about the switch, I was forgetting one thing: the feeble filer is old and slow and can’t push that much IO to the Nimble in the first place.

As best I can figure it, I can do about five storage live migrations simultaneously. Beyond that, I fear that luns will drop on the filer.

To the Nimble, it’s no sweat:

Floor it!!
Floor it!!

Netflow’s view from the same period:

Image 12
Raw naked iSCSI-flavored packet aggression….I like it

Love it when a plan comes together! I should have this complete within a few days, then I can safely upgrade my filer.

Branch office surveillance in a box :Ubiquiti Aircam, Ubunutu Linux & Hyper-V

I pause today from migrating .vhdxs to this:

0217141307

 

and stressing this:

WS-C6509-E

to deploying six of these to a new small branch office:

aircam_3_pack_1

In my industry, we’re constantly standing up new branch offices, tearing down old ones, and so sometimes I have to take off the virtualization/storage guy hat and put on the project management, facilities & security hat, something I admit to enjoying.

And one of my focuses in this area is on rapid deployment of branch offices. I want to be able to deploy a branch office from an IT, security & infrastructure perspective as quickly as overnight and at or below budgeted cost. Tearing down branch offices is more leisurely, but building new ones? I aim for excellence; I want to be the Amazon Prime of branch office rollouts.

Lack of 802.3af PoE standard makes standards guy cry, but for the price, I'll tolerate and use a dongle
Lack of 802.3af PoE standard makes standards guy cry, but for the price, I’ll tolerate and use a dongle

So I’ve tried to templatize the branch office stack as much as possible. Ideally, I’d have a hardened, secure rolling 19″ 12 or 16u rack, complete with a 8 or 16 port switch (SG300 maybe?), patch panel, a Dell R210 II server, 16GB RAM, and 1 terabyte in RAID 1 as a Hyper-V host, a short-depth but sufficient capacity UPS, and a router of some type: it should have 4G LTE & 1Gbase-T as a WAN-connectivity option, VPN ability (to connect to our MPLS) and, ipv6 dreams aside for now, NAT ability, and, of course, the one thing that will never become virtualized or software-defined, a workgroup printer.

Give me that in a rolling rack, and I can drop-ship it anywhere in CONUS overnight. Boom, Instant Branch Office in a Box (structured cabling comes later).

But one of the things that’s gotten in the way of this dream (besides getting the $ spend ahead of time, which is also a big issue) has been provisioning camera security. We need to watch our valuables, but how?

Weather resistant I suppose though I've read the little door that covers this hatch can let moisture in
Weather resistant I suppose though I’ve read the little door that covers this hatch can let moisture in

Usually that means contracting with some slow-moving local security company, going through a lengthy scoping process, choosing between cheap CCTV & DVR vs ip cameras & DVR, then going through a separate structured cabling process, and finally, validating. Major pain, and can get pricey very quickly: the last office I built required six 720p non-IR cameras + IP DVR + Mobile access to camera feeds. Price:$10k, 1.5x the cost of all the equipment I purchased for the 12u rolling rack!!

Meanwhile, you’ve got the business’ stakeholders wondering why it’s all so complicated. At home, they can connect a $100 720p IP camera up to their wifi, and stream video of their son/dog/whatever to their iPhone while they’re out and about, all without hiring anyone. I know it’s not as hardened or reliable as a real security camera system, but in IT, if you’re explaining, you’re losing.

And they do have a point.

This is a space begging for some good old fashioned disruption, especially if you don’t want the montly OpEx for the security company & your goal is only to get adequate surveillance (Two big Ifs, I recognize).

Enter Ubiquti Networks, an unusual but interesting wireless company that targets enterprise, carrier and pro-sumers with some neat solutions (60GhZ point-to-point wifi for the win!). After selling the boss on the vision & showing him the security company quote, I was able to get approval for six Ubiquiti Networks Airvision cameras, a Dome camera all for about $850 off Amazon, via the magical procurement powers of the corporate credit card.

The potential for my pet Branch Office in a Box project is huge and the cost was low. Here’s the vision:

  • Cat5e structured cabling contractor can now hang my cameras and run Cat 5e to them, especially since I’m familiar with aperture & focal length characteristics of the cameras and can estimate location without being on site.
  • DVR unit is an Ubuntu virtual machine in Hyper-V 3, recording to local storage which is backed up off-site via normal processes (it’s just a *.vhdx afterall) . That alone is huge; it’s been very painful to off-site footage from proprietary DVR systems
  • Reserve IPs for cameras prior to deployment via MAC address and normal process
  • Simple affair to secure via HTTPS/ssh the Linux appliance, NAT it out to the internet, then send a URL for the Apple Store & Play Store Ubiquiti camera compatible software, of which there seem to be several

Fantastic. I mean all that’s missing from making BiB into something stupid-proof and ready today is fire & alarm systems (yes, I’ve looked at NEST but regulations made me run for traditional vendors).

Demerits on this package so far:

  • Feels a bit cheap but not complaining too much. However it won't survive an attack
    Feels a bit cheap but not complaining too much. However it won’t survive an attack

    The cameras feel a little cheap. They offer minimal weather-resistance but the plastic casing feels like it was recycled from a 1995 CRT monitor: this thing’s going to turn yellow & brittle

  • No vandal-resistance. Maybe I missed the sku for that add-on. May need to improvise here; these won’t survive a single lucky struck from a hoodlum and his Louisville Slugger
  • Passive POE: So much for standards right? These cameras, sadly, require passive PoE dongle-injectors. And no, 802.3af active PoE, the kind in your switch, won’t work. You need a dongle-injector.

Other than, color me impressed.

Out of the box, the cameras are set for DHCP, but if you reserve the MAC on your DHCP server, you can neatly provision them in your chosen range without going through the usual pain.

Building the Ubuntu virtual machine -our DIY IP cam DVR system- on the Hyper-V host couldn’t be simpler. I followed Willie Howe’s steps here and recorded a few Gifcam shots to show you how easy it was.

As far as the management interface and DVR system: well I’ll say it feels much more integrated, thoughtful and enterprise-level than any of the small IP DVR systems I’ve struggled with at branch offices to date.

airvision

The big question is on performance, reliability, and sensitivity to recording when there’s movement in the zones I need to be monitored. And whether the stakeholder at the remote office will like the app.

But so far, I have to say: I’m impressed. I just did in 90 minutes what would have taken a local security company contractor about 2 weeks to do at a cost about 90% less than they wanted from me.

That’s good value even if these cheap $99 cameras don’t last for more than a year or two.

Airvision https interface allows you to post a floorplan, schedule and manage cameras, and  set recordings settings.
Airvision https interface allows you to post a floorplan, schedule and manage cameras, and set recordings settings.

 

Wargamming a mass Storage Live Migration with a 6509e, part 1

Storage Live Migration is something we Hyper-V guys only got in Server 2012 and it was one of the features I wanted the most after watching jealously as VMware guys storage vMotion .vhdks since George Bush was in office (or was it Clinton?).

I use Live Migration all the time during maintenance cycles and such, but pushing .vhdx hard drives around is more of a rare event for me.

Until now. See, I’ve got a new, moderately-performing array, a Nimble CS260 + an EL-125 add-on SAS shelf. It’s the same Nimble I abused in my popular January bakeoff blog post, and I’m thrilled to finally have some decent hybrid storage up in my datacenter.

WS-C6509-E
Big Iron Switching baby

However before I can push the button or press enter at the end of a cmdlet and begin the .vhdx parade from the land of slow to the promised land of speed, I’ve gotta worry about my switch.

You see, I’ve got another dinosaur in the rack just below the Nimble: a Cisco 6509e with three awful desktop-class blades, two Sup-720 mods in layer 3 and with HSRP, and then, the crown jewels of the fleet: two WS-X6748-GE-TX, where all my Hyper-V hosts & two SAN arrays are plugged in, each of them with two port-groups each with 20Gb/s fabric capacity.

Ahhh, the 6509: love it or hate it, you can’t help but respect it. I love it because it’s everything the fancy-pants switches of today are not: huge, heavy, with shitty cable management, extremely expensive to maintain (TAC…why do you hurt me even as I love you?), and hungry for your datacenter’s amperage.

I mean look at this thing, it gobbles amps like my filer gobbles spindles for RAID-DP:

show power cisco

325 watts per 6748, or, just about 12 watts less than my entire home lab consumes with four PCs, two switches, and a pfSense box. The X6748s are like a big block V8 belching out smoke & dripping oil in an age of Teslas & Priuses…just to get these blades into the chassis forced me to buy a 220v circuit & to achieve PSU redundancy required heavy & loud 3,000 watt supplies.

The efficiency nerd in me despises this switch for its cost & its Rush Limbaugh attitude toward the environment, yet I love it because even though it’s seven or eight years old, it’s only just now (perhaps) hitting the downward slope on my cost/performance bell curve. Even with those spendy power supplies and with increasing TAC costs, it still gives me enough performance thanks to this design & Hyper-V’s converged switching model:

6748fabrichyper-v
Errr sorry for the colors. The Visio looks much better but I had to create a diagram that even networking people could understand

Now MIke Laverick, all-star VMware blogger & employee, has had a great series of posts lately on switching and virtualization. I suggest you download them to your brain stat if you’re a VMware shop; especiallythe ones enabling netflow on your vSwitch & installing a vApp Scrutinizer, the new distributed switch features offered in ESXi 5.5 and migrating from standard to distributed switches. Great stuff there.

But if you’re at all interested in Hyper-V and/or haven’t gone to 10/40Gig yet and want to wring some more out of your old 5e patch cables, Hyper-V’s converged switching model is a damned fine option. Essentially a Hyper-V converged switch is a L2/L3 virtual switch fabricated on top of a Microsoft multiplexor driver that does GigE NIC teaming on the parent partition of your Hyper-V host.

This is something of a cross between a physical and logical diagram and it’s a bit silly and cartoonish, but a fair representation of the setup:

converged fabric
The red highlight is where the magic happens

So this is the setup I’ve adopted in all my Hyper-V instances….it’s the setup that changed the way we do things in Hyper-V 3.0, it’s the setup that allows you to add/subtract physical NIC adapters or shutdown Cisco interfaces on the fly, without any effect on the vNICs on the host or the vNICs on the guest. It is one of the chief drivers for my continuing love affair with LACP, but you don’t need an LACP-capable switch to use this model; that’s what’s great about the multiplexor driver.

It’s awesome and durable and scalable and, oh yeah, you can make it run like a Supercharged V-6. This setup is tunable!

Distributed Switching & Enhanced LACP got nothing on converged Hyper-V switching, and that is all the smack I shall talk.

Now sharp readers will notice two things: #1 I’ve oversubscribed the 6748 blades (the white spaces on the switch diagram are standard virtual switches, iSCSI HBAs for host/guests and these switches function just like the unsexy Standard switch in ESXi) and #2 just because you team it doesn’t mean you can magically make 8 1GbE prots into a single 8 Gb interface.

Correct on both counts, which is why I have to at least give the beastly old 6509 some consideration. It’s only got 20Gb/s of fabric bandwidth per 24 port port-group. Best to test before I move my .vhdxs.

In part 2, I’ll show you in detail some of those tests. In the meantime, here’s some of my Netflows & results from some tests I”m running ahead of moves this weekend.

Hitting nearly 2gb/s on each of the Nimble iSCSI vEthernets
Hitting nearly 2gb/s on each of the Nimble iSCSI vEthernets

 

 

What? These 6748 have been holding out on me and still have 80% left of its fabric to give me. So give it. I will not settle for 20% I want at least 50% utilization to make the moves fast and smooth. How to get there?
What? These 6748 have been holding out on me and still have 80% left of its fabric to give me. So give it. I will not settle for 20% I want at least 50% utilization to make the moves fast and smooth. How to get there?

 

My Netflow's not as sexy as Scrutinizer, but the spikey thing on the right shows one of my move/stress tests went way past the 95th percentile. More fun this weekend!
My Netflow’s not as sexy as Scrutinizer, but the spikey thing on the right shows one of my move/stress tests went way past the 95th percentile. More fun this weekend!