Nothing Finer than a Well-Considered Powershell Module

Kudos to Intel  for recognizing & implementing a full Powershell module for their network adapters.

This is probably old news to most of you (and indeed, I think this was released in 2013) but I’ve just now managed to explore them.

How do I love them? Let me count the ways.

  1. With IntelNetCmdlets, you no longer have to fart around with netsh cmds to get your NICs primed to push packets properly
  2. With IntelNetCmdlets, your Network Engineering colleague in the cube next to you will no longer laugh as you suffer from Restless Finger Syndrome. RFS is characterized by furious mouse clicking interspersed with curses such as, “Goddamnit, I don’t have time to hunt through all these Device Manager menus just to input the Receive Buffer values I want! And I have four adapters! Somebody kill me. Now!”
  3. With IntelNetCmdlets, engineers who dabble in the virtual arts now have yet another tool in the box that can reduce/eliminate human error prior to the creation of an important virtual switch in a well-considered Hyper-V infrastructure.
  4. With IntelNetCmdlets, even your beater lab environment shines a little brighter because these babies work with my favorite NIC of all time, the  I350 T-4 quad port server adapter, which you can now buy brand new (Probably a Chinese knock-off…but the drivers work!) for about $70 on eBay. Suck on that Broadcom NetExtreme and goofy BroadcomCLI!

Here’s an example of what Intel’s Net cmdlets can do for you.

Let’s say you’re building out a host in your homelab, or you just received some new Whitebox x86 servers for a dev environment at work. Now, naturally this box is going to host virtual machines, and it’s likely those VMs will be on shared storage or will be resources in a new cluster…whatever the case, proper care & raising of your physical NICs at this stage in your infrastructure project not only sets you up for success and makes you a winner, but saves potentially hours or days of troubleshooting after you’ve abstracted all this nonsense away with your hypervisor.

Of course this could all be scripted out as part of a Config Mgr task sequence, but let’s not get too fancy here! I’m no MVP and I just want you to kill your need for Device Manager and the cryptic netsh commands, ok?

Gifcam demo time. Here I’m setting the Jumbo packet value in the Windows registry for the four Intel adapters on my I350-T4 card:

jumbopacket

What I love about this is that Intel’s gone the extra-mile with their Netcmdlets. There’s a full Powershell helpfile, with extras if you tag -verbose or -examples to the end of your get-help query. Any setting you need to toggle, it’s there, from “Green Ethernet” to how many RSS queues you want, to whether VMQ is enabled or disabled.

All you need? A quality Intel card (the Pro1000 cards prior to the I350 family don’t support this officially, but you may be able to trick the Proset drivers into it!), the Proset driver package utility (here) and Powershell. Hell, you can even do this while PS Remoting!

 

What are you going to do with all the time I Just saved you? Cheers

Integrating Trendnet IP Cameras with my homelab

What do you get when you take an IT Systems Engineer with more time on his hands than usual and an unfinished home project list that isn’t getting any shorter?

You get this:

Daytime
My home automation/Internet of Things ‘play’

That’s right. I’ve stood-up some IP surveillance infrastructure at my home, not because I’m a creepy Big Brother type with a God-Complex, rather:

  1. Once my 2.5 year old son figured out how to unlock the patio door and bolt outside, well, game over boys and girls….I needed some ‘insight’ and ‘visibility’ into the Child Partition’s whereabouts pronto and chasing him while he giggles is fun for only so long
  2. My home is exposed on three sides to suburban streets, and it’s nice to be able to see what’s going on outside
  3. I have creepy Big Brother tendences and/or God complex

I had rather simple rules for my home surveillance project:

  • IP cameras: ain’t no CCTV/600 lines of resolution here, I wanted IP so I could tie it into my enterprise home lab
  • Virtual DVR, not physical: Already have enough pieces of hardware with 16 cores, 128GB of RAM, and about 16TB of storage at home.
  • No Wifi, Ethernet only: Wifi from the camera itself was a non-starter for me because 1) while it makes getting video from the cameras easier, it limits where I can place them both from a power & signal strength perspective 2) Spectrum & bandwidth is limited & noisy at distance-friendly 2.4GhZ, wide & open at 5ghz, but 5 has half the range of 2.4. For those reasons, I went old-school: Cat5e, the Reliable Choice of Professionals Evereywhere
  • Active PoE: 802.3af as I already own about four PoE injectors and I’ve already run Cat5e all over the house
  • Endpoint agnostic:  In the IP camera space, it’s tough to find an agnostic camera system that will work on any end-device with as little friction as possible. ONVIF is, I suppose, the closest “standard” to that, and I don’t even know what it entails. But I know what I have: Samsung GS6, iPhone 6, a Windows Tiered Storage box, four Hyper-V hosts, System Center, an XBox One and 100 megabit internet connection.
  • Directional, no omni-PTZ required: I could have saved money on at least one corner of my house by buying a domed, movable PTZ camera rather than use 2 directionals, but 1) this needed to work on any end-point and PTZ controls often don’t

And so, over the course of a few months, I picked up four of these babies:

TV-IP310PI_d02_2

Trendnet TV-IP310PI

Design

I liked these cameras from the start. They’re housed in a nice, heavyweight steel enclosure, have a hood to shade the lens and just feel solid and sturdy. Trendnet markets them as outdoor cameras, and I found no reason to dispute that.

My one complaint about these cameras is the rather finicky mount. The camera can rotate and pivot within the mount’s attachment system, but you need to be careful here as an ethernet cable (inside of a shroud) runs through the mount. Twist & rotate your camera too much, and you may tear your cable apart.

And while the mount itself is steel and needs only three screws to attach, the interior mechanism that allows you to move the camera once mounted is cheaper. It’s hard to describe and I didn’t take any pictures as I was cursing up a storm when I realized I almost snapped the cable, so just know this: be cognizant that you should be gentle with this thing as you mount it and then as you adjust it. You only have to do that once, so take your time.

Imaging and Performance:

ircam
Nighttime

Trendnet says the camera’s sensor & processing is capable of pushing out 1080p at 30 frames per second, but once you get into one of these systems, you’ll notice it can also do 2560×1440, or QHD resolutions. Most of the time, images and video off the camera are buttery smooth, and it’s great.

I’m not sharp enough on video and sensors to comment on color quality, whether F 1.2 on a camera like this means the same as it would on a still DSLR, or understand IR Lux, so let me just say this: These cameras produce really sharp, detailed and wide-enough (70 degrees) images for me, day or night. Color seems right too; my lawn is various hues of brown & green thanks to the heat and California drought, and my son’s colorful playthings that are scattered all over do in indeed remind me of a clown’s vomit. And at night, I can see far enough thanks to ambient light. Trendnet claims 100 foot IR-assisted viewing at night. I see no reason to dispute that.

Let the camera geeks geek out on teh camera; this is an enterprise tech blog, and I’ve already talked abou the hardware, so let’s dig into the software-defined & networking bits that make this expensive project worthwhile.

Power & Networking

These cameras couldn’t be easier to connect and configure, once you’ve got the power & cabling sorted out. The camera features a 10/100 ethernet port; on all four of my cameras, that connects to four of Trendnet’s own PoE injectors. All PoE injectors are inside my home; I’d rather extend ethernet with power than put a fragile PoE device outside. The longest cable run is approximately 75′, well within the spec. Not much more to say here other than Trendnet claims the cameras will use 5 watts maximum, and that’s probably at night when the IR sensors are on.

From each injector, a data cable connects to a switch. In my lab, I’ve got two enterprise-level switches.

One camera, the garage/driveway camera, is plugged into trunked, native vlan 410 port on my 2960s in the garage,

The other switch is a small CIsco SG-300 10p. The three other cameras connect to it. The SG-300 serves the role of access-layer switch and has a 3x1GbE port-channel back to the 2960s. This switch wasn’t getting used enough in my living room, so I moved it to my home office, where all ports are now used. Here’s my home lab environment, updated with cameras:

The Homelab as it stands today
The Homelab as it stands today

Like any other IP cam, the Trendnet will obtain an IP off your DHCP server. Trendnet includes software with the camera that will help you find/provision the camera on your network, but I just saved a few minutes and looked in my DHCP table. As expected, the cameras all received a routable IP, DNS, NTP and other values from my DHCP.

Once I had the IP, it was off to the races:

  • Set DHCP reservation
  • Verify an A record was created DNS so I could refer to the cameras by names rather than IP
  • Login, configure new password, update firmware, rename camera, turn-off UPNP, turn-off telnet
  • Adjust camera views

Software bits – Server Side

Trendnet is nice enough to include a fairly robust and rebadged version of Luxriot camera software, which has two primary components: Trendnet View Pro (Fat Client & Server app) and VMS Broacast server, an http server. Trendnet View Pro is a server-like application that you can install on your PC to view, control, and edit all your cameras. I say server-like because this is the free-version of the software, and it has the following limits:

  • Cannot run as a Windows Service
  • An account must be logged in to ‘keep it running’
  • You can install View Pro on as many PCs as you like, but only one is licensed to receive streaming video at a time

Upgrading the free software to a version that supports more simultaneously viewers is steep: $315 to be exact.

Smoking the airwaves with my beater kiosk PC in the kitchen. This is the TrendNet View client, limited to one viewer at a time
Smoking the airwaves with my beater kiosk PC in the kitchen. This is the TrendNet View client, limited to one viewer at a time

Naturally, I went looking for an alternative, but after dicking around with Zoneminder & VLC for awhile (both of which work but aren’t viewable on the XBox), I settled on VMS Broadcast server, the http component of the free software.

Just like View Pro, VMS Broadcast won’t run as a service, but, well, sysinternals!

So after deliberating a bit, I said screw it, and stood-up a Windows 8.1 Pro VM on a node in the garage. The VM is Domain-joined, which the Trendnet software ignored or didn’t flag, and I’ve provisioned 2 cores & 2GB of RAM to serve, compress, and redistribute the streams using the Trendnet fat client server piece as well as the VMS web server.

Client Side

On that same Windows 8.1 VM, I’ve enabled DLNA-sharing on VLAN 410, which is my trusted wireless & wired internal network. The thinking here was that I could redistribute via DLNA the four camera feeds into something the XBox One would be able to show on our family’s single 48″ LCD TV in the living room via the Media App. So far, no luck getting that to work, though IE on the XBox One will view and play all four feeds from the Trendnet web server, which for the purposes of this project, was good enough for me.

Additionally, I have a junker Lenovo laptop (Ideapad, 11″) that I’ve essentially built into a Kiosk PC for the kitchen/dining area, the busiest part of the house. This PC automatically logs in, opens the fat client and loads the file to view the four live feeds. And it does this all over wifi, giving instant home intel to my wife, mother-in-law, and myself as we go about our day.

Finally, both the iOS & Android devices in my house can successfully view the camera streams, not from the server, but directly (and annoyingly) from the cameras themselves.

The Impact of RTSP 1080p/30fps x 4 on Home Lab 

I knew going into this that streaming live video from four quality cameras 24×7 would require some serious horsepower from my homelab, but I didn’t realize how much.

From the compute side of things, it was indeed alot. The Windows 8.1 VM is currently on Node2, a Xeon E3-1241v3 with 32GB of RAM.

Typically Node2’s physical CPU hovers around 8% utilization as it hosts about six VMs in total.

With the 8.1 VM serving up the streams as well as compressing them with a variable bit rate, the tax for this DIY Home surveillance project was steep: Node2’s CPU now averages 16% utilized, and I’ve seen it hit 30%. The VM itself is above 90% utilization.hosts

More utilization = more worries about thermal as Node2 sits in the garage. In southern California. In the summertime.

Ambient air temperature in my garage over the last three weeks.
Ambient air temperature in my garage over the last three weeks.

Node2’s average CPU temperature varies between 22c and 36c on any given warm day in the garage (ambient air is 21c – 36c). But with the 8.1 VM, Node2 has hit as high as 48c. Good thing I used some primo thermal paste!

trsp

All your Part 15 FCC Spectrum are belong to me, on channel 10 at least
All your Part 15 FCC Spectrum are belong to me, on channel 10 at least

From the network side, results have been interesting. First, my Meraki is a champ. The humble MR-18 802.11n access point doesn’t break a sweat streaming the broadcast feed from the VM to the Lenovo Kiosk laptop in the kitchen. Indeed, it sustains north of 21mb/s as this graph shows, without interrupting my mother in law’s consumption of TV broadcasts over wifi (separate SSID & VLAN, from the SiliconDust TV tuner), nor my wife’s Facebooking & Instagramming needs, nor my own tests with the Trendnet application which interfaces with the cameras directly.

Meraki’s analysis says that this makes the 2.4ghz spectrum in my area over 50% utilized, which probably frustrates my neighbors. Someday perhaps I’ll upgrade the laptop to a 5ghz radio.

vSwitch, the name of my Converged SCVMM switch, is showing anywhere from 2megabits to 20 megabits of Tx/Rx for the server VM. Pretty impressive performance for a software switch!network

Storage-wise, I love that the Trendnets can mount an SMB share, and I’ve been saving snapshots of movement to one of the SMB shares on my WindowsSAN box.

I am also using Trendnet’s email alerting feature to take snapshots and email them to me whenever there’s motion in a given area. Which is happening a lot now as my 2 year old walks up to the cameras, smiles and says “Say cheeeese!”

All in all, a tidy & fun sub-$1000 project!

The Value of Community Editions

I was excited to hear on the In Tech We Trust podcast this week that the godfather of all the hyperconverged things -Nutanix- may release a community edition of their infrastructure software this year.

That. Would. Be. Amazing.

I’ve crossed paths with Nutanix a few times in my career, but they’ve always remained just a bit out of reach in my various infrastructure projects. Getting some hands-on experience with the Google-inspired infrastructure system in my lab at home would be most excellent, not just for me, but for them, as I like to recommend product stacks I’ve touched above ones I haven’t.

Take Nexenta as an example. As Hans D. pointed out on the show, aside from downloading & running Oracle Solaris 12, Nexenta’s just about the only way one can experience a mature & enterprise-focused implementation of ZFS. I had a blast testing Nexenta out in my lab in 2014 and though I can’t say my posts on ZFS helped them move copies of NexentaStore, it surely didn’t hurt in my view.

VEEAM is also big in the community space, and though I’ve not tested their various products, I have used their awesome stencil collection.

Lest you think storage & hyperconvergence vendors are the only ones thinking ‘community, today my favorite yellow load balancer Kemp announced in effect a community edition of their L4/L7 Loadmaster vAppliance. Kemp holds a special place in the hearts of Hyper-V guys; as long as I can remember, yes even back in the dark days of 2008 R2, they’ve always released a Loadmaster that’s just about on-par with what they offer to VMware shops. In 2015 that support is paying off I think; Kemp’s best-in-class for Microsoft shops running Hyper-V or building out Azure, and with the announcement you can now stress a Kemp at home in your lab or in Azure with your MSDN sub. Excellent.

Speaking of Microsoft, I’d be remiss if I didn’t mention Visual Studio 2013, which got a community edition last fall.

I’d love to see more community editions, namely:

  • Nimble Storage: I’ve had a lot of success in the last 18 months racking/stacking Nimble arrays in environments with older, riskier storage. I must not be the only one;  the company recently celebrated its 5,000th customer. Yet, Nimble’s rapid evolution from storage startup with potential to serious storage player is somewhat bittersweet for me as I no longer work at the places I’ve installed Nimble arrays and can’t tinker with their rapidly-evolving features & support. Come on guys, just give me the CASL caching system in download form and let me evaluate your Fiber Channel support and test out your support for System Center
  • NetApp: A community release of Clustered Data OnTAP 8.2x would accomplish something few NetApp products have accomplished in the last few years: create some genuine excitement about the big blocky blue N. I’m certain they’ve got a software-only release in-house as they’ve already got an appliance for vSphere and I heard rumors about this from channel sources for years. So what are you waiting for NetApp? Let us build-out, support, and get excited about cDOT community-style since it’s been too hard to see past the 7-mode–>clustered mode transition pain in production.

On his Graybeards on Storage podcast, Howard Marks once reminisced about his time testing real enterprise technology products in a magazine’s tech lab. His observations became a column, printed on paper in an old-school pulp magazine which was shipped to readers. This was beneficial relationship for all.

Those days may be gone but thanks to scalable software infrastructure systems, the agnostic properties of x86, bloggers & community edition software, perhaps they’re back!

Hunting Lettered Drives in a Microsoft Enterprise

Of all the lazy, out-dated constructs still hanging around in computing,SMB shares mapped as drive letters to client PCs has to be the worst.

Microsoft Windows is the only operating system that still employs these stubborn, vestigal organs of 1980s computing. Why?

Search me. Backwards compatibility perhaps, but  really? It’s not like you can install programs to shares mapped as drive letters, block-storage style.

If you work in Microsoft-powered shops like me, then you’re all too familiar with lettered drive pains. Let’s review:

  1. Lettered drives are paradigms from another era: Back in the dial-up and 300 baud modem days you got in your car and drove to Babbages to purchase a big box on a shelf. The box contained floppy diskettes, which contained the program you wanted to use. You put the floppy in your computer and you knew instinctively to type a: on your PC. Several hours later after installing the full program to your C: drive, you took the floppy out of its drive and A: ceased to exist. If this sounds archaic to you (it is), then welcome to IT’s version of Back to the Future, wherein we deploy, manage and try to secure systems tied to this model
  2. Lettered drives are dangerous:  The Crytpo* malware viruses of the last two years have proven that lettered drives = file server attack vector. I have friends dealing with Gen 3 of this problem today; a drive map from one server to all client PCs must be a Russian crypto-criminal’s dream come true.
  3. Your Users Don’t Understand Absolute/Relative paths:  When users want to share a cat video from the internet, they copy + paste the URL into an email, press send, and joyous hilarity ensues. But anger, confusion, despair & Help Desk tickets result when those same users paste a relative path of G:FridayFunDebsFunnyCatVids into an email and press send. Guess what Deb? Not everyone in the world has a G: drive. This is frustrating for IT, and Deb doesn’t understand why they’re so mad when she opens a ticket.
  4. Lettered drives spawn bad practice offspring: Many IT guys believe that lettered drives suck, but they end up making more of them out of laziness, fear or uncertainty. For instance: say the P:HR_Benefits folder is mapped to every PC via Group Policy, and everyone is happy. Then one day someone in HR decides to put something on the P: drive that users in a certain department shouldn’t see. IT hears about this and figures, “Well! Isn’t this a pickle. I think, good sir, that the only way out of this storm of bad design is to go through it!” and either stands-up a new share on a new letter (\fsSecretHRStuff maps to Q:) or puts an NTFS Deny ACL on the sub-folder rather than disabling inheritance. More Help Desk tickets result, twice as many if the drive mapping spans AD Sites and is dependent on Group Policy.
  5. Lettered drives don’t scale: Good on your company for surviving and thriving throughout the 90s, 2000s, and into the roaring teens, but it’s time for a heart-to-heart. That M:Deals thing you stood-up in 1997 isn’t the best way to share documents and information in 2015 when the company you helped scale from one small site to a global enterprise needs access to its files 24/7 from the nearest egress point.

I wish Microsoft would just tear the band-aid off and prevent disk mapping of SMB shares altogether. Barring that, they should kill it by subterfuge & pain ((Make it painful, like disabling signed drivers or something))

But at the end of the day, we the consumers of the Microsoft stack bear responsibility for how we use it. And unfortunately, there is no easy way to kill the lettered drive, but I’ll give you some alternatives. It’s up to you to sell them in your organization:

  1. OneDrive for Business: Good on Microsoft for putting advanced and updated OneDrive clients everywhere. This is about as close to a panacea as we get in IT. OneDrive should be your goal for files and your project plan should go a little something like this: 1) Classify your on-prem file shares, 2) upload those files & classification metadata to OneDrive for Business, and 3) install OneDrive for Business on every PC, device, and mobile phone in your enterprise, 4) unceremoniously kill your lettered drive shares
  2. What’s wrong with wack-wack? Barring OneDrive, it’s trivial to map a \sharefolder to a user’s Library so that it appears in Window Explorer in a univeral fashion just like a mapped drive would
  3. DFS: DFS is getting old, but it’s still really useful tech, and it’s on by default in an AD Domain. Don’t believe me? Type \yourdomain and see DFS in action via your NETLOGON & SYSVOL shares. You can build out a file server infrastructure -for free- using Distributed File Sharing tech, the same kit Microsoft uses for Active Directory. Say goodbye to to mapping \sharesharename to Site1 via Group Policy, say hello to automatic putting bits of data close to the user viaGroup Policy.
  4. Alternatives: If killing off the F: drive is too much of an ask for your organization, consider locking them down top prirority with tools like SMB signing, access-based enumeration and other security bits available in Server 2012 and 2012 R2.

My Little Red Zed Edge – ZyXEL Zywall USG-50 Review

So I have a confession to make. I love Zyxel USG firewalls.

There, I said it. Feels good to finally admit it, to come out of the closet as a ZedHead, more or less.

I do not fear the judgment of the packet-pushing literati on twitter, because my little Red Zed edge device is loaded with features and packed with value.  Way more value than an ASA 5505 at any rate.

And after like six months of trying to understand the damn thing, I finally get it. Let me tell you a little about RedZed.daisettalabs.net, the edge device guarding the home lab, Child Partition, Supervisor Mod spouse and me from the big bad internet.

redzed2

The Good

It’s so loaded with features, it’s practically a hyperconverged play: For $200 and change, my Zyxel USG-50 Zywall is packed with features other vendors would have sharded  out as separate SKUs long ago.  Just take a look at the feature list here. Granted, the sexier ones are subscriptions, but Zyxel lets you take them for a test drive for 30 days, which I of course did the moment I got it. I haven’t subscribed to any since they expired, and frankly was disappointed with the BlueCoat implementation, but I’m considering the IDP subscription.

Even excepting all of the subscription programs, the Zed punches above its class with features that offer real value for a small/medium business, or even nerds guarding the LAN at home. The ones I really appreciate are listed below.

It’s PKI in a box, with some good identity integration: I like Public Key Infrastructure systems and so should you. The ZyXEL comes with one built-in. Though modest in scope (essentially you can generate/sign certs, no revocation/responder pieces) this is a nifty thing to have at this pricepoint, just the kind of value-add a small business might look for.

The Zed also capably integrates with AD directly, though in my testing it was a bit clunky & quite slow to authenticate against a 2012R2 domain. So, you can do what I did and switch to RADIUS, or LDAP if that’s your speed.

Easy WAN LBFO:  With the USG-50, you get two WAN links with easy ability to failover or spillover between them.

I’m using this in the lab at home and it works quite well. Though I only have one consumer internet connection, I’ve found that my provider hands out two public, routable IP addresses if I I connect two cables to my modem. This is awesome -worth its own post really- as I’ve been able to test WAN failure on Zed.

On WAN Port 1, I’ve got my last edge firewall device, a small PFsense box with an AMD Sempron and privoxy.

On WAN Port 2, I’m cabled directly to the modem. You get quite a few options to manage failover/spillover between the links, just like when you’re making an MPIO storage policy to your array! Perfect.

Both links work (double-natting behind pfsense works too, though I only ran it like that for a short while), and failover is pretty much transparent on general web stuff,  even a VPN service I run on node1 maintains connectivity during the failover.

Time for some Gifcam action:

wanfailover

Zyxel seems to know its target market quite well, and that market has commodity internet circuits -not private leased lines- connecting branch to HQ and branch to internet. WAN failover (no aggregation here, but I’m not sold on WAN aggregation yet) is important, and it’s huge that the Zed rocks LBFO out of the box, no licenses needed, and a few clicks to configure.

Zone-based firewall: I am not a security guy, but I understand the state of the art thinking to be less Internal/External as it used to be, and more segmentation everywhere via zones based on a sort of defense-in-depth concept; Create checkpoints or at least rules between external & internal segments of your network, in other words.

Zones come built in by default with ZyXEL, and figuring out the proper way to use them is what caused me so much pain & suffering with this device for so many months.

Now, I think I’ve got the concept down, but I’m not confident enough to talk about how well this device secures zones internally or externally, so just know this: it’s there. The firewall is ICSA certified, though reading through those docs it didn’t seem like that was much more than a rubber-stamp.

Object-Oriented ports, interfaces, zones, and VLANs: So this is the heart of USG line, more or less. It’s why some  dislike working with USGs, and others, like me, warm up to and eventually appreciate it. YMMV.

 

So what’s this OO thing about? I like to think of it as an abstraction, just like anything else in virtualization. Let’s take a look at how the docs define Zones, for instance:

zones

Oh. That’s not so bad, right? As long as I know the rules, I should just be able to click this thing here, hit apply on that thing there, and voila! ping my SVI…ahhh damnit!

Locked out again.

But seriously, when you go to configure this screen:Untitled picture

lock yourself out again, and refer back to the manual to figure out what you did wrong and you see this:

zones2

then you hop on putty post-factory default and it shows you something like this:

zywalcli

you feel kind of stupid and you start to hate this device, which seems to suffer from an acute case of Layer 2/Layer 3 identity disorder.

But struggle through it packeteer, because what awaits you on the other end is, if not the SDN you’ve been waiting for, then at least pretty damn flexible.

Here’s a primer to help you through:

Zone: A group of interfaces + a security context. You get three on the USG-50 line, DMZ, LAN1, LAN2

Interfaces: Software-based, not hardware. Three: LAN1, LAN2, DMZ. RENAME THESE!

Ports: The physical RJ-45; you get four

Port Groups: Hardware-based links connecting ports with each other

And the soft bits:

VLANs: VLANs exist on interfaces and cannot span multiple Zones. They act like trunked ports, and they tag outbound, and look for tags on inbound. Do you like SVIs? Well if you do then you gotta put an IP on it (required)

Bridges: A software link between interfaces at Layer 2. More or less the traditional definition of a switch, right? But you can put an IP on a bridge and -strangely- span zones with Bridges.

Vifs: As you would expect,simple vifs can be created in the contexts above. Useful.

I’m a visual person, so I made a little chart to help me get it.

zywall-oo

The chart shows a couple of things: 1) there are three zones in the four boxe. All the things inside each box belong to those zones, but not other zones. 2) Center Circle area shows re-named port-groups. My best advice is to rename LAN 1 & LAN2 into something else, so that you don’t get mixed up as I did consistently. 3) VLANs can exist in only the same Zone but effectively span ports. 4) Bridge is all sorts of Twilight Zone as a Bridge can join a VLAN  in Zone 2 with  the DMZ port group in Zone DMZ (but not its VLAN). 5) Ports are really nothing, just agnostic Layer 1 interfaces, or at least you can turn them into that.

From your Cisco switch, this is great, and enabled me to finally do what I wanted to do in my lab: tagging, everywhere and always from edge to core, and out back again over the airwaves! From my Meraki (VLAN 420 is for 2.4GhZ and devices I don’t really trust, 421 is laptop net + 5ghZ) to the Zywall through my 2960s, all is tagged, all is controlled and segmented.

Was it worth the six month fight with Zed to get to this point?

Why yes, yes it was.

All around decent performance: Again, punching at  or a little above its weight in firewall performance, and still offering good bang for buck value on encryption & IPS compared to the ASA 5505 ($340 retail) the other device I see everywhere in SME. Performance table based off spec-sheets below:

[table]

Item,Zed USG-50, Cisco ASA 5505

SPI Firewall throughput, up to 225mb/s, up to150mb/s

3DES/AES VPN Throughput, 90mb/s, up to 100 mb/s

IPS Throughput, 30mb/s, Upto 75mb/s

RJ-45 Ports,2xGbE WAN+ 4xGbE LAN, 8xFaE two with PoE

IPSec Tunnels (Max), 10, 10 (base)

[/table]

You can buy it at Fry’s, which is how I got mine: Oh man, I am really putting myself out there by admitting to occasionally shopping at Fry’s Electronics. Visiting Fry’s usually depresses me…as a retail experience, it’s not aged well and seeing one row after another filled with discarded, rejected & returned technology items is a real downer.

But sometimes the sales are really compelling. I had my eye on the USG-50 for months at $240, but I couldn’t pull the trigger until I saw it was on sale at Fry’s one weekend for $200. So I bought it, racked/stacked it in my lab that evening, and now, six months later, I’m astounded that you can just walk in and buy a value & feature-packed device like this without talking to a VAR first.

ZyXel could probably make more money if they parsed out the features as SKUs & sold the USG through exclusively through the channel, but they don’t. They sell it in places you can find consumer/prosumer equipment and pack it with some nice features an IT guy can appreciate.

Good Update Tempo: No gripes on the amount of firmware updates ZyXel continuously pushes out for free. I watch the CVE list for vulnerabilities, and while ZyXel has a spotty record in other product lines, it looks like you have to go back to 2008 to find a CVE that applies to the USG line.

No one knows how to pronounce the goofy name, so you can nickname it: Wikipedia’s description of the origin of the ZyXEL name is fun:

When ZyXEL unveiled its first chip-design (ZyXEL was originally a modem-chip design company) back in the late 1980s, the company only had a Chinese name (pronounced Her-Chin = “people work together very hard”). So it had to come up with an English name for a trade show in Asia. The original idea was ZyTEL (“Zy” means nothing, “TEL” for telecommunications). The problem was that someone already had this name announced for the show. So they played around with the letters and came up with ZyXEL instead.

The name does not actually mean anything, although some people claim “XEL” is a word-play on “excellence”.

The next challenge was how to pronounce it (everybody in the company was Chinese at that time).

So they fed the name into an old speech synthesizer (reportedly it was an Amiga). And the synthesizer pronounced it “Zai-Cel

I gave up and call it Zed, the proper British phoentic for the letter Z.

Embrace color in your stack:  Everyone’s putting some flourish & color into rack-mounted equipment, but Zed’s been Red for years.

Great, readable, dense documentation: Though I poke fun at the documentation above, it’s actually very very good at this price range. Six hundred pages good. Well-written too, with adequate diagrams, organization and scenarios.

Links at the bottom.

The Bad

Don’t use it as your DG for everything: If you are using a USG line device, my advice is not to think of those LAN-side ports as a Layer 2 switch ports, and furthermore, not to use this device as the default gateway handed out to clients that need LAN performance. Why?

Simple. It’s not really a switch, and it doesn’t perform well if you use it as such at Layer 2, and especially at Layer 3. Remember the zones above? Well they are security contexts, which means that your packets must gate through them, which will -mark my words- slow them down.

Simple example: using Red Zed as DG on my LAN, I tested large (4GB) SMB 3 file copies to my storage box. I peaked at about 180 megabits/second, a truly pathetic number, but within the the performance spec listed for the inspection engine looking at packets flowing between zones. Even within the same Zone (same port-group, so effectively switching @ layer2) I couldn’t hit above 45 megabytes/second, far less than the 260MB/s transfers I can achieve wtih my switch & LACP.

If you need performance but you like the Zone model, I recommend you use your switch as DG for servers and make the USG the gateway of last resort on the switch. Assuming your packets are tagged, you stay in your VLAN context throughout.

For untrusted or clients that don’t need wired performance, use the USG-50 as DG.

The Ugly & Conclusion

I can’t find anything ‘ugly’ about the USG. It’s a great device with a ton of functionality and neat features that make it a superb value against a more traditional ASA 5505.

[dg]

Zywall USG50 CLI

ZyWALL USG 50_v3 manual

Should have used FQDN in your malware, North Korea

Bad technology habits are universal, even among the strange and isolated yet apparently elite hacker dev community of North Korea.

From the FBI statement this morning assigning blame for the Sony hack directly on the hermit kingdom:

  • Technical analysis of the data deletion malware used in this attack revealed links to other malware that the FBI knows North Korean actors previously developed. For example, there were similarities in specific lines of code, encryption algorithms, data deletion methods, and compromised networks.
  • The FBI also observed significant overlap between the infrastructure used in this attack and other malicious cyber activity the U.S. government has previously linked directly to North Korea. For example, the FBI discovered that several Internet protocol (IP) addresses associated with known North Korean infrastructure communicated with IP addresses that were hardcoded into the data deletion malware used in this attack.

Devs can be really lazy, hardcoding an IP address where they should put an FQDN, though I suppose for their purposes, North Korea didn’t really care to cover their tracks (perhaps pointing the A record at someone else).

All kidding aside, this is really going to shake things up in IT environments small and large. I’m not sure if this is the first State-sponsored cyberattack on a private corporation on another nation’s soil, but it’s going to be the first one widely remembered.

Time to start implementing that which was once considered exotic and too burdensome….doing things like encrypting your data even when it’s at rest on the SAN’s spindles, off-lining your CA, encrypting its contents,and storing it on a USB stick inside a safe, governance procedures & paper-based chain-of-custody forms for your organization’s private keys.

Assume breach, in other words.

Hyper-V + VXLAN and more from Tech Ed Europe

If you thought -as I admittedly did- that on-prem Windows Server was being left for dead on the side of the Azure road, then boy were we wrong.

Not sure where to start here, but some incredible announcements from Microsoft in Barcelona, most of which I got from Windows Server MVP reporter Aidan Finn

Among them:

  • VXLAN, NVGRE & Network Controller, courtesy of Azure: This is something I’ve hoped for in the next version of Windows Server: a more compelling SDN story, something more than Network Function Virtualization & NVGRE encapsulation. If bringing the some of the best -and widely supported- bits of the VMware ecosystem to on-prem Hyper-V & System Center isn’t a virtualization engineer’s wet dream, I don’t know what is.
  • VMware meet Azure Site Recovery: Coming soon to a datacenter near you, failover your VMware infrastructure via Azure Site Recovery, the same way Hyper-V shops can

    Not sure what to do with this yet, but gimme!
    Not sure what to do with this yet, but gimme!
  • In-place/rolling upgrades for Hyper-V Clusters: This feature was announced with the release of Windows Server Technical Preview (of course, I only read about it after I wiped out my lab 2012 R2 cluster) but there’s a lot more detail on it from TechEd via Finn:  rebuild physical nodes without evicting them first.You keep the same Cluster Name Object, simply live migrating your VMs off your targeted hosts. Killer.
  • Single cluster node failure: In the old days, I used to lose sleep over clusres.dll, or clussvc.exe, two important pieces in Microsoft Clustering technology. Sure, your VMs will failover & restart on a new host, but that’s no fun.  Ben Armstrong demonstrated how vNext handles node failure by killing the cluster service live during his presentation. Finn says the VMs didn’t failover,but the host was isolated by the other nodes and the cluster simply paused and waited for the node to recovery (up to 4 minutes). Awesome!
  • Azure Witness: Also for clustering fans who are torn (as I am) between selecting file or disk witness for clusters: you will soon be able to add mighty Azure as a witness to your on-prem cluster. Split brain fears no more!
  • More enhancements for Storage QoS: Ensure that your tenant doesn’t rob IOPS from everyone else.
  • The Windows SAN, for real: Yes, we can soon do offsite block-level replication from our on-prem Tiered Storage Spaces servers.
  • New System Center coming next year: So much to unpack here, but I’ll keep it brief. You may love System Center, you may hate it, but it’s not dead. I’m a fan of the big two: VMM, and ConfigMan. OpsMan I’ve had a love/hate relationship with. Well the news out of TechEd Europe is that System Center is still alive, but more integration with Azure + a substantial new release will debut next summer. So the VMM Technical Preview I’m running in the Daisetta Lab (which installs to C:Program FilesVMM 2012 R2 btw) is not the VMM I was looking for.

Other incredible announcements:

  • Docker, CoreOS & Azure: Integration of the market-leading container technology with Azure is apparently further along than I believed. A demo was shown that hurts my brain to think about: Azure + Docker + CoreOS, the linux OS that has two OS partitions and is fault-tolerant. Wow
  • Enhancements to Rights Management Service: Stop users from CTRL-Cing/CTRL-Ving your company’s data to Twitter
  • Audiocodes announces an on-prem device that appears to bring us one step closer to the dream: Lync for voice, O365 for the PBX, all switched out to the PSTN. I said one step closer!
  • Azure Operational Insights: I’m a fan of the Splunk model (point your firehose of data/logs/events at a server, and let it make sense of it) and it appears Azure Operational Insights is a product that will jump into that space. Screen cap from Finn

This is really exciting stuff.

Commentary

Looking back on the last few years in Microsoft’s history, one thing stands out: the painful change from the old Server 2008R2 model to the new 2012 model was worth it. All of the things I’ve raved about on this blog in Hyper-V (converged network, storage spaces etc) were just teasers -but also important architectural elements- that made the things we see announced today possible.

The overhaul* of Windows Server is paying huge dividends for Microsoft and for IT pros who can adapt & master it. Exciting times.

* unlike the Windows mobile > Windows Phone transition, which was not worth it

More than good hygiene : applying a proper cert to my Nimble array

So one of my main complaints about implementing a cost-effective Nimble Storage array at my last job was this:

Who is Jetty Mortbay and why does he want inside my root CA store?
Who is Jetty Mortbay and why does he want inside my root CA store?

I remarked back in April about this unfortunate problem in a post about an otherwise-flawless & easy Nimble implementation:

The SSL cert situation is embarrassing and I’m glad my former boss hasn’t seen it. Namely that situation is this: you can’t replace the stock cert, which, frankly looks like something I would do while tooling around with OpenSSL in the lab.

I understand this is fixed in the new 2.x OS version but holy shit what a fail.

Well, fail-file no more,  because my new Nimble array at my current job has been measured and validated by the CA Gods:

verified
Green padlocks. I want green padlocks everywhere

Oh yeah baby. Validated in Chrome, Firefox and IE. And it only cost me market rates for a SAN certificate from a respected CA, a few hours back ‘n forth with Nimble, and only a few IT McGuyver-style tricks to get this outcome.

Now look. I know some of my readers are probably seeing this and thinking…”But that proves nothing. A false sense of security you have.”

Maybe you’re right, but consider.

I take a sort of Broken Windows Theory approach to IT. The Broken Windows Theory, if you’re not familiar with it, states that:

Under the broken windows theory, an ordered and clean environment – one which is maintained – sends the signal that the area is monitored and that criminal behavior will not be tolerated. Conversely, a disordered environment – one which is not maintained (broken windows, graffiti, excessive litter) – sends the signal that the area is not monitored and that one can engage in criminal behavior with little risk of detection.

Now I’m not saying that adding a proper certificate to my behind-the-firewall Nimble array so that Chrome shows me Green Padlocks rather than scary warnings is akin to reducing violent crime in urban areas. But I am saying that little details, such as these, ought to be considered and fixed in your environment.

Why? Well, somehow fixing even little things like this amount to something more than just good hygiene, something more than just ‘best practice.’

Ultimately, we infrastructurists are what we build, are we not? Even little ‘security theater’ elements like the one above are a reflection on our attention to detail, a validation of our ability to not only design a resilient infrastructure on paper at the macro level, but to execute on that design to perfection at the micro level.

It shows we’re not lazy as well, that we care to repair the ‘broken windows’ in our environment.

And besides: Google (and Microsoft & Mozilla & Apple) are right to call out untrusted certificates in increasingly disruptive & work-impairing ways.

*If you’re reading this and saying: Why don’t you just access the array via IP address, well, GoFQDNorGoHomeSon.com

Microsoft to introduce the New Shiny Windows

Devoted readers of Agnostic Computing.com, I write today to implore you to set your powershell scripts to Signed, get your Windows Key + R trigger fingers ready, and prep your forests and domains for a functional upgrade because today ladies and gentlemen, today, we get a new Windows. 

Ahhh yeah.

There’s some excitement in Microsoft Country again.

No one knows what it’ll be called. Windows 9 is the front-runner, but late-breaking rumors say big MS could throw us for a loop too and name it Windows TH (Threshold?!?! the pundits echo) or just plain old Windows.

It's always a good day when a new Windows is detailed
It’s always a good day when a new Windows is detailed

I say they should name it Windows TNS: Windows The New Shiny. Because among the rumors I’ve enjoyed hearing most is the one Microsoft may offer a sort of Windows 365 subscription for fanbois like me, a continuously morphing and changing OS, just like my O365 experience has been. New Shiny Windows every month…well maybe I’d tell ConfigMan to delay updates for a week or so, just to shake the bugs loose. But still. A subscription OS would be great.

But that’s a long-shot and probably not a very strong selling point for today’s event, which is, as everyone has noted, focused entirely on enterprise computing.

You see, Microsoft is trying desperately to court Enterprise IT people, to bring us back into the fold, targeting this entire event today at IT people like me who were aghast & horrified two years ago when they first installed Windows 8 in a VM.

“No. No. To get to start screen, hover your mouse in the lower corner. The lower corner, not the charms bar.There it is. Click that. Ahh shit, you missed it. Try again.” was how the conversation went throughout IT departments in ‘Merica.

As I’ve written before, the experience of Windows 8 & Server 2012 was so shocking and painful, it sent me running and crying into the Mac OS X camp, and then into ChromeBook fantasyland.

But I got over it. I overcame, and I figured out how to move all that nonsense touch stuff away when Windows 8.1/Server 2012 R2 debuted about a year ago.

Apparently other IT pros haven’t, and are still sticking to Windows 7 as if it’s the greatest thing since Active Directory. Thus today’s event.

To them I say: get with the program, or get left behind. Windows 8 did suck, but 8.1 & 2012 R2 were fine recoveries. If you decided to punt on learning about Windows 8.1/2012 R2, you missed a whole bunch of incredible advancements that are only going to improve with Windows TNS. Have fun catching up on this:

  • Baked in Hyper-V. Free on Windows 8.1 Pro and up. A virtual lab on every desktop.
  • Tiered Storage Spaces in Windows server 2012 R2: yet another software abstraction framework, but for your storage! You missed out on this too!
  • An awesome networking stack, totally rewritten: Native support for teaming, network function virtualizations, Layer 3 routing protocols via PowerShell…oh my. I’d hate to be you stuck with a Server 2008 R2 box, running your old tired batch files, your dated vbs scripts and ipconfig. You missed out on some incredible advancements

And the great thing is that all this is going to get better, I think (hope). True, we won’t be learning about Windows Server today (Aidan Finn reckons that + nextgen System Center will be next month) but there will be lots of detail about our next Enterprise desktop product, by which you can bet people like me will make inferences for the next server product.

Things are looking up in Microsoft Country. We’ve a ten year head start on Trustworthy Computing (ShellShock couldn’t have had better timing for MS), a highly-modular & secure OS, a mature cloud stack, a SaaS offering second to-none (O365) and now, today, a new Windows OS.

Good times.

Stack Builders ‘R Us

This is a really lame but (IMHO) effective drawing of what I think of as a modern small/medium business enterprise ‘stack’:

stack

As you can see, just about every element of a modern IT is portrayed.

Down at the base of the pyramid, you got your storage. IOPS, RAID, rotational & ssd, snapshots, dedupes, inline compression, site to site storage replication, clones and oh me oh my…all the things we really really love are right here. It’s the Luntastic layer and always will be.

Above that, your compute & Memory. The denser the better, 2U Pizza Boxes don’t grow on trees and the business isn’t going to shell out more $$$ if you get it wrong.

Above that, we have what my networking friends would call the “Underlay network.” Right. Some cat 6, twinax, fiber, whatever. This is where we push some packets, whether to our storage from our compute, northbound out to the world, southbound & down the stack, or east/west across it. Leafs, spines, encapsulation, control & data planes, it’s all here.

And going higher -still in Infrastructure Land mind you- we have the virtualization layer. Yeah baby. This is what it’s all about, this is the layer that saved my career in IT and made things interesting again. This layer is designed to abstract all that is beneath it with two goals in mind: cost savings via efficiency gains & ease of provisioning/use.

And boy,has this layer changed the game, hasn’t it?

So if you’re a virtualization engineer like I am, maybe this is all you care about. I wouldn’t blame you. The infrastructure layer is, after all, the best part of the stack, the only part of the stack that can claim to be #Glorious.

But in my career, I always get roped in (willingly or not) into the upper layers of the stack. And so that is where I shall take you, if you let me.

Next up, the Platform layer. This is the layer where that special DBA in your life likes to live. He optimizes his query plans atop your Infrastructure layer, and though he is old-school in the ways of storage, he’s learned to trust you and your fancy QoS .vhdxs, or your incredibly awesome DRS fault-tolerant vCPUs.

Or maybe you don’t have a DBA in your Valentine’s card rotation. Maybe this is the layer at which the devs in your life, whether they are running Eclipse or Visual Studio, make your life hell. They’re always asking for more x (x= memory, storage, compute, IP), and though they’re highly-technical folks, their eyes kind of glaze over when you bring up NVGRE or VXLAN or Converged/Distributed Switching or whatever tech you heart at the layer below.

Then again, maybe you work in this layer. Maybe you’re responsible for building & maintaining session virtualization tech like RDS or XenApp, or maybe you maintain file shares, web farms, or something else.

Point is, the people at this layer are platform builders. To borrow from the automotive industry, platform guys build the car that travels on the road infrastructure guys build. It does no good for either of us if the road is bumpy or the car isn’t reliable, does it? The user doesn’t distinguish between ‘road’ and ‘car’, do they? They just blame IT.

Next up: software & service layer. Our users exist here, and so do we. Maybe for you this layer is about supporting & deploying Android & iPhone handsets and thinking about MDM. Or maybe you spend your day supporting old-school fat client applications, or pushing them out.

And finally, now we arrive to the top of the pyramid. User-space. The business.

This is where (and the metaphor really fits, doesn’t it?) the rubber meets the road ladies and gentlemen. It’s where the business user drives the car (platform) on the road (infrastructure). This is where we sink or swim, where wins are tallied and heros made, or careers are shattered and the cycle of failure>begets>blame>begets>fear>begets failure begins in earnest.

That’s the stack. And if you’re in IT, you’re in some part of that stack, whether you know it or not.

But the stack is changing. I made a silly graphic for that too. Maybe tomorrow.