Wilson.tech

Enterprise Secrets + Privileged Account Management | CyberArk at #XFD1

Managing Enterprise Secrets & Privileged accounts has to be one of the most difficult jobs in Information Technology today, and one of the least transparent to the business. Bad guys have painted a target on admin’s backs, regulators are chomping at the bit as more consumer data is lost online, and Compliance officers are scrambling to understand the landscape and adapt to new rules from overseas. And yet the business may not even realize that unsung heroes in IT are still managing a stack of hardware & software designed to fulfill 1990s-era security models.

Take it from me: I know this pain well. Even if you do have an internal identity system, say Active Directory, it can be difficult to get all the bits from your Storage, Network, Compute & cloud systems to run a proper AAA model against your AD Forest. Even more difficult: figuring out how to audit the records of Active Directory (or NPS/RADIUS or ADFS or OAuth2/SAML glues) to present to your Compliance officers.

Yet in the background, a constant churn of news that only raises the pessimism bar higher: Target. Anthem. Maersk. Equifax. Facebook. Marriot. The goddamned CIA and the f****** National Security Agency. I made a Visio Timeline because I was having difficulty tracking all the breaches, and I’ve run out of room! And let’s not forget the business and your user colleagues’ need for secrets too as consumer technology continues to eat away at the Enterprise and as more of the economy is digitized. By 5pm most days, IT admins are just hoping to make it to retirement in 10 years without their orgs getting popped by a black hat.

Enter CyberArk. This Silicon Valley company was founded in 1999, which is impressive to me. It’s not often you’ll find a company that’s been selling a product that handles Enterprise secrets + PAM for 20 years, at least a decade longer by my count than the popular consumer password management companies that are now sashaying their way into your Enterprise, as if they understand the challenge you’re facing. At Security Field Day 1 (#XFD1), CyberArk’s maturity & comprehension of the challenge of securing the enterprise really showed.

CyberArk’s Privileged Access Security Suite is a mature & fully-featured secrets + PAM tool. I was super-impressed with the demo their Global Director of Systems Engineering, Brandon Traffanstedt, gave us back in December 2018 in sunny San Jose. I came prepared to endure a boring password management demo; I left impressed at what I had seen, with only a single caveat.

Not only was CyberArk’s product comprehensive, it was bad-ass, with one exception. I saw:

An SSH session opened to a network device’s command line, with a second factor prompt before access was granted
Full auditing + screen recordings of a Privileged Account accessing a protected server, just the kind of thing that reassures the business that you, as an admin, have nothing to hide, are not an ‘insider threat’ and are 100% transparent in your work.
Deep integration into Windows’ Win32 API, hooking into parts of the OS I’d not seen before outside of Microsoft products, including Credential Management
Full integration & support for MacOS
OAUTH2/SAML support and full support for your ADFS infrastructure
Cloud secrets & PAM management across AWS (and soon) Azure
Full support for your RADIUS infrastructure & 802.11x, whether via Microsoft’s NPS or some other solution
Automated credential rotation so that you don’t have to scramble when a fellow admin changes jobs, is fired for negligence, or joins Edward Snowden in Moscow
Secure sharing of secrets among your privileged IT colleagues
An offline, secured, and high-entropy password in a sealed envelope you can hand to the business for peace of mind

I’ve been working in IT for about as long as CyberArk’s been pounding the pavement and trying to convince IT Teams to invest in Enterprise Secrets & PAM software. I was impressed…..particularly because CyberArk scratches an itch that many IT Teams don’t know they have: the security costs & technical debt that a legacy of tactical, rather than strategic, investments that tend to leave an org arrears in 2019’s security landscape.

Por ejemplo: say you’re a mid-market SMB IT shop in the healthcare sector that’s experienced a lot of turnover among its IT admin staff through the years. If you’re the business, you’ve watched as IT Admins come and go, and listened as they’ve pitched tactical solutions to various challenges facing the business. You’ve invested in a few, and most work well enough, but gluing them all together into a comprehensive, strategic, and business-enabling solution has been a challenge.

cyberark While your solutions are working, you’re paying a cost whether you know it or not because more than likely, the technical legwork needed to glue those solutions together into a comprehensive & auditable security framework hasn’t been done. Meanwhile, the regulators are knocking at your door, the pace of breaches quicken, and Brian Krebs’ pen is waiting to write about your company.

CyberArk is a good fit there. No, check that. It’s a *great* fit in that scenario. The product addresses threats to your business from both the inside and the outside. It protects Enterprise secrets -the very thing your admins are targeted for- while shining a bright light on your employee’s Privileged Accounts and how they are used.

It’s a product that’s far beyond anything the consumer password management companies are offering…trust me, I’ve looked at them all. It’s a true Enterprise solution. However….

I will say that one area where CyberArk felt a bit less than polished was in how they’ve architected the sharing & use of secrets with non-admin users working in the business. If we return to the healthcare example, think of a person in your business who needs the credentials to login to a state Medicaid site in order to bill the payor of a medical product.

In fairness, this is a complicated problem…while it’s in the business’ interests to control/maintain/audit all secrets, including to third party sites & services that are outside of IT’s domain, the mix of devices/browser here is a difficult puzzle to solve. Yet it’s here that CyberArk’s product left me perplexed. They propose intercepting TLS traffic on your user’s endpoints & injecting credentials into your business user’s browsers, whatever they may be.

This seemed to me -at the ass-end of 2018- to be a poor solution. For starters, we’ll soon see TLS 1.3 across more and more websites. TLS 1.3, as my fellow Delegate Jerry Gamblin pointed out, is not something you can intercept, decrypt, and inject credentials into. Indeed, other vendors in the security space seem to be steering Enterprise customers away from the expectation that we’ll be able to intercept/inspect/fiddle with TLS 1.3 connections. At best, we’ll be able to refuse TLS 1.3 connections in favor of the more Enterprise-friendly TLS 1.2 connections, but even here, the Enterprise’s political power & ability to influence the market & standards bodies is lacking, and Google, for better & worse, rules the roost. Even Microsoft is playing second fiddle here and announced in late 2018 that it would ditch its new Edge browser’s Trident engine in favor of Chromium open source.

Secondly, CyberArk’s solution even here feels archaic. They propose that you put a middlebox in front of your users to accomplish this. This is definitely old-school, calling to mind the many nights/weekends I spent configuring & troubleshooting BlueCoat devices in server rooms across many Southern California businesses. If you’re going to tackle a problem like TLS intercept, you need to think 21st century and go with a cloud interception service, that will follow your users around on the internet. Middleboxes often make your security posture worse, not better.

In my day job, I intercept/inspect TLS connections across several continents and on several thousand endpoints; it’s a tricky science and one that’s filled with compliance & policy questions above my paygrade. Microsoft’s move in the browser arena fills me with questions, and that’s before we consider mobile devices; so too should it fill you with questions if you are looking at CyberArk with an eye towards sharing secrets with non-admin users.

So, caveat emptor on this narrow point friends: a significant selling point of CyberArk’s featured product (injecting secrets into an HTTPS session) may not work a year or two from now. We raised this issue at #XFD1 and CyberArk says they have a plan for it, but eyes open!

Other than that though, I was really impressed. CyberArk gets the challenge facing Enterprise IT in this Wild West era. It understands intuitively complexities of Enterprise secrets, PAM, insider vs outsider threats, and auditing/compliance requirements. The only place it seems to fall short is in sharing credentials from the ‘Vault’ to non-privileged users.

Check it out if:

You’ve got a heterogenous stack of best of breed IT hardware & software and you’ve neglected integrating AAA security across that stack
You’re in an environment requiring heavy compliance & auditable proof across your stack against both insider & outsider threats
You want 2FA/MFA on old network switches, Macs, and Windows Servers
You want screen captures of your admin’s work on devices, servers, and services that you consider privileged
You’ve got cloud/SaaS management challenges even as you’ve centralized identity in on-prem Active Directory or other system

Ignore it if:

You’ve only ever bought Microsoft, only have Windows PCs & servers and Microsoft applications, and you have an MCSE on staff who understands Kerberos, Active Directory, NPS, RADIUS, ADFS, OAUTH2/SAML, and has configured your AD environment to comply with various regulatory statutes and compliance regimes

Other Coverage:

Entire Presentation from #XFD1 (Vimeo)
My Tweets on CyberArk during #XFD1 Presentation
Other (check back soon as more Delegates write reviews)
Chris Gunderman, friend & #XFD1 Delegate review

Disclosures

This blog post was written by me, Jeff Wilson, for publication on my blog, wilson.tech. I was not compensated by CyberArk to compose this blog post, and CyberArk did not see it prior to its publication. I learned about the CyberArk products during Security Field Day 1 (#XFD1) an event for IT, Security, and Enterprise influencers that was held in December 2018 in & around Silicon Valley, California. The Gestalt IT group paid for my airfare, accommodations, and meals during the time I was in greater San Jose, CA area. CyberArk and other sponsors paid Gestalt IT to bring Delegate influencers like me to #XFD1.

I received no monetary compensation otherwise, save for the swag listed below

CyberArk swag I took home:

A ballpoint pen

About Me: My name is Jeff Wilson. I am a 20 year IT Professional with a security focus. I hold a GSEC from the SANS Institute, as well as a Bachelor’s Degree in History & a Master’s in Public Administration, both of which are from CalState. I live & work in Southern California. You can reach me on twitter @jeffwilsontech or via email at blog@wilson.tech

Cloud Field Day 3 | Morpheus Data | #CFD3

Morpheus Data was our first sponsor at #CFD3 and, as is my custom before Tech Field Day events, I had done zero prep work on Morpheus. I had never heard of the firm, and as first-at-bat sponsors for #CFD3, they were facing 12 delegates full of energy and with decades of Information Technology experience between them. So how’d they do? I came away impressed. Let me tell you why: they have a heart for operations, and I’m an operations guy.

Morpheus Data – Background

I found Morpheus Data’s story pretty compelling when I read up on it later. The company started off more or less as an internal product inside a cost center of Bertram Capital, a private equity firm in the Bay Area. Now every company has a founding mythology, but Morpheus’s range true to me. Here, I’ll quote from their site:

Bertram Labs is a world-class team of software developers and ops professionals whose sole purpose is to rapidly implement IT solutions to fuel the growth of the Bertram portfolio. In 2010, that team needed a 100% infrastructure agnostic cloud management platform which would integrate with the DevOps tools they were using to develop and deploy applications for a range of customers on an unpredictable mix of heterogeneous infrastructure. Such a tool didn’t exist so Bertram Labs created their own solution…

Just that phrase right there -an unpredictable mix of heterogenous infrastructure- comprises the je nais se qua of my success as an 18 year IT Pro. Using ratified standards sent to us from on high by the greyhairs at the IETF & IEEE ivory towers, a competent IT Pro like myself can string together disparate hardware systems into something rational because most vendors sometimes follow those standards.

But it’s very hard work. It’s not cheap either. And that act -that integration of a Cisco PoE switch with an Aruba access point or an iSCSI storage array with a bunch of Dell servers- isn’t bringing much value to the business. Perhaps it would be different if IT Shops could just start over with a rational greenfield infrastructure design, but that’s rare in my experience because the needs of IT aren’t necessarily aligned with the needs of the business.

Morpheus Data says they grew out of that exact scenario, which is immediately familiar to me as an ops guy. I find that story pretty encouraging; an internal DevOps team working for a private equity firm was able to productize their in-house scripts & techniques and are now a separate company. Damn near inspiring!

So what are they selling?

It’s Glue, basically. But well-articulated & rational glue

Morpheus’ pitch is that their suite of products can take the pain out of managing & provisioning services from your stack of heterogenous stuff whether it’s on-premises, in one cloud, or several clouds. And by taking the pain out, you can move faster and bring more value to the business.

I’m not going to get into each product because frankly, I think they’re poorly named and not very exciting (Sharepoint-esque in a way: Analytics, Governance, Automation, Evolution, Integrations). But don’t let the naming confuse or dissaude you; it’s an exciting product and the pricing model is simple to understand. clover-b4ff8d514c9356e8860551f79c48ff7c

Instead, let me describe to you what I saw during Morpheus’ Demo at #CFD:

Performance data from On-Premise virtualization servers running Hyper-V, VMware, and even Citrix’s XenServer all in one part of the Morpheus web-based portal
You can drill-down from each host to look at VM performance data too. Morpheus says they’re able to hook into both Hyper-V performance counters and VMware’s performance counters. That’s pretty awesome for a hetergeonous shop
Performance & controls over IaaS & PaaS instances in both Azure & AWS, again in the same screen
Menu-driven wizards that let you instantly provision a new virtual machine pre-configured for whatever service you want to run on it. Again -this could be done in the same tool and you can pick where you want it to go
Cost data from each public clouds
Rich RBAC controls, which is very important to me from a security & integrity standpoint
A composable role-based interface. Por ejemplo, you can let your dev team login to Morpheus and not worry about him or her offlining a .vhdx on a Hyper-V server

This chart from their website sums up their offering nicely in comparison with other vendors in this space.

morpheus

Concluding Thoughts

I’ve worked in IT environments where purchasing has been less than most people would consider as rational. Indeed, I’ve worked at places where we had the very best equipment from multiple vendors, but nobody had the time or talent to integrate it all into a smooth & functional machine in service to the business.

Stepping back, the very nature of the integration puzzle has changed. I mentioned above that a competent IT Pro could stitch together infrastructure that used IETF, IEEE, w3c and other standards-based technologies. Indeed that’s been the story of my career.

But in 2018, the world’s moved on from that, for better and worse. The world’s moved on to proprietary Application Programming Interfaces (APIs), and so I’ve moved with it, creating my own Powershell functions and Python scripts to interact with cloud-based APIs. You can do this too, given enough time & study.

But let’s be honest: it’s hard enough to manage & integrate a heteregenous stack of best-of-breed stuff on-premises. Now your boss comes to you and wants you to add some Azure services & Office 365. And then someone on the business side orders up some Lambdas in AWS, surprise! Or perhaps a distant IT group at your company just went and bought Cloudflare or Rackspace. If you’re still trying to solve standards-based puzzles of yesteryear, while learning how to develop scripts & tools for use in a world of proprietary APIs, you’re probably not bringing much value to the business.

And that’s where Morpheus sees itself slotting in nicely…they’ve done the hard work of integrating with both your legacy on-premises standards-based systems and the API-driven cloud ones, and they release new integrations ‘every two or three weeks.’ They even take requests, so if you’ve got a bespoke stack of stuff that doesn’t surface SNMP properly, you can propose Morpheus build an integration for it.

Sidenote: One of the more dev-focused delegates at #CFD3 criticized the prodcut as too ops-friendly (nobody cares to see all that stuff! he said), but I had to push back on him because details are important for ops teams, and Morpheus can surface an interface that’s safe for devs to use. And that’s why I say they’ve got a heart for operations teams.

On pricing: the products which again, have somewhat confusing names, at least offer simplified pricing. To get workload & ‘core features’ running on a VM in your datacenter, you’ll need to spend $25k to start. That seems high to me, but you’re essentially buying a DevOps integrator & engineer who can work 24/7 and doesn’t need health insurance or take vacation, which is pretty cool, and which helps you bring value to the business.

Disclosures

This blog post was written by me, Jeff Wilson, for publication on my blog, wilson.tech. I was not compensated by Morpheus Data to compose this blog post, and Morpheus did not see it prior to its publication. I learned about the Morpheus Data products during Cloud Field Day 3, an event for IT & Enterprise influencers that was held in April 2018 in Santa Clara California. The Gestalt IT group paid for my airfare, accomodations, and meals during the time I was in Santa Clara. Morpheus and other sponsors paid Gestalt IT to bring Delegate influencers like me to #CFD3

Morpheus Data shwag I took home

Cool stickers
A t-shirt

Defending IT amidst the novel WannaCry worm

It’s been a hell of a few days here in the trenches of Information Technology in 2017. Where to begin?

Between explaining how this all works to concerned friends & family, answering my employer’s questions about our patching posture & status, and reading the news & analysis, I think it’s safe to say that WCry has been in my thoughts for every one of the last 72 hours, including the 24 hours of Mother’s Day and all the hours I spent in restless slumber.

Yes, that’s right. WCry was on my mind even as I celebrated Mother’s day for the three women I’m close to in my life who are mothers. Wow. Just wow.

Having had the chance to catch my breath, I’ve got some informed observations about this global incident from my perspective as an IT Pro. Why is WCry as interesting & novel as it is potent and effective in 2017? And is there any defense of an IT team one might make if their organization got pwned by WCry?

I contemplate both questions below.

WCry successfully chains a social engineering attack with a technical exploit resulting in automated organization pwnage
WCry begins as a social engineering/phishing attack on users in the place they love and hate by equal measure: their Inbox. Using Subject lines that draw the eye, the messages include malicious attachments. This facet of WCry is not new of course…..it’s routine and has been in IT for at least two decades.

Once the attachment is clicked, WCry pivots, unleashing an NSA-built cyberweapon upon the enterprise by scanning port 445 across the local /24, cycling through cached RDP accounts and calling special attention to SQL & Exchange services, presumably to price the ransom accordingly.

Then it encrypts. Nearly everything.

All of this from a single email opened by a gullible user.

This behavior -socially engineered attack on human meatbag + scan + pivot to the rest of the network- is also not novel, new or remarkable. In fact, security Pros call this behavior “moving laterally” through an enterprise and they usually talk about it being done from “jump box” or “beach head” that’s been compromised via social engineering. Typically, security pros will reserve those terms to describe the behavior of a skilled & hostile hacker meatbag intent on pwning a targeted organization.

Where WCry is novel is that it in effect automates the hacker out of the picture, making the whole org pwnage process way more efficient. This is Organization-crippling, self-replicating malware at scale. Think Sony Pictures 2014, applied everywhere automatically minus the North Korean hacker units at the keyboard.

The red Wcry “Ooops” message is both informative and visually impressive, which multiplies its influence beyond its victims
As these things go, I couldn’t help but be impressed with Wcry’s incredibly detailed and anxiety-inducing UI announcing a host’s Wcry infection:

This image, or some variant thereof, has appeared on everything from train station arrival/departure boards to manufacturing floor PCs to hospital MRIs to good old-fashioned desktop PCs in Russia’s Interior Ministry. The psychological effects of seeing this image on infected hardware, then seeing it again on popular social media sites, the evening news, and newspapers around the world over the last few days are hard to determine, but I know this: this had an effect on normal consumers and users of technology across the globe. Sitting on my lap Saturday, my four year old saw the image in my personal OneNote pastebin and asked me, “Daddy, is that an alarm? Why does it show a lock? Do you have key?”

What’s interesting is that while computer users saw this or a screensaver version of this image, in reality you could click past it or minimize it in some way. Yet images of this application have proliferated on Twitter, FaceTube and elsewhere. Ransomware used to just announce itself in the root of your file share or your c:\user\username\documents folder: now it poses for screen caps and cell phone pics which multiplies its effectiveness as a PsyOps weapon. By Saturday I was reading multiple articles in my iPad’s Apple News about how regular people could protect themselves from the ‘global cyberattack.’

Its function is not just about encrypting file shares like earlier ransomware campaigns, but about owning Enterprises
If my organization or any organization I was advising got hit by WCry, my gut feeling is that I wouldn’t feel secure about my Forest/Domain integrity until I burned it down and started over. Why? Well, big IT security organizations like Verizon’s Enterprise Security group typically don’t classify ransomware as a ‘data breach’ event. Yet, as we know, Wcry installs a Pulsar backdoor that enables persistent access in the future. This feels like a very effective escalation of what it means to be ransomed in modern IT organizations, so yeah, I wouldn’t feel secure until our forest/domain was burned to the ground.

It is the manifestation of a Snoverism : Today’s nation-state cyberweapon is tomorrow’s script-kiddie attack
I was listening to the father of Powershell, Jeff Snover once and he implanted yet another Snoverism in my brain. He said, paraphrasing here, that Today’s nation-state attack is tomorrow’s script-kiddie attack. What the what?

Let’s unpack: the democratization of technology, the shift to agile, DevOps, and other development disciplines along with infrastructure automation has lead to a lot of great things being developed, released and consumed by users very quickly. In the consumer world this has been great -Alexa is always improving with new skills…Apple can release security patches rapidly, and FaceTube can instantly perform A/B testing on billions of people simultaneously. But not well understood by many is the fact that Enterprises and even individuals can harness these tools and techniques to instantly build and operate data systems globally, to get their product, whatever it may be, to market faster. The classic example of this is Shadow IT, wherein someone in your finance team purchases a few seats on Salesforce to get around the slow & plodding IT team.

I think Snover was observing that bad guys get the same benefits from modern technology techniques & the cloud as consumers and business users do.

And as I write this on Monday, what are we seeing? WCry is posted on GitHub and new variants are being created without the kill-switch/sandbox detection domain. Eternal Blue, the component of Wcry that exploits SMB1, was literally just a few months ago a specialized tool in the NSA’s cyber weapons arsenal. By tomorrow it will be available to any kid who wants it, or, even worse, as a push-button turn-key service anybody can employ against anybody else.

The democratization of technology means that no elite or special knowledge, techniques or tools are required to harness technology to some end. All you need is motive and motivation to do things at scale. This week, we learned that the democratization of technology is a huge double-edged sword.

It was blunted by a clever researcher for about $11
Again on the democratization of technology front, I find it fascinating that MalwareTech was able to blunt this attack by spending $11 of his own money to purchase the domain he found encoded in the output of his decompile. He’s the best example of what a can-do technologist can do, given the right amount of tools and freedom to pursue his craft.

It has laid bare the heavy costs of technical debt for which there is no obvious solution
Technical debt is a term used in software engineering circles and computer science curricula, but I also think it can and should apply to infrastructure thinking. What’s technical debt? Take it away Wikipedia:

Technical Debt is a metaphor referring to the eventual consequences of poor system design, software architecture, or software development within a codebase. The debt can be thought of as work that needs to be done before a particular job can be considered proper or complete. If the debt is not repaid, then it will keep on accumulating interest, making it hard to implement changes later on.

I can’t tell you how many times and at how many organizations I’ve seen this play out. Technical Debt, from an IT Pro’s perspective, can be the refusal to correct a misconfiguration of an important device upon which many services are dependent, or it can be a poorly-designed security regime that takes bad practice and cements it into formal process & habit, or it can be a refusal to give IT the necessary political cover & power to change bad practices or bad design into something durable and agile, or it can be refusing to patch your systems out of fear or a desire to kick the can down the road a bit.
Over time, efforts will be made to pay that technical debt down, but unless a conscious effort is made consistently to keep it low, technical debt eventually -inevitably- becomes just as crippling to an organization as credit card debt becomes to a consumer. Changes to IT systems that in other organizations are routine & easy become hard and difficult; and hard changes in other companies are close to impossible in yours.

This is a really bad place to be for an IT Pro, and now WCry made it even worse by exploiting organizations that have high technical debt, particularly as it relates to patching. Indeed, it’s almost as if the author of this malware understood at a basic fundamental level how much technical debt organizations in the real world carry.

There is no obvious solution to this. We can’t force people to use technology a certain way, or even to think of technology in a certain way. The point of going into business is to make money, not to build durable & secure and flexible technology systems, unless that is your business. Cloud services are the obvious answer, but they can’t do things like run MRI machines or interface with robots on the Nissan assembly line. At least not yet. And nobody wants regulation, but that’s a topic for another post.

It has shown how hard it is to maintain & patch systems that are in-use for more than a typical workday
If we ignore the way WCry rampaged through Russia, China and other places where properly licensing your software is considered optional, something else interesting emerges: the organizations that were hardest hit by Wcry were ones in which technology is likely in use beyond the standard 8 hour workday, which likely makes patching those technology systems all the more difficult.

While reporting on the NHS fiasco has zoomed in on the fact that the UK’s healthcare system had Windows XP widely deployed, I don’t think that tells the whole story, even if it’s true that 100% of NHS systems ran XP, it still doesn’t tell the whole story. I can easily see how patching in such environments could be difficult based on how much those systems are used. Hospitals and even out-patient facilities typically operate more than 8 hours a day; finding a slot of time in a given 24 hour period in which you can with the consent of the hospital, offline healthcare devices like MRI machines to update & reboot them is probably more difficult than it is in a company where systems are only required to be up between 7am and 6pm, for instance.

On and on down the list of Wcrypt’s corporate vicitms this pattern continues:

Nissan: factory controlled machines were infected with WCry. How easy is it to patch these systems amid what is surely a fast-paced, multi-shift, high-volume operating tempo?
German Train system: Literally computers that make the trains run on time have been hit by WCry. Trains and planes operate more than 8 hours a day, making them difficult to patch
Telefonica & Portugal Telecom: another infrastructure company that operates beyond a standard 8 hour day that got hit by WCry

I know banks & universities were hit as well, but they’re the exception that points at the rule emerging: Security is hard enough in an 8 hour a day organization. But it’s extra, extra hard when half of a 24 hour day, or even 2/3rds of a 24 hour day is off-limits for patching. Without well-understood processes, buy-in and support from management, discipline and focus on the part of a talented IT team, such high tempo operating environments will inevitably fall behind the security curve and be preyed upon by WCry and its successors.

It has demonstrated dramatically the perpetual tension between uptime, security and the incentives thereof for IT
This is similar to the patching-is-hard-in-high-tempo organizations claim, but focuses on IT incentives. For the first 2o or 30 years of Information Technology, our collective goal and mission in life was to create, build and maintain business systems that have as much uptime as possible. We call this ‘9s’ as in, “how many ya got?!?”, and it’s about the only useful objective measure by which management continues to sign our check.

Here, I’ll show you how it works:

IT Pro # 1: I got five 9s of uptime this month, that’s less than 26 seconds of unplanned downtime!

IT Pro #2: Still doesn’t touch my record in March of 2015, where I had six 9s (2.59 seconds of downtime) for this service!

Uptime is our raison d’etre, the thing we get paid to deliver the most. We do not get paid, in general, to practice our craft the right way, or the best practice way, per se. We certainly do not get paid to guard against science-fiction tales of security threats involving cyber-weapon worms that encrypt all our data.

We are paid to keep things up and running because, at the end of the day, we’re a cost center in the business. It takes a rare and unique and charismatic manager with support from the business to change that mindset, to get an organization beyond a place where it merely views IT as a cost-center and a place to call when things that are supposed to be up are down.

And that’s part of the reason why Wcry was so effective around the globe.

It has spawned a bunch of ignorant commentary from non-technical people who are outraged at Microsoft

Zeynep Tufecki, an outstanding scholar of good reputation studying the impact of technology on society wrote a piece in the NYT this weekend that had my blood boiling. Effectively, she blames Microsoft and incompetent IT teams for this mess:

First, companies like Microsoft should discard the idea that they can abandon people using older software. The money they made from these customers hasn’t expired; neither has their responsibility to fix defects. Besides, Microsoft is sitting on a cash hoard estimated at more than $100 billion (the result of how little tax modern corporations pay and how profitable it is to sell a dominant operating system under monopolistic dynamics with no liability for defects).

This is absurd on its face. She’s essentially arguing that software manufacturers extend warranties on software forever. She continues:

For example, Chromebooks and Apple’s iOS are structurally much more secure because they were designed from the ground up with security in mind, unlike Microsoft’s operating systems.

Tufecki, whom I really like and enjoy reading, is trolling us. 93% of Google’s handsets don’t run the latest Google OS, which means many people -close to a billion by my count- are, through now fault of their own, carrying around devices that aren’t up to date. Should they be supported forever too? And Apple’s iPhone, as much as I love it, can’t run an Assembly line that manufacturers cars nevermind coordinate an MRI machine.

Rubbish. Disappointed she wrote this.

For all the reasons above, Wcry is not the fault of Microsoft any more than it’s the fault of the element Copper. If anything, the fault for this lies in the way we think about and use technology as businesses and as individuals. Certainly, IT shares some of the blame in these organizations, but there are mitigating factors as I spoke about above.

Mostly, I lay the blame at the NSA for losing these damned things in the first place. If they can’t keep things secure, what hope do most IT shops have?

It has inspired at least one headline writer to say your data is safer with FaceTube than with your hospital
Again, more rubbish and uninformed nonsense from the normals. Sure, my data might be safer from third party hackers if I were to house it inside FaceTube, but then again, adtech companies might just buy that same dataset, anonymized, connect dots from that set to my online behavior dataset, and figure out who I really am. That’s FaceTube’s business, after all!

Nothing Finer than a Well-Considered Powershell Module

Kudos to Intel for recognizing & implementing a full Powershell module for their network adapters.

This is probably old news to most of you (and indeed, I think this was released in 2013) but I’ve just now managed to explore them.

How do I love them? Let me count the ways.

With IntelNetCmdlets, you no longer have to fart around with netsh cmds to get your NICs primed to push packets properly
With IntelNetCmdlets, your Network Engineering colleague in the cube next to you will no longer laugh as you suffer from Restless Finger Syndrome. RFS is characterized by furious mouse clicking interspersed with curses such as, “Goddamnit, I don’t have time to hunt through all these Device Manager menus just to input the Receive Buffer values I want! And I have four adapters! Somebody kill me. Now!”
With IntelNetCmdlets, engineers who dabble in the virtual arts now have yet another tool in the box that can reduce/eliminate human error prior to the creation of an important virtual switch in a well-considered Hyper-V infrastructure.
With IntelNetCmdlets, even your beater lab environment shines a little brighter because these babies work with my favorite NIC of all time, the I350 T-4 quad port server adapter, which you can now buy brand new (Probably a Chinese knock-off…but the drivers work!) for about $70 on eBay. Suck on that Broadcom NetExtreme and goofy BroadcomCLI!

Here’s an example of what Intel’s Net cmdlets can do for you.

Let’s say you’re building out a host in your homelab, or you just received some new Whitebox x86 servers for a dev environment at work. Now, naturally this box is going to host virtual machines, and it’s likely those VMs will be on shared storage or will be resources in a new cluster…whatever the case, proper care & raising of your physical NICs at this stage in your infrastructure project not only sets you up for success and makes you a winner, but saves potentially hours or days of troubleshooting after you’ve abstracted all this nonsense away with your hypervisor.

Of course this could all be scripted out as part of a Config Mgr task sequence, but let’s not get too fancy here! I’m no MVP and I just want you to kill your need for Device Manager and the cryptic netsh commands, ok?

Gifcam demo time. Here I’m setting the Jumbo packet value in the Windows registry for the four Intel adapters on my I350-T4 card:

jumbopacket

What I love about this is that Intel’s gone the extra-mile with their Netcmdlets. There’s a full Powershell helpfile, with extras if you tag -verbose or -examples to the end of your get-help query. Any setting you need to toggle, it’s there, from “Green Ethernet” to how many RSS queues you want, to whether VMQ is enabled or disabled.

All you need? A quality Intel card (the Pro1000 cards prior to the I350 family don’t support this officially, but you may be able to trick the Proset drivers into it!), the Proset driver package utility (here) and Powershell. Hell, you can even do this while PS Remoting!

What are you going to do with all the time I Just saved you? Cheers

Integrating Trendnet IP Cameras with my homelab

What do you get when you take an IT Systems Engineer with more time on his hands than usual and an unfinished home project list that isn’t getting any shorter?

You get this:

Daytime — My home automation/Internet of Things ‘play’

That’s right. I’ve stood-up some IP surveillance infrastructure at my home, not because I’m a creepy Big Brother type with a God-Complex, rather:

Once my 2.5 year old son figured out how to unlock the patio door and bolt outside, well, game over boys and girls….I needed some ‘insight’ and ‘visibility’ into the Child Partition’s whereabouts pronto and chasing him while he giggles is fun for only so long
My home is exposed on three sides to suburban streets, and it’s nice to be able to see what’s going on outside
I have creepy Big Brother tendences and/or God complex

I had rather simple rules for my home surveillance project:

IP cameras: ain’t no CCTV/600 lines of resolution here, I wanted IP so I could tie it into my enterprise home lab
Virtual DVR, not physical: Already have enough pieces of hardware with 16 cores, 128GB of RAM, and about 16TB of storage at home.
No Wifi, Ethernet only: Wifi from the camera itself was a non-starter for me because 1) while it makes getting video from the cameras easier, it limits where I can place them both from a power & signal strength perspective 2) Spectrum & bandwidth is limited & noisy at distance-friendly 2.4GhZ, wide & open at 5ghz, but 5 has half the range of 2.4. For those reasons, I went old-school: Cat5e, the Reliable Choice of Professionals Evereywhere
Active PoE: 802.3af as I already own about four PoE injectors and I’ve already run Cat5e all over the house
Endpoint agnostic: In the IP camera space, it’s tough to find an agnostic camera system that will work on any end-device with as little friction as possible. ONVIF is, I suppose, the closest “standard” to that, and I don’t even know what it entails. But I know what I have: Samsung GS6, iPhone 6, a Windows Tiered Storage box, four Hyper-V hosts, System Center, an XBox One and 100 megabit internet connection.
Directional, no omni-PTZ required: I could have saved money on at least one corner of my house by buying a domed, movable PTZ camera rather than use 2 directionals, but 1) this needed to work on any end-point and PTZ controls often don’t

And so, over the course of a few months, I picked up four of these babies:

Trendnet TV-IP310PI

Design

I liked these cameras from the start. They’re housed in a nice, heavyweight steel enclosure, have a hood to shade the lens and just feel solid and sturdy. Trendnet markets them as outdoor cameras, and I found no reason to dispute that.

My one complaint about these cameras is the rather finicky mount. The camera can rotate and pivot within the mount’s attachment system, but you need to be careful here as an ethernet cable (inside of a shroud) runs through the mount. Twist & rotate your camera too much, and you may tear your cable apart.

And while the mount itself is steel and needs only three screws to attach, the interior mechanism that allows you to move the camera once mounted is cheaper. It’s hard to describe and I didn’t take any pictures as I was cursing up a storm when I realized I almost snapped the cable, so just know this: be cognizant that you should be gentle with this thing as you mount it and then as you adjust it. You only have to do that once, so take your time.

Imaging and Performance:

Trendnet says the camera’s sensor & processing is capable of pushing out 1080p at 30 frames per second, but once you get into one of these systems, you’ll notice it can also do 2560×1440, or QHD resolutions. Most of the time, images and video off the camera are buttery smooth, and it’s great.

I’m not sharp enough on video and sensors to comment on color quality, whether F 1.2 on a camera like this means the same as it would on a still DSLR, or understand IR Lux, so let me just say this: These cameras produce really sharp, detailed and wide-enough (70 degrees) images for me, day or night. Color seems right too; my lawn is various hues of brown & green thanks to the heat and California drought, and my son’s colorful playthings that are scattered all over do in indeed remind me of a clown’s vomit. And at night, I can see far enough thanks to ambient light. Trendnet claims 100 foot IR-assisted viewing at night. I see no reason to dispute that.

Let the camera geeks geek out on teh camera; this is an enterprise tech blog, and I’ve already talked abou the hardware, so let’s dig into the software-defined & networking bits that make this expensive project worthwhile.

Power & Networking

These cameras couldn’t be easier to connect and configure, once you’ve got the power & cabling sorted out. The camera features a 10/100 ethernet port; on all four of my cameras, that connects to four of Trendnet’s own PoE injectors. All PoE injectors are inside my home; I’d rather extend ethernet with power than put a fragile PoE device outside. The longest cable run is approximately 75′, well within the spec. Not much more to say here other than Trendnet claims the cameras will use 5 watts maximum, and that’s probably at night when the IR sensors are on.

From each injector, a data cable connects to a switch. In my lab, I’ve got two enterprise-level switches.

One camera, the garage/driveway camera, is plugged into trunked, native vlan 410 port on my 2960s in the garage,

The other switch is a small CIsco SG-300 10p. The three other cameras connect to it. The SG-300 serves the role of access-layer switch and has a 3x1GbE port-channel back to the 2960s. This switch wasn’t getting used enough in my living room, so I moved it to my home office, where all ports are now used. Here’s my home lab environment, updated with cameras:

Like any other IP cam, the Trendnet will obtain an IP off your DHCP server. Trendnet includes software with the camera that will help you find/provision the camera on your network, but I just saved a few minutes and looked in my DHCP table. As expected, the cameras all received a routable IP, DNS, NTP and other values from my DHCP.

Once I had the IP, it was off to the races:

Set DHCP reservation
Verify an A record was created DNS so I could refer to the cameras by names rather than IP
Login, configure new password, update firmware, rename camera, turn-off UPNP, turn-off telnet
Adjust camera views

Software bits – Server Side

Trendnet is nice enough to include a fairly robust and rebadged version of Luxriot camera software, which has two primary components: Trendnet View Pro (Fat Client & Server app) and VMS Broacast server, an http server. Trendnet View Pro is a server-like application that you can install on your PC to view, control, and edit all your cameras. I say server-like because this is the free-version of the software, and it has the following limits:

Cannot run as a Windows Service
An account must be logged in to ‘keep it running’
You can install View Pro on as many PCs as you like, but only one is licensed to receive streaming video at a time

Upgrading the free software to a version that supports more simultaneously viewers is steep: $315 to be exact.

Smoking the airwaves with my beater kiosk PC in the kitchen. This is the TrendNet View client, limited to one viewer at a time

Naturally, I went looking for an alternative, but after dicking around with Zoneminder & VLC for awhile (both of which work but aren’t viewable on the XBox), I settled on VMS Broadcast server, the http component of the free software.

Just like View Pro, VMS Broadcast won’t run as a service, but, well, sysinternals!

So after deliberating a bit, I said screw it, and stood-up a Windows 8.1 Pro VM on a node in the garage. The VM is Domain-joined, which the Trendnet software ignored or didn’t flag, and I’ve provisioned 2 cores & 2GB of RAM to serve, compress, and redistribute the streams using the Trendnet fat client server piece as well as the VMS web server.

Client Side

On that same Windows 8.1 VM, I’ve enabled DLNA-sharing on VLAN 410, which is my trusted wireless & wired internal network. The thinking here was that I could redistribute via DLNA the four camera feeds into something the XBox One would be able to show on our family’s single 48″ LCD TV in the living room via the Media App. So far, no luck getting that to work, though IE on the XBox One will view and play all four feeds from the Trendnet web server, which for the purposes of this project, was good enough for me.

Additionally, I have a junker Lenovo laptop (Ideapad, 11″) that I’ve essentially built into a Kiosk PC for the kitchen/dining area, the busiest part of the house. This PC automatically logs in, opens the fat client and loads the file to view the four live feeds. And it does this all over wifi, giving instant home intel to my wife, mother-in-law, and myself as we go about our day.

Finally, both the iOS & Android devices in my house can successfully view the camera streams, not from the server, but directly (and annoyingly) from the cameras themselves.

The Impact of RTSP 1080p/30fps x 4 on Home Lab

I knew going into this that streaming live video from four quality cameras 24×7 would require some serious horsepower from my homelab, but I didn’t realize how much.

From the compute side of things, it was indeed alot. The Windows 8.1 VM is currently on Node2, a Xeon E3-1241v3 with 32GB of RAM.

Typically Node2’s physical CPU hovers around 8% utilization as it hosts about six VMs in total.

With the 8.1 VM serving up the streams as well as compressing them with a variable bit rate, the tax for this DIY Home surveillance project was steep: Node2’s CPU now averages 16% utilized, and I’ve seen it hit 30%. The VM itself is above 90% utilization.

More utilization = more worries about thermal as Node2 sits in the garage. In southern California. In the summertime.

Ambient air temperature in my garage over the last three weeks.

Node2’s average CPU temperature varies between 22c and 36c on any given warm day in the garage (ambient air is 21c – 36c). But with the 8.1 VM, Node2 has hit as high as 48c. Good thing I used some primo thermal paste!

All your Part 15 FCC Spectrum are belong to me, on channel 10 at least

From the network side, results have been interesting. First, my Meraki is a champ. The humble MR-18 802.11n access point doesn’t break a sweat streaming the broadcast feed from the VM to the Lenovo Kiosk laptop in the kitchen. Indeed, it sustains north of 21mb/s as this graph shows, without interrupting my mother in law’s consumption of TV broadcasts over wifi (separate SSID & VLAN, from the SiliconDust TV tuner), nor my wife’s Facebooking & Instagramming needs, nor my own tests with the Trendnet application which interfaces with the cameras directly.

Meraki’s analysis says that this makes the 2.4ghz spectrum in my area over 50% utilized, which probably frustrates my neighbors. Someday perhaps I’ll upgrade the laptop to a 5ghz radio.

vSwitch, the name of my Converged SCVMM switch, is showing anywhere from 2megabits to 20 megabits of Tx/Rx for the server VM. Pretty impressive performance for a software switch!

Storage-wise, I love that the Trendnets can mount an SMB share, and I’ve been saving snapshots of movement to one of the SMB shares on my WindowsSAN box.

I am also using Trendnet’s email alerting feature to take snapshots and email them to me whenever there’s motion in a given area. Which is happening a lot now as my 2 year old walks up to the cameras, smiles and says “Say cheeeese!”

All in all, a tidy & fun sub-$1000 project!

The Trouble with Cipher Suites

So I was tooling around one day in the lab, reading Ivan Ristic’s book on SSL/TLS, when I came across his advice on securing Windows-based Infrastructures from offering up the use of out-of-date/obsolete or otherwise insecure cipher suites to hosts on the other end of an https connection.

I read Ristic’s chapter a couple of times, reviewed TechNet, and selected a set of cipher suites in Group Policy in the order I wanted them used, based largely on Ristic’s text, but with a few others I knew I’d need after the policy went live. Then I pushed out the new policy, named “Strong Crypto,” to all physical, virtual and laptops in my home lab.

A few gpupdates later, I was pleased to see that nothing was broken. Schannel wasn’t showing any errors, User & Computer accounts were authenticating and getting kerb tickets, and pleasantly, my Outlook fat client didn’t even hiccup; it happily was using TLS 1.2 cipher suites to talk with my Office 365 Exchange instance.

Happy dance.

And then, two days later, I noticed it. OneDrive for Business was busted, had gone Pear Shaped, and was now totally t***-up as my English friends would say.

A couple hundred gigabytes of files no longer syncing to my Sharepoint Online site, as evidenced by these Microsoft Icons of Distress:

So, what’d I break?

I’ll get to that in a moment, but first: why would you bother with something as obscure as cipher suites and their order? I mean beyond the fact that toggling the cipher suite sounds cool?

Why Cipher Suites are Important

Cipher suites are a critical part of your AD infrastructure. They’re critical as they represent a sort of baseline set of standards that client & server negotiate over during the complicated and very important tête à tête that is the TLS/SSL handshake between client/server.

You can and should read more about TLS handshakes in this RFC, but the bottom line is this: client & server are supposed to negotiate with each other, find the most secure and common set of cipher package, and use it during the secured session.

If client & server can’t find at least one common cipher suite, you have a busted TLS connection. And that’s no bueno, unless it was your intent.

In Microsoft-land, the default set of cipher suites is pretty good. Who am I kidding, it’s an acronym rich playground of security paradigms, as evidenced by the Group Policy editor:

ciphertsuite — Holy Acronym soup, batman!

Don’t be intimidated by all the crypto terms on this screen. What you see is the list of cipher suites -and the order in which they are presented to a host- by default.

The way to read one of these cipher suites is by breaking it down into its constituent parts:

So, the Cipher suite above uses TLS as its protocol (vs SSL), can exchange keys via the Elliptic Curve Diffie Helman ephemeral mechanism, accepts an RSA x.509 certificate, and is willing to encrypt the session via the AES 256 bit block cipher. The last bit, we’ll get to in a moment.

Be cautious when modifying

Since I was doing this in my lab, I had no concern about legacy applications, but in a production environment, you’ll want to tread lightly and deliberately here. Consider:

If your’e in a typical Microsoft IT shop, you probably have a few legacy applications hanging around that may rely on old cipher suites, or vice-versa, the application server can’t use the newer cipher suites that come built into your desktops & laptops.

Take Windows Server 2003, for example. The base OS doesn’t support Elliptic Curve Diffie Helman for Key Exchange, so right off the bat, if you’ve got 2003 Hosts serving up https Sharepoint or Exchange in-house, your clients & servers will never utilize TLS_ECDHE as that suite is not common to both of them. The contrary is also true; your Windows 8.1 laptop isn’t going to support the oldest suites that your 2003 server does; TLS_RSA_WITH_DES_CBC_SHA is never going to be the cipher suite watering hole your clients/servers meet around ((thank Goodness!!)) unless you go out of your way to make it happen.

The lesson here is that old cipher suites never die, dependency on them just fades away as your modernize/replace your legacy in-house applications with modern, streamlined, and properly TLS-secured ones. So be cautious, lest you break a legacy application.

You might be thinking I’m full of great advice, yet I still managed to wreck my OneDrive for Business sync app. And you’d be right!

So what happened?

Essentially, I broke my little OneDrive for Business sync app because I didn’t include SHA1 as possible hash algorithm in any of the cipher suites I selected.

By leaving SHA1 out of my cipher suites, OneDrive for Business couldn't find common ground with Sharepoint Online, which broke my OneDrive Sync. — By leaving SHA1 out of my cipher suites, OneDrive for Business couldn’t find common ground with Sharepoint Online, which broke my OneDrive Sync.

And SHA1 is used by Microsoft IT ((as a side note, it’s really awesome to see Microsoft IT’s PKI, built out as it should be. Here’s a PKI serving not just Microsoft internal employees -all 100k of them- but millions of customers. If Microsoft IT can build a PKI to that scale, surely you and I can build one for the users dependent on us!)) in at least two places: as the Signature Hash algorithm on the root certificate of my Sharepoint site, and as the hashing mechanism for the Thumbprint on *.sharepoint.com certificate.

Had I visited my Sharepoint site in IE, I would likely have seen an error message in my browser; but I use Opera normally, and Opera -like Chrome & Firefox- have cipher suites apart from Windows’ so I never saw an error.

Adding the strongest cipher suite that included SHA1 fixed the error right away. ((Interesting aside: Google, and many security researchers, consider SHA1 to be end-of-life as it is now, or will be very soon now, computationally feasible to crack it, if that’s the right word. Google wants to sunset SHA1 in its browser this year; Ivan Ristic’s site will give https sites that use SHA1 a D- rating by the end of 2015. Microsoft IT, meanwhile, still uses it in production, but plans to deprecate it at the end of 2016. What gives? You could say there’s a pissing match between these leviathans of technology, or that one is trying to screw the other. But in essence, all parties agree SHA1 should fade away, they just differ on how aggressive deprecation efforts should be.))

Fixed Wireless is the WAN builder’s best friend

This is Joe. He's an American hero. — This is Joe. He’s an American hero.

Just how hard is it in 2015 to order & deploy a cheap commodity internet circuit to connect a remote office/branch office (ROBO) to the rest of your corporate WAN via the internet? ((Commodity = business class internet, something less reliable but orders of magnitude less expensive than a traditional private line, T1, or managed MPLS circuit. Commodity also means fat, dumb internet pipe, a product that cable internet companies consider an existential threat))

Pretty damned hard.

Why so difficult Jeff?!? you’re thinking. I stand-up tunnels and tear them down all day long, I route/switch in my sleep and verily I say unto you that my packets always find their way home, tags intact, whether on the WAN, between switch closets in the campus, or between nodes in the datacenter!

Verily they do indeed, and I salute you, you herder of stray packets!

It’s not that the technology connecting core to branch is hard or difficult, no, what I’m bitching about today is connecting the branch site to the internet in the first place.

It’s layer 1, stupid.

Truly, ordering internet service for a small or even medium-sized branch office is one of the most painful exercises in modern IT.

Here, let me show you:

You Bing/Google various iterations of “Lake Winnepesaukah ISPs,” , “Punxatawney Packet Delivery,” , “Broadband Service in Topeka,” “Ethernet over Copper + Albuquerque,” “Business Cable Internet – Pompano Beach, FL” and such. Dismissing the spam URL results on Page 1-12, you eventually arrive at Comcast, Time Warner, or ~~Charter~~ nee Spectrum Business, or whatever little coax fiefdom has carved out a franchise at the edge of your business. You visit their website, click “Business” and fight your way through pop-ups and interstitials to a page that says it can verify service at your branch office’s address.
Right, you think, I’ll just Tab-tab my way through this form, input my branch office address here, punch that green submit button there, and get these nasty Layer 1 bits out of the way. But this isn’t the old days of 2009 when you could order a circuit online or at least verify service…oh no, no sir, this is the future…this is 2015. In 2015, you see, the Cable providers demand audience with you, so that they can add value.
Pay the Last Mile Toll: So you surrender your digits and wait for a phone call. When it rings 36-72 hours later, you’re determined to keep it short. What you want is a simple yes/no on service at your ROBO, or an install date, but what you get is a salesperson who can’t spell TCP/IP and wants to sell you substandard VoIP & TV. “Will you be uploading or downloading with this internet connection?” is just one of the questions you’ll suffer through to mollify the last mile gatekeepers standing between you and #PacketGlory on the WAN.
At long last, install day arrives: You’ve drop-shipped the edge router/overlay device, you’ve coordinated with the L-con, and the CableCo tech is on site at your ROBO to install your circuit. Hallalelujah, you think, as you wait for the tunnel to come up. But it never does, because between your awesome zero-touch edge device & your datacenter lies some crazy bespoke 2Wire gateway device that NATs or offers up a free wifi connection to the public on your dime. Another phone call, another fight to get those things turned off.

Nuts to all that, I say.

This is America jack, and the great thing about America is choice. Even when you don’t have choice (and you don’t in the case of cable franchises & municipalities), all you may need is line of sight to one of these things:

That’s right. Fixed wireless, baby. I’m hot on fixed wireless in 2015. It’s everything CableCo isn’t. It’s:

Friction free: In place of the coax fiefdoms and gatekeepers, the 1-800 numbers, and the aggressive salespeople, there’s just Joe, a real engineer at a local fixed wireless ISP. Joe’s great because Joe’s local, and Joe takes your order, gives you his mobile, installs the antenna at your branch, and hands you a blue wire with three static IPs.
Super-fast to deploy. You want internet at your ROBO? Well guess what? It’s already there, you just need the equipment to catch it.
More reliable than it used to be: Now of course this all depends on the application you’re trying to deliver to your ROBO, but I’ll say this: Fixed Wireless has improved. You don’t need to fear (as much) a freak snowstorm, a confused flock of Canada Geese, or rain. For a small ROBO, a fixed wireless connection might be enough to serve as the primary WAN link. For larger ROBOs, I think the technology is mature enough to serve as a secondary WAN link, or even your primary Internet circuit. ((Routing business traffic over the expensive wired link and internet over the cheap fixed wireless link is a recipe I’d recommend all day long and twice on Sundays ))
As Secure as Anything Else These Days: How difficult would it be to perform a man in the middle attack via interception of a fixed wireless connection? I’m not sure, to be honest, but if you aren’t encrypting your data before it leaves your datacenter, you have a whole lot more to worry about than a blackhat with a laptop, a stick, and a microwave antenna.
Cost competitive: I’ve deployed a couple of fixed wireless connections and I find the cost to be very competitive with traditional cable company offerings. Typically you’ll pay about $200 for the antenna install, but unlike the fee Comcast would charge you to install their modem, I think this is justified as it involves real labor and a certain amount of risk.
Regional/Hyper-local but still innovative: For whatever reason, fixed wireless ISPs have proven resistant to the same market forces that killed off your local dial-up/DSL ISP. Yet this isn’t a stagnant industry; quite the opposite in fact, with players like Ubiquiti Networks releasing new products.

I’ve been working on the WAN a lot lately and I’ve deployed two fixed wireless circuits at ROBOs. If you’ve got similar ROBO WAN pains, you should have a look at fixed wireless, you might be surprised!

Find Office problems before they find you with Telemetry server

I’ve not always had a bromance with Microsoft’s Office suite. I cut my word processing teeth on WordPerfect 5.1, did most of my undergrad papers in BeOS’ one productivity suite ((GoBe Productive, still the best Office suite name)) , and touch-typed my way to graduating cum laude in grad school with countless Turabian-style Google Docs papers.

Office?

That was for corporate suits, man. Rich corporate suits.

But all that’s ancient history. Or maybe I’ve become a suit. Either way, I’m loving Office today.

In 2015, Office has transformed into the ultimate agnostic git ‘r done productivity package. It’s free to use in many cases, but if you want to ‘own’ it, you can subscribe to it, just like HBO ((For the IT Pro, this is a huge advantage, as a cheap E-class sub gives you access to your own Exchange instance, your own Sharepoint server, and your own Office tenant. It’s awesome!)) . It’s also available on just about any device or computing system you can think of, works just as well inside a browser as Google Docs does, and has an enormous install base.

telemetry — From the Office Telemetry PDF guide, linked below

Office has become so impressive and so ubiquitous that it’s truly a platform unto itself, consumed a la carte or as part of a well-balanced Microsoft meal. I’m bullish on Windows but if Office’s former partner ever sunsets, I’m convinced my kid and his kid will still grow up in an Office world.

All of that makes Office really important for IT, so important that you as an IT Guy should consider standing-up some easy instrumentation around it.

Enter Office Telemetry, a super-simple package that flows your Office data to a SQL collector, mashes it up, and gives you important insight into how your users are using Office. It also surfaces the problems in Office -or Office documents- before your users do, and it’s free.

Oh, did I mention it’s called Office Telemetry? This thing makes you feel like an astronaut when you’re using it!

Here’s how you deploy it. Total time: about an hour.

Download the Office 2013 ADMX/ADML files for Group Policy and deploy them to your Domain Controllers.
Spin-up a 2008 R2 or 2012 VM, or find a modestly-equipped physical box that at least has Windows Management Framework 3.0/Powershell 3.0 on it. If it has a SQL 2012 instance on it that you can use, even better. If not, don’t stress and proceed to the next step.
Set-aside a folder on a separate volume (ideally) for the telemetry data. If you’re going to flow data from hundreds of Office users, plan for a minimum of 5-25 megabytes per user, at a minimum.
- If your users are on the WAN, plan accordingly. Telemetry data is pretty lightweight (50k chunks for older Office clients, 64k chunks for Office 2013)
Install Office ProPlus 2013 or 365 on the VM. You do not need to use an Office 365 license for it to run.
Download the Deploy Office Telemetry powershell script package from TechNet or via Script Browser in Powershell ISE.
Because it’s a script, you’ll need to temporarily change your server’s execution policy, self-sign it, or configure Group Policy as appropriate to run it. TechNet has instructions.
Run the script; it will download SQL 2012 express and install it for you if you don’t have SQL. It will also set proper SMB read/modify permissions on that folder you set up earlier.
As if that wasn’t enough, the script will give you a single registry keyfile you can use to deploy to your user’s machines.
But I prefer the Group Policy/SCCM route. Remember the ADMX files you deployed? Flip the switches as appropriate under User Configuration>Administrative Templates>Microsoft Office 2013> Telemetry Dashboard.
Sit back, and watch the data flow in, and pat yourself on the back because you’re being a proactive IT Pro!

As I’ve deployed this solution, I’ve found broken documents, expensive add-ons that delay Office, and multiple other issues that were easy to resolve but difficult to surface. It’s totally worth your time to install it.

Office Telemetry PDF

Sign of the Times or just the best PKI book ever?

Like a lot of IT Pros, I’ve been studying up on security topics lately, both as a reaction to the increasing amount of breach news (Who got breached this week, Alex?) and because I felt weak in this area.

So, I went shopping for some books. My goals were simply to get a baseline understanding of crypto systems and best-practice guidance on setting up Microsoft Public Key Infrastructures, which I’ve done in the past but without much confidence in the end result.

Well, it turns out there’s not a whole lot of literature on Microsoft PKI systems. It seems the best of the genre is Windows Server 2008 PKI & Certificate Security, a Microsoft Press book published in 2008 and authored by Brian Komar:

This 3.2lb, 800 page book has a 4.9 out of 5 star rating on Amazon, with reviewers calling it the best Microsoft PKI guide out there.

Great! I thought, as I prepared to shell out about $80 and One Click my way to PKI knowledge.

That’s when I noticed that the book is out of print. There are digital versions available from O’Reilly, but it appears most don’t know that.

For the physical book itself, the least expensive used one on Amazon is $749.99. You read that right. $750!

If you want a new copy, there’s one available on Amazon, and it’s $1000.

I immediately jumped over to Camelcamelcamel.com to check the history of this book, thinking there must have been a run on Mr. Komar’s tome as Target, Home Depot, JP Morgan, and Sony Pictures fell.

Result:

The price of this book has spiked recently, but Peak PKI was a full three years ago.

I looked up security breaches/events of early 2012. Now correlation != causation, but it’s interesting nonetheless. Hopefully this means there’s a lot of solid Microsoft PKI systems being built out there!

Rather than shell out $750 for the physical book, I decided to get Ivan Ristic’s fantastic Bulletproof SSL/TLS, which I highly recommend. It’s got a chapter on securing Windows infrastructure, but is mostly focused on crypto theory & practical OpenSSL. I’ll buy Komar’s as a digital version next or wait for his forthcoming 2012 R2 revision.

Big Data for Server Guys : Azure OpsInsight Review

Maybe it’s just my IT scars that bias me, but when I hear a vendor push a “monitoring” solution, I visualize an IT guy sitting in front of his screen, passively watching his monitors & counters, essentially waiting for that green thing over there to turn red.

He’s waiting for failure, Godot-style.

That’s not a recipe for success in my view. I don’t wait upon failure to visit, I seek it out, kick its ass, and prevent it from ever establishing a beachhead in my infrastructure. The problem is that I, just like that IT Guy waiting around for failure, am human, and I’m prone to failure myself.

Enter machine learning or Big Data for Server Guys as I like to think of it.

Big Data for Server Guys is a bit like flow monitoring on your switch. The idea here is to actively flow all your server events into some sort of a collector, which crunches them, finds patterns, and surfaces the signal from the noise.

Big Data for Server Guys is all about letting the computer do what the computer’s good at doing: sifting data, finding patterns, and letting you do what you are good at doing: empowering your organization for tech success.

But we Windows guys have a lot of noise to deal with: Windows instruments just about everything imaginable in the Microsoft kingdom, and the Microsoft kingdom is vast.

So how do we borrow flow-monitoring techniques from the Cisco jockeys and apply it to Windows?

Splunk is one option, and it’s great: it’s agnostic and will hoover events from Windows, logs from your Cisco’s syslog, and can sift through your Apache/IIS logs too. It’s got a thriving community and loads of sexy, AJAX-licious dashboards, and you can issue powerful searches and queries that can help you find problems before problems find you.

It’s also pretty costly, and I’d argue not the best-in-class solution for Hoovering Windows infrastructure.

Fortunately, Microsoft’s been busy in the last few years. Microsoft shops have had SCOM and MOM before that, but now there’s a new kid in town ((He’s been working out and looks nothing like that the old kid, System Center Advisor)) : Azure Operational Insights, and OpsInsight functions a lot like a good flow collector.

And I just put the finishing touches on my second Big Data for Server Guys/OpsInsight deployment. Here’s a mini-review:

The Good:

It watches your events and finds useful data, which saves you time: OpsInsight is like a giant Hoover in the sky, sucking up on average about 36MB/day of Windows events from my fleet of nearly ~150 VMs in a VMware infrastructure. Getting data on this fleet via Powershell is trivial, but building logic that gives insight into that data is not trivial. OpsInsight is wonderful in this regard; it saves you from spending time in SSRS, Excel, or diving through the event viewer haystack MMC or via get-event looking for a nugget of truth.
It has a decent config recommendation engine: If you’re an IT Generalist/Converged IT Guy like me, you touch every element in your Infrastructure stack, from the app on down to the storage array’s rotating rust. And that’s hard work because you can’t be an expert in everything. One great thing about OpsInsight is that it saves you from searching Bing/Google (at worst) or thumbing through your well-worn AD Cookbook (at best) and offers Best practice advice and KB articles in the same tab in your browser. Awesome!
Thanks Opsinsight for keeping me out of this thing

Query your data rather than surfing the fail tree: Querying your data is infinitely better than walking the Fail Tree that is the Windows Event Viewer looking for errors. OpsInsight has a powerful query engine that’s not difficult to learn or manipulate, and for me, that’s a huge win over the old school method of Event Viewer Subscriptions.
Dashboards you can throw in front of an executive: I can’t understate how great it is to have automagically configured dashboards via OpsInsight. As an IT Pro, the less time I spend in SSRS trying to build a pretty report the better. OpsInsight delivers decent dashboards I’m proud to show off. SCOM 2012 R2’s dashboards are great, but SCOM’s fat client works better than its IIS pages. Though it’s Silverlight-powered, OpsInsight wins the award for friction-free dashboarding.
Flexible Architecture: Do you like SCOM? Well then OpsInsight is a natural fit for you. I really appreciate how the System Center team re-structured OpsInsight late last year: you can deploy it at the tail end of your SCOM build, or you can forego SCOM altogether and attach agents directly to your servers. The latter offers you speed in deployment, the former allows you to essentially proxy events from your fleet, through your Management Group, and thence onto Azure. I chose the latter in both of my deployments. Let OpsInsight gate through SCOM, and let both do what they are good at doing.
It’s secure: The architecture for OpsInsight is Azure, so if you’re comfortable doing work in Azure Storage blobs, you should be comfortable with this. That + encrypted uploads of events, SCOM data and other data means less friction with the security/compliance guy on your team.

The Bad:

It’s silverlight, which makes me feel like I’m flowing my server events to Steve Ballmer: I’m sure this will be changed out at some point. I used to love Silverlight -and maybe there’s still room in my cold black heart for it- but it’s kind of an orphan media/web child at the moment.
There’s no app for iOS or Android…yet: I had to dig out my 2014 Lumia Icon just to try out the OpsInsight app for Windows phone. It’s decent, just what I’d like to see on my 2015 Droid Turbo. Alas there is no app for Android or IOS yet, but it’s the #1 and #2 most requested feature at the OpsInsight feedback page (add your vote, I did!)
It’s only Windows at the moment: I love what Microsoft is doing with Big Data crunching; Machine Learning, Stream Analytics and OpsInsight. But while you can point just about any flow or data at AzureML or Stream Analytics, OpsInsight only accepts Windows, IIS, SQL,Sharepoint, Exchange. Which is great, don’t get me wrong, but limited. SCOM at least can monitor SNMP traps, interface with Unix/Linux and such, but that is not available in OpsInsight. However, it’s still in Preview, so I’ll be patient.
It’s really only Windows/IIS/SQL/Exchange at the moment: Sadface for the lack of Office 365/Azure intelligence packs for OpsInsight, but SCOM will do for now.
Pricing forecast is definitely…cloudy: Every link I find takes me to the general Azure pricing page. On the plus side, you can strip this bad boy down to the bare essentials if you have cost pressures.

The Ugly:

Where are my cmdlets? My interface of choice with the world of IT these days is Powershell ISE. But when I typed get-help *opsinsight, only errors resulted. How’d this get past Snover’s desk? All kidding aside, SCOM cmdlets work well enough if you deploy OpsInsight following SCOM, and I’m sure it’s coming. I can wait.

All in all, this is shaping up to be a great service for your on-prem Windows infrastructure, which, let’s face it, is probably neglected.

System Center MVP Stanislav Zhelyazkov has a great 9-part deep dive on OpsInsight if you want to learn more.

Disclosures

I received no monetary compensation otherwise, save for the swag listed below

CyberArk swag I took home:

A ballpoint pen

Morpheus Data – Background

It’s Glue, basically. But well-articulated & rational glue

Concluding Thoughts

Disclosures

Morpheus Data shwag I took home

Cool stickers

A t-shirt