Defending IT amidst the novel WannaCry worm

It’s been a hell of a few days here in the trenches of Information Technology in 2017. Where to begin?

Between explaining how this all works to concerned friends & family, answering my employer’s questions about our patching posture & status, and reading the news & analysis, I think it’s safe to say that WCry has been in my thoughts for every one of the last 72 hours, including the 24 hours of Mother’s Day and all the hours I spent in restless slumber.

Yes, that’s right. WCry was on my mind even as I celebrated Mother’s day for the three women I’m close to in my life who are mothers. Wow. Just wow.

Having had the chance to catch my breath, I’ve got some informed observations about this global incident from my perspective as an IT Pro. Why is WCry as interesting & novel as it is potent and effective in 2017? And is there any defense of an IT team one might make if their organization got pwned by WCry?

I contemplate both questions below.

WCry successfully chains a social engineering attack with a technical exploit resulting in automated organization pwnage
WCry begins as a social engineering/phishing attack on users in the place they love and hate by equal measure: their Inbox. Using Subject lines that draw the eye, the messages include malicious attachments. This facet of WCry is not new of course…..it’s routine and has been in IT for at least two decades.

How WannaCry works

Once the attachment is clicked, WCry pivots, unleashing an NSA-built cyberweapon upon the enterprise by scanning port 445 across the local /24, cycling through cached RDP accounts and calling special attention to SQL & Exchange services, presumably to price the ransom accordingly.

Then it encrypts. Nearly everything.

All of this from a single email opened by a gullible user.

This behavior -socially engineered attack on human meatbag + scan + pivot to the rest of the network- is also not novel, new or remarkable.  In fact, security Pros call this behavior “moving laterally” through an enterprise and they usually talk about it being done from “jump box” or “beach head” that’s been compromised via social engineering. Typically, security pros will reserve those terms to describe the behavior of a skilled & hostile hacker meatbag intent on pwning a targeted organization.

Where WCry is novel is that it in effect automates the hacker out of the picture, making the whole org pwnage process way more efficient. This is Organization-crippling, self-replicating malware at scale. Think Sony Pictures 2014, applied everywhere automatically minus the North Korean hacker units at the keyboard.

 

The red Wcry “Ooops” message is both informative and visually impressive, which multiplies its influence beyond its victims
As these things go, I couldn’t help but be impressed with Wcry’s incredibly detailed and anxiety-inducing UI announcing a host’s Wcry infection:

This image, or some variant thereof, has appeared on everything from train station arrival/departure boards to manufacturing floor PCs to hospital MRIs to good old-fashioned desktop PCs in Russia’s Interior Ministry. The psychological effects of seeing this image on infected hardware, then seeing it again on popular social media sites, the evening news, and newspapers around the world over the last few days are hard to determine, but I know this: this had an effect on normal consumers and users of technology across the globe. Sitting on my lap Saturday, my four year old saw the image in my personal OneNote pastebin and asked me, “Daddy, is that an alarm? Why does it show a lock? Do you have key?”

What’s interesting is that while computer users saw this or a screensaver version of this image, in reality you could click past it or minimize it in some way. Yet images of this application have proliferated on Twitter, FaceTube and elsewhere. Ransomware used to just announce itself in the root of your file share or your c:\user\username\documents folder: now it poses for screen caps and cell phone pics which multiplies its effectiveness as a PsyOps weapon. By Saturday I was reading multiple articles in my iPad’s Apple News about how regular people could protect themselves from the ‘global cyberattack.’

Its function is not just about encrypting file shares like earlier ransomware campaigns, but about owning Enterprises
If my organization or any organization I was advising got hit by WCry, my gut feeling is that I wouldn’t feel secure about my Forest/Domain integrity until I burned it down and started over. Why? Well, big IT security organizations like Verizon’s Enterprise Security group typically don’t classify ransomware as a ‘data breach’ event. Yet, as we know, Wcry installs a Pulsar backdoor that enables persistent access in the future. This feels like a very effective escalation of what it means to be ransomed in modern IT organizations, so yeah, I wouldn’t feel secure until our forest/domain was burned to the ground.

It is the manifestation of a Snoverism : Today’s nation-state cyberweapon is tomorrow’s script-kiddie attack
I was listening to the father of Powershell, Jeff Snover once and he implanted yet another Snoverism in my brain.  He said, paraphrasing here, that Today’s nation-state attack is tomorrow’s script-kiddie attack. What the what?

Jeff Snover, speaker of wisdom

Let’s unpack: the democratization of technology, the shift to agile, DevOps, and other development disciplines along with infrastructure automation has lead to a lot of great things being developed, released and consumed by users very quickly. In the consumer world this has been great -Alexa is always improving with new skills…Apple can release security patches rapidly, and FaceTube can instantly perform A/B testing on billions of people simultaneously. But not well understood by many is the fact that Enterprises and even individuals can harness these tools and techniques to instantly build and operate data systems globally, to get their product, whatever it may be, to market faster. The classic example of this is Shadow IT, wherein someone in your finance team purchases a few seats on Salesforce to get around the slow & plodding IT team.

I think Snover was observing that bad guys get the same benefits from modern technology techniques & the cloud as consumers and business users do.

And as I write this on Monday, what are we seeing? WCry is posted on GitHub and new variants are being created without the kill-switch/sandbox detection domain. Eternal Blue, the component of Wcry that exploits SMB1, was literally just a few months ago a specialized tool in the NSA’s cyber weapons arsenal. By tomorrow it will be available to any kid who wants it, or, even worse, as a push-button turn-key service anybody can employ against anybody else.

The democratization of technology means that no elite or special knowledge, techniques or tools are required to harness technology to some end. All you need is motive and motivation to do things at scale. This week, we learned that the democratization of technology is a huge double-edged sword.

It was blunted by a clever researcher for about $11
Again on the democratization of technology front, I find it fascinating that MalwareTech was able to blunt this attack by spending $11 of his own money to purchase the domain he found encoded in the output of his decompile. He’s the best example of what a can-do technologist can do, given the right amount of tools and freedom to pursue his craft.

It has laid bare the heavy costs of technical debt for which there is no obvious solution
Technical debt is a term used in software engineering circles and computer science curricula, but I also think it can and should apply to infrastructure thinking. What’s technical debt? Take it away Wikipedia:

Technical Debt is a metaphor referring to the eventual consequences of poor system design, software architecture, or software development within a codebase. The debt can be thought of as work that needs to be done before a particular job can be considered proper or complete. If the debt is not repaid, then it will keep on accumulating interest, making it hard to implement changes later on.

I can’t tell you how many times and at how many organizations I’ve seen this play out. Technical Debt, from an IT Pro’s perspective, can be the refusal to correct a misconfiguration of an important device upon which many services are dependent, or it can be a poorly-designed security regime that takes bad practice and cements it into formal process & habit, or it can be a refusal to give IT the necessary political cover & power to change bad practices or bad design into something durable and agile, or it can be refusing to patch your systems out of fear or a desire to kick the can down the road a bit.
Over time, efforts will be made to pay that technical debt down, but unless a conscious effort is made consistently to keep it low, technical debt eventually -inevitably- becomes just as crippling to an organization as credit card debt becomes to a consumer. Changes to IT systems that in other organizations are routine & easy become hard and difficult; and hard changes in other companies are close to impossible in yours.

This is a really bad place to be for an IT Pro, and now WCry made it even worse by exploiting organizations that have high technical debt, particularly as it relates to patching. Indeed, it’s almost as if the author of this malware understood at a basic fundamental level how much technical debt organizations in the real world carry.

There is no obvious solution to this. We can’t force people to use technology a certain way, or even to think of technology in a certain way. The point of going into business is to make money, not to build durable & secure and flexible technology systems, unless that is your business. Cloud services are the obvious answer, but they can’t do things like run MRI machines or interface with robots on the Nissan assembly line. At least not yet. And nobody wants regulation, but that’s a topic for another post.

It has shown how hard it is to maintain & patch systems that are in-use for more than a typical workday
If we ignore the way WCry rampaged through Russia, China and other places where properly licensing your software is considered optional, something else interesting emerges: the organizations that were hardest hit by Wcry were ones in which technology is likely in use beyond the standard 8 hour workday, which likely makes patching those technology systems all the more difficult.

While reporting on the NHS fiasco has zoomed in on the fact that the UK’s healthcare system had Windows XP widely deployed, I don’t think that tells the whole story, even if it’s true that 100% of NHS systems ran XP, it still doesn’t tell the whole story.  I can easily see how patching in such environments could be difficult based on how much those systems are used.  Hospitals and even out-patient facilities typically operate more than 8 hours a day; finding a slot of time in a given 24 hour period in which you can with the consent of the hospital, offline healthcare devices like MRI machines to update & reboot them is probably more difficult than it is in a company where systems are only required to be up between 7am and 6pm, for instance.

On and on down the list of Wcrypt’s corporate vicitms this pattern continues:

  • Nissan: factory controlled machines were infected with WCry. How easy is it to patch these systems amid what is surely a fast-paced, multi-shift, high-volume operating tempo?
  • German Train system: Literally computers that make the trains run on time have been hit by WCry. Trains and planes operate more than 8 hours a day, making them difficult to patch
  • Telefonica & Portugal Telecom: another infrastructure company that operates beyond a standard 8 hour day that got hit by WCry

I know banks & universities were hit as well, but they’re the exception that points at the rule emerging: Security is hard enough in an 8 hour a day organization. But it’s extra, extra hard when half of a 24 hour day, or even 2/3rds of a 24 hour day is off-limits for patching. Without well-understood processes, buy-in and support from management, discipline and focus on the part of a talented IT team,  such high tempo operating environments will inevitably fall behind the security curve and be preyed upon by WCry and its successors.

It has demonstrated dramatically the perpetual tension between uptime, security and the incentives thereof for IT
This is similar to the patching-is-hard-in-high-tempo organizations claim, but focuses on IT incentives. For the first 2o or 30 years of Information Technology, our collective goal and mission in life was to create, build and maintain business systems that have as much uptime as possible. We call this ‘9s’ as in, “how many ya got?!?”, and it’s about the only useful objective measure by which management continues to sign our check.

Here, I’ll show you how it works:

IT Pro # 1: I got five 9s of uptime this month, that’s less than 26 seconds of unplanned downtime!

IT Pro #2: Still doesn’t touch my record in March of 2015, where I had six 9s (2.59 seconds of downtime) for this service!

Uptime is our raison d’etre, the thing we get paid to deliver the most. We do not get paid, in general, to practice our craft the right way, or the best practice way, per se. We certainly do not get paid to guard against science-fiction tales of security threats involving cyber-weapon worms that encrypt all our data.

We are paid to keep things up and running because, at the end of the day, we’re a cost center in the business. It takes a rare and unique and charismatic manager with support from the business to change that mindset, to get an organization beyond a place where it merely views IT as a cost-center and a place to call when things that are supposed to be up are down.

And that’s part of the reason why Wcry was so effective around the globe.

It has spawned a bunch of ignorant commentary from non-technical people who are outraged at Microsoft

Zeynep Tufecki, an outstanding scholar of good reputation studying the impact of technology on society wrote a piece in the NYT this weekend that had my blood boiling. Effectively, she blames Microsoft and incompetent IT teams for this mess:

First, companies like Microsoft should discard the idea that they can abandon people using older software. The money they made from these customers hasn’t expired; neither has their responsibility to fix defects. Besides, Microsoft is sitting on a cash hoard estimated at more than $100 billion (the result of how little tax modern corporations pay and how profitable it is to sell a dominant operating system under monopolistic dynamics with no liability for defects).

This is absurd on its face. She’s essentially arguing that software manufacturers extend warranties on software forever. She continues:

For example, Chromebooks and Apple’s iOS are structurally much more secure because they were designed from the ground up with security in mind, unlike Microsoft’s operating systems.

Tufecki, whom I really like and enjoy reading, is trolling us. 93% of Google’s handsets don’t run the latest Google OS, which means many people -close to a billion by my count- are, through now fault of their own, carrying around devices that aren’t up to date. Should they be supported forever too? And Apple’s iPhone, as much as I love it, can’t run an Assembly line that manufacturers cars nevermind coordinate an MRI machine.

Rubbish. Disappointed she wrote this.

For all the reasons above, Wcry is not the fault of Microsoft any more than it’s the fault of the element Copper. If anything, the fault for this lies in the way we think about and use technology as businesses and as individuals. Certainly, IT shares some of the blame in these organizations, but there are mitigating factors as I spoke about above.

Mostly, I lay the blame at the NSA for losing these damned things in the first place. If they can’t keep things secure, what hope do most IT shops have?

It has inspired at least one headline writer to say your data is safer with FaceTube than with your hospital
Again, more rubbish and uninformed nonsense from the normals. Sure, my data might be safer from third party hackers if I were to house it inside FaceTube, but then again, adtech companies might just buy that same dataset, anonymized, connect dots from that set to my online behavior dataset, and figure out who I really am. That’s FaceTube’s business, after all!

Using Powershell and IPInfo.io to Ease the Burden of WAN management

Imagine for a moment that you are an IT Professional charged with the care, feeding, and security of a classic Wide Area Network (WAN). Further, assume that, like any properly-designed WAN, your remote networks (whether MPLS or classic Hub-spoke) egress their internet connections directly, that is to say, internet traffic from remote networks isn’t back-hauled to your datacenter or HQ.

In such a scenario, you will need to have a list of each remote network’s public IP address and other pertinent details in order to manage routing and security at each branch. In my case, I needed up-to-date public IP address information in order to properly segment & report on internet traffic traversing our SSL/TLS proxy inspection service, Zscaler.

So how would you do this? An earlier version of myself, say 15 years ago, would respond this way:

I’d remote desktop to a node in each remote network, open up a browser window, and visit IPChicken.com. Then I’d carefully copy/paste the IP address details into my Excel document, and happy days! – Jeff, 15 years ago

Wrong answer, Jeff from 15 years ago! That’s bad practice, takes way too much time, involves using the cursed mouse, and is fraught with security risk because it involves browser use.

Fortunately, there is a much better, simpler, faster and more secure way to do this. Even better, it involves my favorite tool in the world, Powershell, as well as IPInfo.io, a web service that blows IPChicken.com out of the water.

Best of all, you can do it all without your hands ever leaving your keyboard. Check it out

Let’s use Powershell’s invoke-webrequest cmdlet to see what IPInfo.io returns to us:

Nice! As you can see, IPInfo.io returns to us an HTTP content-type of application/json, which stands for JavaScript Object Notation.

JSON, if you’re not familiar with it, is an open standard that has superseded-in practice- XML and other structured document standards. It’s in widespread use across the internet, and it’s really great for us Windows admins that IPINfo.io feeds us a JSON response to our query. Why?

Because we’ve got Powershell to make it look pretty for us! We just need to pipe the results of the invoke-webrequest command into the handy convertfrom-json cmdlet. Voila!

This is great, now I’ve got high-quality IP Information on my workstation. So how do I scale this out to my remote WAN networks? how do I get the public IP address of my Lake Winnepesaukee branch office using Powershell?

Assuming you’ve got a Windows domain and have configured Windows Remote Management in a secure fashion, the way to do this is simple. Let’s use Powershell to tell a WIndows node at each branch to fetch us the public IP address it’s sitting behind, format it in a pretty way, and bring it back to my beautiful blue console. In fact, let’s do all the branches at once by using invoke-command:

Boom! That’s how we do it in 2017! It took less than 20 seconds to invoke our simple invoke-webrequest + convertfrom-json command across five remote hosts. No remote desktop needed….all of it done securely via secure WinRM which I’ve set up my nodes to listen for.

With these results in your console, it’d be trivially easy to dump out each WAN’s public IP information into a CSV, or, even better, create a new Excel spreadsheet using new-comobject and save/send the information from there.