Imagine for a moment that you are an IT Professional charged with the care, feeding, and security of a classic Wide Area Network (WAN). Further, assume that, like any properly-designed WAN, your remote networks (whether MPLS or classic Hub-spoke) egress their internet connections directly, that is to say, internet traffic from remote networks isn’t back-hauled to your datacenter or HQ.
In such a scenario, you will need to have a list of each remote network’s public IP address and other pertinent details in order to manage routing and security at each branch. In my case, I needed up-to-date public IP address information in order to properly segment & report on internet traffic traversing our SSL/TLS proxy inspection service, Zscaler.
So how would you do this? An earlier version of myself, say 15 years ago, would respond this way:
I’d remote desktop to a node in each remote network, open up a browser window, and visit IPChicken.com.Then I’d carefully copy/paste the IP address details into my Excel document, and happy days! – Jeff, 15 years ago
Wrong answer, Jeff from 15 years ago! That’s bad practice, takes way too much time, involves using the cursed mouse, and is fraught with security risk because it involves browser use.
Fortunately, there is a much better, simpler, faster and more secure way to do this. Even better, it involves my favorite tool in the world, Powershell, as well as IPInfo.io, a web service that blows IPChicken.com out of the water.
Best of all, you can do it all without your hands ever leaving your keyboard. Check it out
Let’s use Powershell’s invoke-webrequest cmdlet to see what IPInfo.io returns to us:
JSON, if you’re not familiar with it, is an open standard that has superseded-in practice- XML and other structured document standards. It’s in widespread use across the internet, and it’s really great for us Windows admins that IPINfo.io feeds us a JSON response to our query. Why?
Because we’ve got Powershell to make it look pretty for us! We just need to pipe the results of the invoke-webrequest command into the handy convertfrom-json cmdlet. Voila!
This is great, now I’ve got high-quality IP Information on my workstation. So how do I scale this out to my remote WAN networks? how do I get the public IP address of my Lake Winnepesaukee branch office using Powershell?
Assuming you’ve got a Windows domain and have configured Windows Remote Management in a secure fashion, the way to do this is simple. Let’s use Powershell to tell a WIndows node at each branch to fetch us the public IP address it’s sitting behind, format it in a pretty way, and bring it back to my beautiful blue console. In fact, let’s do all the branches at once by using invoke-command:
Boom! That’s how we do it in 2017! It took less than 20 seconds to invoke our simple invoke-webrequest + convertfrom-json command across five remote hosts. No remote desktop needed….all of it done securely via secure WinRM which I’ve set up my nodes to listen for.
With these results in your console, it’d be trivially easy to dump out each WAN’s public IP information into a CSV, or, even better, create a new Excel spreadsheet using new-comobject and save/send the information from there.
So I was tooling around one day in the lab, reading Ivan Ristic’s book on SSL/TLS, when I came across his advice on securing Windows-based Infrastructures from offering up the use of out-of-date/obsolete or otherwise insecure cipher suites to hosts on the other end of an https connection.
I read Ristic’s chapter a couple of times, reviewed TechNet, and selected a set of cipher suites in Group Policy in the order I wanted them used, based largely on Ristic’s text, but with a few others I knew I’d need after the policy went live. Then I pushed out the new policy, named “Strong Crypto,” to all physical, virtual and laptops in my home lab.
A few gpupdates later, I was pleased to see that nothing was broken. Schannel wasn’t showing any errors, User & Computer accounts were authenticating and getting kerb tickets, and pleasantly, my Outlook fat client didn’t even hiccup; it happily was using TLS 1.2 cipher suites to talk with my Office 365 Exchange instance.
And then, two days later, I noticed it. OneDrive for Business was busted, had gone Pear Shaped, and was now totally t***-up as my English friends would say.
A couple hundred gigabytes of files no longer syncing to my Sharepoint Online site, as evidenced by these Microsoft Icons of Distress:
So, what’d I break?
I’ll get to that in a moment, but first: why would you bother with something as obscure as cipher suites and their order? I mean beyond the fact that toggling the cipher suite sounds cool?
Why Cipher Suites are Important
Cipher suites are a critical part of your AD infrastructure. They’re critical as they represent a sort of baseline set of standards that client & server negotiate over during the complicated and very important tête à tête that is the TLS/SSL handshake between client/server.
You can and should read more about TLS handshakes in this RFC, but the bottom line is this: client & server are supposed to negotiate with each other, find the most secure and common set of cipher package, and use it during the secured session.
If client & server can’t find at least one common cipher suite, you have a busted TLS connection. And that’s no bueno,unless it was your intent.
In Microsoft-land, the default set of cipher suites is pretty good. Who am I kidding, it’s an acronym rich playground of security paradigms, as evidenced by the Group Policy editor:
Don’t be intimidated by all the crypto terms on this screen. What you see is the list of cipher suites -and the order in which they are presented to a host- by default.
The way to read one of these cipher suites is by breaking it down into its constituent parts:
So, the Cipher suite above uses TLS as its protocol (vs SSL), can exchange keys via the Elliptic Curve Diffie Helman ephemeral mechanism, accepts an RSA x.509 certificate, and is willing to encrypt the session via the AES 256 bit block cipher. The last bit, we’ll get to in a moment.
Be cautious when modifying
Since I was doing this in my lab, I had no concern about legacy applications, but in a production environment, you’ll want to tread lightly and deliberately here. Consider:
If your’e in a typical Microsoft IT shop, you probably have a few legacy applications hanging around that may rely on old cipher suites, or vice-versa, the application server can’t use the newer cipher suites that come built into your desktops & laptops.
Take Windows Server 2003, for example. The base OS doesn’t support Elliptic Curve Diffie Helman for Key Exchange, so right off the bat, if you’ve got 2003 Hosts serving up https Sharepoint or Exchange in-house, your clients & servers will never utilize TLS_ECDHE as that suite is not common to both of them. The contrary is also true; your Windows 8.1 laptop isn’t going to support the oldest suites that your 2003 server does; TLS_RSA_WITH_DES_CBC_SHA is never going to be the cipher suite watering hole your clients/servers meet around ((thank Goodness!!)) unless you go out of your way to make it happen.
The lesson here is that old cipher suites never die, dependency on them just fades away as your modernize/replace your legacy in-house applications with modern, streamlined, and properly TLS-secured ones. So be cautious, lest you break a legacy application.
You might be thinking I’m full of great advice, yet I still managed to wreck my OneDrive for Business sync app. And you’d be right!
So what happened?
Essentially, I broke my little OneDrive for Business sync app because I didn’t include SHA1 as possible hash algorithm in any of the cipher suites I selected.
And SHA1 is used by Microsoft IT ((as a side note, it’s really awesome to see Microsoft IT’s PKI, built out as it should be. Here’s a PKI serving not just Microsoft internal employees -all 100k of them- but millions of customers. If Microsoft IT can build a PKI to that scale, surely you and I can build one for the users dependent on us!)) in at least two places: as the Signature Hash algorithm on the root certificate of my Sharepoint site, and as the hashing mechanism for the Thumbprint on *.sharepoint.com certificate.
Had I visited my Sharepoint site in IE, I would likely have seen an error message in my browser; but I use Opera normally, and Opera -like Chrome & Firefox- have cipher suites apart from Windows’ so I never saw an error.
Adding the strongest cipher suite that included SHA1 fixed the error right away. ((Interesting aside: Google, and many security researchers, consider SHA1 to be end-of-life as it is now, or will be very soon now, computationally feasible to crack it, if that’s the right word. Google wants to sunset SHA1 in its browser this year; Ivan Ristic’s site will give https sites that use SHA1 a D- rating by the end of 2015. Microsoft IT, meanwhile, still uses it in production, but plans to deprecate it at the end of 2016. What gives? You could say there’s a pissing match between these leviathans of technology, or that one is trying to screw the other. But in essence, all parties agree SHA1 should fade away, they just differ on how aggressive deprecation efforts should be.))
Like a lot of IT Pros, I’ve been studying up on security topics lately, both as a reaction to the increasing amount of breach news (Who got breached this week, Alex?) and because I felt weak in this area.
So, I went shopping for some books. My goals were simply to get a baseline understanding of crypto systems and best-practice guidance on setting up Microsoft Public Key Infrastructures, which I’ve done in the past but without much confidence in the end result.
This 3.2lb, 800 page book has a 4.9 out of 5 star rating on Amazon, with reviewers calling it the best Microsoft PKI guide out there.
Great! I thought, as I prepared to shell out about $80 and One Click my way to PKI knowledge.
That’s when I noticed that the book is out of print. There are digital versions available from O’Reilly, but it appears most don’t know that.
For the physical book itself, the least expensive used one on Amazon is $749.99. You read that right. $750!
If you want a new copy, there’s one available on Amazon, and it’s $1000.
I immediately jumped over to Camelcamelcamel.com to check the history of this book, thinking there must have been a run on Mr. Komar’s tome as Target, Home Depot, JP Morgan, and Sony Pictures fell.
The price of this book has spiked recently, but Peak PKI was a full three years ago.
I looked up security breaches/events of early 2012. Now correlation != causation, but it’s interesting nonetheless. Hopefully this means there’s a lot of solid Microsoft PKI systems being built out there!
Rather than shell out $750 for the physical book, I decided to get Ivan Ristic’s fantastic Bulletproof SSL/TLS, which I highly recommend. It’s got a chapter on securing Windows infrastructure, but is mostly focused on crypto theory & practical OpenSSL. I’ll buy Komar’s as a digital version next or wait for his forthcoming 2012 R2 revision.
Security has been on my mind lately. I think that in the Spring of 2015, we’re in a new landscape regarding security, one that is much more sinister, serious and threatening than it was in years past. I used to think anonymity was enough, that there was saftey in the herd. But the rules & landscape have changed, and it’s different now than it was just 12 or 24 months ago. So, let’s do an exercise, let’s suppose for the sake of this post that the following are true:
Your credit history and your identity are objects in the marketplace that have value and thus are bought and sold between certain agents freely
These things are also true of your spouse or significant other’s credit history & identity, and even your child’s
Because these things are true, they are also true for malefactors (literally, bad actors) just like any other object that has value and can be traded
There is no legal structure in America aside from power of attorney that allows a single member of a family to protect the identity and credit history of another member of his/her family.
The same market forces that create innovation in enterprise technology are now increasing the potency of weaponized malware systems, that is to say that financial success attracts talent which begets better results which begets more financial success.
The engineers who build malware are probably better than you are at defending against them, and what’s more,they are largely beyond the reach of local, state, or national law enforcement agencies. ((Supposing that your local Sheriff’s Department even has the in-house know-how to handle security breaches, they lack jurisdiction in Ukraine))
The data breaches and mass identity theft of 2014 & 2015 are similar somehwat to a classic market failure, but no cure for this will be forthcoming from Washington, and the trial attorneys & courts who usually play a role in correcting market failures have determined your identity & credit history are worth about $0.14 (($10 million settlement for the 70 million victims of Target breach = $0.14))
Generally speaking most IT departments are bad and suffer from poor leadership, poorly-motivated staff, conflicting directions from the business, an inability to meet the business’ demands, or lack of C-level support. IT is Broken, in other words
All of this means it’s open season on you and your family’s identity & credit history, which we have to assume rest unencrypted on unpatched SQL servers behind an ASA with a list of unmitigated CVEs maintained by some guys in an IT department who hate their job
There it is. That’s the state of personal identity & credit security in 2015 in America, in my view.
And worst of all, it’s not going to get better as every company in America with your data has done the math from the Target settlement and the beancounters have realized one thing: it’s cheaper to settle than to secure your information.
Assume breach at home
If this is truly the state of play -and I think it is- then you as an interested father/mother husband/wife need to take action. I suggest an approach in which you:
Own your Identity online by taking SMTP back: Your SMTP address is to the online world what your birth certificate and/or social security number is to the meatspace world: everything. Your SMTP address is the de facto unique identifier for you online ((By virtue of the fact that these two things are true of SMTP but are not true of rival identity systems, like Facebook or Google profiles: 1) Your SMTP address is required to transact business or utilize services online or is required at some point in a chain of identity systems and 2) SMTP is accepted by all systems and services as prima facie evidence of your identity because of its uniqueness & global acceptance and rival systems are not)) , which begs the question: why are you still using some hippy-dippy free email account you signed up for in college, and why are you letting disinterested third party companies host & mine something for free that is so vital to your identity? Own your identity and your personal security by owning and manipulating SMTP like corporations do: buy a domain, find a hosting service you like, and pay them to host your email. It doesn’t cost much, and besides, you should pay for that which you value. And owning your email has value in abundance: with your own domain, you can make alias SMTP addresses for each of the following things: social media, financial, shopping, food, bills, bulk and direct your accounts to them as appropriate. This works especially well in a family context, where you can point various monthly recurring accounts at a single SMTP address that you can redistribute via other methods and burn/kill as needed. ((Pretty soon, you and your loved ones will get the hang of it, and you and your family will be handing out firstname.lastname@example.org to the grocery store checkout person, email@example.com for receipts, firstname.lastname@example.org for the ‘etailers’ and email@example.com for the two iPhones & three other Apple devices you own.))
Proxy your financial accounts wherever possible: Mask your finances behind a useful proxy, like Paypal, perhaps even Mint. The idea here is to put a buffer between your financial accounts and the services, people, and corporations that want access to them and probably don’t give two shits about protecting your identity or vetting their own IT systems properly. Whenever possible, I buy things online/pay people/services via Paypal or other tools so that use of my real accounts is minimized. Paypal even offers a business credit card backed by the Visa logo, which means you can use it in brick ‘n mortar stores like Target, where the infosec is as fast and loose as the sales and food quality.
Use Burner Numbers: Similar to SMTP, your standard US 10 digit POTS/Mobile phone is a kind of unique identifier to companies, existing somewhere in a unsecured table no doubt. Use burners where you can as your 10 digit mobile is important as a unique identifier and an off-net secondary notification/authentication channel. If Google Voice is to be killed off, as it appears to be, consider Ooma, where for $100/year, you can spawn burner numbers and use them in the same way you use SMTP. Else, use the app on your phone for quick burner numbers.
Consider Power of Attorney or Incorporation: This is admittedly a little crazy, but words can’t describe how furious you’ll be when a family member’s identity has been stolen and some scummy organization that calls itself a bank is calling to verify that you’ve purchased $1000 in Old Navy gift certificates in Texas -something completely out-of-sync with your credit history- but they refuse to stop the theft because it’s happening to your wife, not you, and your wife can’t come to the phone right now. The solution to this problem is beyond me, but probably involves a “You can’t beat ’em, join ’em” approach coupled with an attorney’s threatening letter.
Learn to Love Sandboxing: Microsoft has a free and incredibly powerful tool called Enhanced Mitigation Experience Tool, or EMET, which allows you to select applications and essentially sandbox them so that they can’t pwn your entire operating system. Learn to use and love it. But the idea here goes beyond Win32 to the heart of what we should be doing as IT Pros: standing-up and tearing-down instances of environments, whether those environments are Docker containers, Windows VMs, jails in BSD, or KVM virtual machines. Such techniques are useful beyond devops, they are also useful as operational security techniques at home in my view.
Go with local rather than national financial institutions: Where possible, consider joining a local credit union, where infosec practices might not be state of the art, but your family’s finances have more influence and weight than they do at a Bank of America.
I am not a security expert, but that’s how I see it. If we IT pros are to assume breach at work, as many experts advise us to, we should assume breach at home too, where our identities and those of our loved ones are even more vulnerable and even more valuable.
When in the course of IT events it becomes necessary to inspect alltraffic that hits your user’s PCs, there is but one thing you can do in 2015: get a proxy server or service, deploy a certificate to your trusted root store, and direct all traffic through the proxy.
Why would you do what amounts to a Man in the Middle Attack on your users as a responsible & honest IT Pro? Why Superfish your users? ((
IT Shakespeare put it like this:
To proxy SSL or not to proxy, that is the question
whether ’tis nobler in the mind to suffer
the breaches and theft of outrageous malware
or to take Arms against a sea of digital foes
and by opposing, only mitigate the threat.
To protect via decrypt ; Aye there’s the rub
Thus Conscience does make Cowards of us all
and lose the name of Action))
Numbers are hard to pin down, ((I am not a security expert, and though I checked sources I respect like the Norse IP Viking security blog, Malwarebytes Unpacked blog, SearchSecurity.com etc, I found very few sources that a percentage on how much malware is encrypted and thus difficult to detect. This NSS Labs report from summer 2013 comparing Next Gen Firewall SSL Decryption performance, for instance, says that “the average proportion of SSL traffic within a typical enterprise is 25-35%” and that only ~1% of malware is encrypted. A GWU security researcher named Andre DiMino has a series of good blog posts on the topic, showing what SSL-encrypted malware looks like in WireShark. Team CYMURU’s TotalHash database is probably the most comprehensive open DB of malware samples, but I don’t feel qualified to search it frankly)) but it seems an increasing amount of virulent & potent malware is arriving at your edge encrypted. Because those packets are encrypted, you essentially can’t scan the contents. All you get is source/destination IP address, some other IP header information, and that’s about it.
One option, really your only option at that point, is to crack open those packets and inspect them. Here’s how.
1.You need a proxy server or service that does security inspection.
I’ve seen ZScaler used at several firms. ZScaler dubs itself the premiere cloud-based, SaaS proxy service, and it’s quite a nifty service.
For a fee per user, ZScaler will proxy most if not all of your internet traffic from several datacenters around the globe, sort of like how CloudFlare protects your websites.
The service scans all that http and https traffic, filters out the bad and malicious stuff, blocks access to sites you tell it to, and sends inspected http/https content to your users, wherever they are, on-prem or connected to the unsecured Starbucks access point.
2. You need to bundle those proxy settings up into a .pac file
Getting the service is one thing, you still need to direct your users and computers through it. The easiest way is via Group Policy & what’s called a .pac file.
A .pac file is a settings file generated by ZScaler that contains your preferences, settings, and lists of sites you’d prefer bypass the filter. It looks like this:
function FindProxyForURL(url, host)
var resolved_host_ip = dnsResolve(host);
if (url.substring(0, 4) == "ftp:")
// If the requested website is hosted within the internal network, send direct
if (isPlainHostName(host) ||
isInNet(resolved_host_ip, "22.214.171.124", "255.0.0.0") ||
// If the requested website is SSL and associated with Microsoft O365, send direct
3. Deploy the .pac file via Group Policy to Users
Next, you need to pick your favorite deployment tool to push the .pac file out and set Windows via IE to proxy through ZScaler. We’ll use Group Policy because it’s fast and easy.
Under User Configuration > Policies > Windows Settings > Internet Explorer Maintenance > Connection / Automatic Browser Configuration, select Enable.
Then point the Auto-proxy URL to your Zscaler .pac file URL. It looks like this:
Keep Group Policy open, because we’re not done quite yet.
4. Download the ZScaler Root CA certificates
You’ll find the certs in the administration control screen of ZScaler. There are two:
ZScaler Root Certificate -2048.crt
ZScalerRoot Certificate -2048-SHA256.crt
The two certificates are scoped similarly, the only difference seems to be SHA1 or SHA256 encoding.
Double-click the certificate you prefer to use, and notice that Windows alerts you to the fact that it’s not trusted. Good on ya Microsoft, you’re right.
To validate this setup, you’ll probably want to test before you deploy. So select Install Certificate, select your Computer (not user) and navigate to the Trusted Root CA Store:
Now that you’ve installed the .pac file and the certificate, ensure that IE (and thus Chrome, but not necessarily Firefox) have been set to proxy through Zscaler:
5. SSL Proxy Achievement Unlocked:
Go to Google or any SSL/TLS encrypted site and check the Certificate in your browser.
You should see something like this:
6. You can now deploy that same certificate via Group Policy to your Computers.
It’s trivial at this point to deploy the ZScaler certificates to end-user PCs via Group Policy. You’ll want to use Computer Preferences.
Once deployed, you’ll get comprehensive scanning, blocking and reporting on your users http/https use. You can of course exempt certain sites from being scanned ((Before you do this, make sure you get your Legal department or corporate controller’s sign-off on this. Your company needs to understand exactly what SSL Proxy means, and the Gordian Knot of encryption.
By making all SSL traffic visible to your proxy service, you may gain some ability to prevent potent malware attacks, but at the cost of your user’s privacy. When a user transacts business with their bank, their session will be secured, but only between the ZScaler cloud and the bank’s webserver. The same is true of Facebook or personal email sites.
By doing this, you’re placing an immense amount of trust in the proxy server/service of your choice. You’re trusting that they know what they’re doing with Certificates, that they didn’t use a weak password. You’re trusting that they have their act together, and you’re doing this on behalf of all your users who trust you. This is not to be taken lightly, so run it up the legal/HR flagpole before you do this. ))
Fellow #VFD3 Delegate and Chicago-area vExpert Eric Shanks has recently posted two great pieces on how to setup an Active Directory Certificate Authority in your home lab environment.
Say what? Why would you want the pain of standing up some certificate & security infrastructure in your home lab?
Home Lab SSL Certificates aren’t exactly a high priority for most people, but they are something you might want to play with before you get into a production environment.
Security & Certificate infrastructure are a weak spot in my portfolio so I’ve been practicing/learning in the Daisetta Lab so that I don’t fail at work. Here’s how:
As I was building out my lab, I knew three things: I wanted a routable Fully Qualified Domain Name for my home lab, I was focused on virtualization but should also practice for the cloud and things like ADFS, and I wanted my lab to be as secure as possible (death to port 80 & NTLM!)
With those loose goals in mind, I decided I wanted Daisetta Labs.net to be legit. To have some Certificate Authority bonafides…to get some respect in the strangely federated yet authoritarian world of certificate authorities, browser and OS certificate revocations, and yellow Chrome browser warning screens.
So I purchased a real wildcard SSL certificate from a real Certificate Authority back in March. It cost about $96 for one year, and I don’t regret it at all because I’m using it now to secure all manner of things in Active Directory, and I’ll soon be using it as Daisetta Labs.net on-prem begins interfacing with DaisettaLabs.net in Azure (it already is, via Office 365 DirSync, but I need to get to the next level and the clock is ticking on the cert).
Building on Eric’s excellent posts, I suggest to any Microsoft-focused IT Pros that you consider doing what I did. I know it sucks to shell out money for an SSL certificate, but labwork is hard so that work-work isn’t so hard.
So, go follow Eric’s outline, buy a cert, wildcard or otherwise (got mine at Comodo, there’s also an Israeli CA that gives SSL certs for free, but it’s a drawn-out process) and stand up a subordinate CA (as opposed to a on-prem only Root CA) and get your 443 on!
Man it sucks to get something so fundamentally wrong. Reader Chris pointed out a few inaccuracies and mistakes about my post in the comments below.
At first I was indignant, then thoughtful & reflective, and finally resigned. He’s right. I built an AD Root -not a subortinate as that’s absurd- Certificate Authority in the lab.
Admittedly, I’m not strong in this area. Thanks to Chris for his coaching and I regret if I mislead anyone.