So there’s a new virus in town and it’s a bad one: The WannaCry ransomware tool infects a whole bunch of Windows operating systems ranging from Windows XP to somewhat current Windows versions such as Windows Server 2012. It mostly does the same thing all recent ransomware tools do: Encrypts all files it can get its hands on and demand a payment in bitcoin to decrypt the kidnapped files. So far so standard.
A few things make this case stand out. First, the exploit’s origin which seems to lead back to the NSA. And second the virus’ success: Screenshots of for example British NHS Services being down spread through social media like wildfire. Articles and opinion pieces wonder how it can be that companies or government entities still run unpatched systems or systems without any vendor support (like Windows XP).
And the solutions are so easy, right? Why don’t companies just patch their servers and keep them updated? It isn’t hard, people do that with their laptops all the time. Oh and of course there’s going to be opportunists trying to tie their pet project to this case (whether it’s connected or not): Politicians asking how European countries can accept a US monopoly (which is somewhat true with regards to the monopoly angle but in this regard mostly not the issue) or why people don’t run their servers on Linux/*BSD/Solaris/NonexistingEUOperatingSystem.
As someone doing project management and solution development/consulting for a company building software systems for industry customers (manufacturing execution systems, maintenance management, SCADA, internal logistics) I thought I’d explain a little bit about how these things happen and what would actually change things.
But let’s start by looking at how things are. Software systems in the real world, especially when it comes to software systems in large organizations are extremely complex and very rarely created synchronously or by one entity. What this means is that a new storage management system will have to interact with the weird, 10-year-old custom mini-ERP system that the organization uses, it’s somewhat strange Active Directory user management, it’s proprietary and 20-year-old PLC systems and most probably Excel. Lots of Excel. Excel is god.
All these parts have a history, have their own requirements, their own vendors, and potentially certifications. So your custom SCADA solution is only supported on Windows XP and Windows 7, your manufacturing execution solution only supports operating systems supported by Microsoft and now your new automation software driver requires at least Windows 8 but definitely will not run on Windows 7. It’s a huge mess to sort these things out, and not just because vendors die or go away (even though that does happen).
It’s almost always a question of cost.
And that’s what most comments ignore. You can write pages upon pages of exploit porn or claims that Microsoft needs to support XP even longer. But in the end, it wouldn’t have mattered.Because this whole mess is not at all a technical problem. Patches for supported Windows systems existed but weren’t applied. Why? Because updates cost a shitton of money.
So you have a factory running on a whole zoo of different software packages. But it runs. Raw material goes in, processes run reliably and products come out. That’s what you want your factory to do, that’s how the money comes in. Profit is what is left after you subtract all expenses (wages, maintenance, new investments) from your income. And – given you live in a capitalist system – profit is supposed to be maximized so every expense has to have a good reason.
Let’s look at a concrete example. The company I work for (just as many other companies in that domain) makes its money by selling services. You call me about something you need. I tell you how many days it will take and our daily rate, we sign a contract and after a while you get software. How would this play out in this case?
WannaCry comes out, the Company gets scared and wants to update. Hmm. Ok, their old Windows XP software to run their expensive production line isn’t certified for this new patch level. So they call the vendor. The vendor now has to estimate how much testing the custom solution they build for that customer 15 years ago to reliably run on this new patch level will take (and cost). So let’s just say that this process will take 3 days. To make calculation easy let’s assume a daily rate of 1000 € (which is on the lower end of things). So our company who wants to update would have to pay 3000€ just to basically have the same system they had before. How do you justify paying this money especially if you are not in IT and estimating potential cost of IT risks isn’t your expertise or even business? You don’t. Now just think how the costs explode when you have to ask 3 different companies to test their software (and all its dependencies) against this new patch. Let’s come up with some numbers. 3 partners and all need about 3 days for tests, fixes, etc. That’s 9000€. You probably need them all to come in to test their fixes as a whole on the actual hardware (which is, in this case, a 15-year-old factory with a bunch of machines). So that’s 3 extra days for them to send a person over. Plus travel, expenses etc. Let’s say the field test costs you another 6000€. And the first date where you can get all the experts into your plant is in 3 months. Oh and for the tests you will have to pause production to put the test environment up which adds another few thousand € in missed production. Let’s just say 5000€. To update our systems to this little patch we have to pay 20000€. And for what? To have the same system that we had before.
Users don’t care about software. And they shouldn’t. Software should solve problems and get out of the way. So why throw thousands of Euros at a software if afterwards you can’t even see it being better, feel it being better? Who would pay for that?
“But support contracts!” I hear people say, “they do exactly pay for these kinds of work!”. Sure, those are nice. We always offer them for anything we deploy. Pay a few thousand bucks a year and we’ll take care of these things. But again: A few thousand bucks each year on the budget. Can you justify that to your CEO if you are not in IT? How high is the actual risk? What can happen? How much will it cost the company? If patching costs you 20000€ and restoring a backup costs you 10000€ that would mean that not patching is cheaper unless you get hit more than twice. That might be a risk worth taking?
Businesses (and increasingly public organizations as well) make decisions based on cost. And properly estimating the costs of an unpatched Windows 7 server is extremely hard if your job is baking donuts or producing plastic cat toys. But you do know that you haven’t patched your servers for 5 years and it never hurt you. And that’s the world we live in.
As someone working in a software company, I cannot just give that kind of service away. Because I have to spend a few days on this. But nobody wants to pay for these days because the risk is abstract and unquantified. That goes double or triple in industry segments with very small profit margins.
What can be done?
Software is everywhere. It runs large parts of our economy and structures. In theory, we want it all to be up to date and perfectly supported. In theory, I also would like to have a pet spidermonkey following me around.
There are two things we can do.
First: We give entities better tools to estimate the costs of not updating to nudge them in the right direction. The problem with this is that many still will take the cheap road and just hope nothing breaks. Because it’s just software, right? If you don’t touch it, it will run forever!(which is the argument used by many software companies and automation experts to sell that whole shebang).
So we have to go for second: We have to force entities running critical software systems to have support contracts for all relevant software systems. Support contracts that guarantee a certain timeframe in which a patch can be deployed and tested. Support contracts also need to include strategies and processes for major operating system updates. Without that support contract you cannot get insurance for your plant and government entities will shut your plant down. We already force companies to show that they properly handle and maintain other critical parts of their infrastructure (for reasons of public or worker safety or the environment for example), software needs to be included in this.
DISCLAIMER: This legislation would benefit the company I work for because all our customers would be forced to buy a support contract. So take my words with a grain of salt.
Supporting and maintaining software systems needs to be a requirement of being able to run your plant. Because if it’s not legally enforced it will not happen. Believe me, I’ve seen how even big companies maintain their critical assets (after all I work in maintenance) and in more cases than I’m comfortable with the answer is: Not at all.
Quick Question Extravaganza
Would running Linux have helped?
Yes and no. Every Software has bugs and you can write ransomware for any OS (some make it harder than others of course). Right now Linux is “safe” because of its installed base: Very few people who open random Email attachments run it so if you want to develop ransomware you probably target Windows (just to have a higher number of targets). But with a growing installed base that attacks would rise as well. You operating system will have bugs and it will be targeted. And updating Linux is just as costly and hard as updating Windows is because you have to go through the same processes. Yes, a proprietary driver can easily bitrot and no longer support current OSes but the same can happen with open source drivers that the original author lost interest in. And do you really want to patch that driver you just kind of understand 10 years after it was written? Is that the risk you want to take?
Europe needs its own Operating System to get out of the dependency on Windows!
A) That is not a question. B) That wouldn’t really solve anything in this case. Yes, the source of the exploit might have been the NSA but they’d exploit European OSes as well. I’m all for breaking up monopolies but that is a whole different ballgame.
Is it Microsoft's fault?
Not really. They offered patches for all their supported OSes months ago. They even added an XP patch in spite of it being out of support. What else could they do? Windows XP was released in 2001 and support ended in 2014. That’s 13 years of support. While it’s easy to say that if Microsoft builds a OS they have to support it for as long as people need it that doesn’t work financially. Does Microsoft have to provide support for MS DOS 3.3 because I have an old machine that still runs on it? If 13 years of supporting one version of their operating system isn’t enough, how long do they need to do it? Let’s say we manage to enforce 25 years of free updates. That cost would be added to the license. We’d force people who do actually update their system much more frequently (end users) to pay a big premium for companies to get free updates. That sounds … wrong.
Not keeping software updated is not really a technical issue, it’s economic. Updates cost a lot of money to develop, deploy and re-certify, costs that do not in many organizations’ perception add anything to justify the expense. The cost of not updating is invisible (until all goes wrong). The only effective way to get companies to do the updates is to force them to have support contracts for every piece of critical software they run.
Photo by zolierdos