Keeping software up to date is crucial but complicated
“Keep your stuff up to date” is one of the closest things to a religious mantra in cybersecurity. And that means everything — your networks, your systems, your applications.
For good reason. If you don’t apply patches when they become available, you’re asking for trouble because malicious hackers know, just like everybody else, when vulnerabilities become public and immediately start looking for organizations that are too distracted or clueless to apply those patches.
Fail to update and you’re in effect asking to be the next Equifax, the credit bureau giant breached in 2017 after it failed to install a patch for a vulnerability in the Apache Struts open source web application framework — a patch that had been available for two months before the breach.
So you’d think that message would be getting through. And it looks like it is — sort of. Positive Technologies, a Moscow-based security firm that conducted an “automated security assessment of the network perimeters of selected corporate information systems,” reported in October 2020 that 74% of the firms it tested had patched their systems against the notorious WannaCry ransomware.
That’s a healthy majority. But it also means there are still a lot of organizations — one in four — that need to get update religion.
Self-described public interest technologist Bruce Schneier, chief of security architecture at Inrupt, linked to the reporton his blog earlier this month and called the statistic a “disaster,” even if one assumes that the report is “self-serving to the company that wrote it, and that the statistic is not generally representative.”
“The number should be 0%,” he wrote, noting that WannaCry is four years old and primarily affects older products like the Windows 7 operating system, which Microsoft no longer supports. “If we can’t keep our systems secure from these vulnerabilities, how are we ever going to secure them from new threats?” he wrote.
But that got a fair amount of blowback in Schneier’s own comment section from multiple respondents who said updating and patching are often neither simple nor cheap. Many times, they said, it requires upgrading other parts of a system, at great cost.
As one unidentified commenter put it, “long ago they bought very expensive applications which won’t run on later versions, whose manufacturers can’t be bothered to update them, or are demanding prohibitive fees for doing so, or even have gone out of business.”
“So those companies are faced with a choice: either completely replace some wildly-expensive equipment which still works perfectly well, simply because there’s no driver for it, or stay with Windows 7 or XP. For top management that choice is a no-brainer, and the CIO has no chance of persuading them otherwise.”
Neal Krawetz, founder of Hacker Factor Solutions, had a similar view. “With Windows, there is usually no simple upgrade path,” he wrote. “Copy off all of your files, write down all of your apps, and then reinstall. Hopefully you can put it all back. (You should allocate days for this effort. That’s days of downtime and resources tied up in the upgrading.)”
“With Mac, OS upgrades usually last 1–4 years, then you need new hardware, even though the current hardware runs fine on the current OS. It’s the same problem with Android smartphones, but with much more expensive equipment,” he wrote, adding, “You want 0%? That’s never going to happen.”
And then there was Mike Anthis who, while not disagreeing with Schneier’s disaster declaration, observed that “the whole world is bad at preventative maintenance.”
Jonathan Knudsen, senior security strategist with the Synopsys Software Integrity Group, agrees. “It’s easy for security people to say ‘keep your stuff up to date,’ but it’s much harder for the people who are actually responsible for keeping the lights on,” he said.
“For people in the real world, patching production systems safely is complicated. You never know if a patch will break something in your environment. So you create a staging (test) environment that’s pretty close to production, and you first deploy patches into the staging environment. Then you do some testing, and if everything checks out, you cross your fingers and push patches into production.”
Ian Ashworth, senior security consultant with the Synopsys Software Integrity Group, said this is a situation where size matters. While large corporations “should have a firm grip on any cyber threat,” it’s much more difficult for smaller organizations to keep up with the complications of patching because “technology is perhaps only an enabler for them to run their business more efficiently or to remain competitive.”
For organizations that don’t have the budget to overhaul their system just to install an update, “almost being forced to make changes merely to permit their business to continue is often seen as an unwanted and unnecessary distraction and expense,” Ashworth said.
Another problem is that people running small organizations usually don’t have the time or staff to keep up with every notification, or even headlines, about cyberthreats and could be unaware of the latest information.
Ashworth said that while just about everybody heard about WannaCry when it appeared in 2017, some may have also heard about the publication of a “kill switch” that stopped WannaCry’s spread and assumed “that obviated their need to worry about it anymore. But there are apparently thousands of variations of WannaCry so the risk is still very real,” he said.
Schneier, via email, had no response to his blog commenters but agreed that “it’s a hard problem.”
What are the options
So what are the 26% supposed to do?
Amit Sharma, security engineer with the Synopsys Software Integrity Group, said one tactic is to balance the risk of patching with the risk of not patching. “The patch management process is not trivial — it takes effort and resources,” he said, noting that it could require testing a patch before applying it to see if it affects the functioning of the system.
So that needs to be balanced with what the patch will protect. “Does the infected system have crucial data that is required to be protected or can we live with the infection?” he said.
Of course, knowing what is high-risk and what isn’t requires organizations to keep track of their assets. “Do you have a Bill of Materials of the software you are making or shipping? If not, that is the first step towards success,” Sharma said.
A possible workaround to doing risky, costly updates, Knudsen said, is to keep older, vulnerable versions of software in an isolated network segment with a restrictive firewall. “When you have something risky, put it in a box, and maybe put that box in another box,” he said.
But that isn’t simple either. Knudsen acknowledged he doesn’t have a good answer for the small business.
Does that mean the small players are essentially left with the false hope of “security by obscurity,” meaning keeping your fingers crossed that the bad guys won’t be interested in somebody that small?
To some extent that’s the reality, although most experts agree that the industry overall needs to do better to “make the secure way the easy way” — another mantra at security conferences. The modern automobile doesn’t require the average driver to be a mechanic to have a relatively safe vehicle. Safety systems like seatbelts, airbags, antilock brakes, lane assist, and side-view and backup cameras are all built in.
Build it better
That ought to be the case in the digital world as well, where networks, systems and applications should have security built in. But to do that requires the use of multiple automated tools available to test software as it is being built and also to build in the capability to update software for the expected life of the product it serves.
And at present, that’s not being done — not even close. Clive Robinson, security consultant with Croma Security Solutions Group, in a comment on Schneier’s blog, noted that things like industrial systems, utility meters, railway engines and telephone systems have an average lifespan of decades while the useful life of smart devices, operating systems and applications are in the months to a few years range.
“Remembering that the first lot are dependent on the second lot you can see there is an approximate ‘months for years’ issue,” he wrote. “Applications and OSs last about 1/12th of the time that physical machinery and systems do.”
So, especially for small organizations but even for the larger ones, it’s going to take some systemic change to make the secure way the easy way.
Ashworth suggested that “small, purposeful business grants or loans at very attractive rates could be really helpful in addressing the risks much sooner.”
Or as Knudsen put it, “In a world where everybody built software with a secure development life cycle, everything would be quite a bit safer and require much less-frequent patching.”
Still, “patching will always be a fact of life,” he said. And even automatic updates can create complications. “Some software updates itself, or can be configured to do so,” Knudsen said. “This is a good solution for most consumers and small businesses, who will never really understand cybersecurity.”
“But it still leaves the core problem: if I operate a pizza shop and my cash register updates itself automatically and no longer works for whatever reason, I’m hosed.”
Some updates are clunky, sometimes a critical application won’t work with an updated system, or “sometimes you have to sign back in again, or it doesn’t work the way it used to,” he said.
So the only real answer is “Build software better,” Knudsen said.