Log4Shell: Eight months out, patches for major software vulnerability available but not universally applied

“Predictions are hard, especially about the future,” the late, great Yogi Berra is said to have said.

But apparently not all the time. Predictions about the patching of software ultra-vulnerability Log4Shell were easy. Which is not a good thing.

It was December 2021 — the height of prediction season — when the catastrophic Log4Shell group of vulnerabilities in the Apache Software Foundation (ASF) open source logging library Log4j became public.

Log4Shell, which could allow hackers to achieve “remote code execution” — the remote takeover of a computer — was called “one of the most serious (vulnerabilities) I’ve seen in my entire career, if not the most serious,” by U.S. Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly.

The National Vulnerability Database gave Log4Shell a severity rating of 10.0 — out of 10.0.

Its disclosure on Dec. 8 set off a mad scramble just before Christmas by the unpaid volunteers of the nonprofit ASF, who set about trying to do the right thing — create a patch to fix the vulnerability — fast. And they did, within 24 hours.

So you’d think, given the potential damage from malicious hackers who were salivating at the prospect of exploiting Log4Shell, that every last one of the hundreds of millions of users of Log4j (written in Java, one of the most popular programming languages) would have scrambled just as fast to apply the patch.

Selective scramble

But you’d be wrong. Steve Povolny, head of advanced threat research for McAfee Enterprise and FireEye, told ZDNet in mid December, just a week after the disclosure, that while many users would apply the patches being rolled out by the ASF, “We believe Log4Shell exploits will persist for months, if not years to come.”

Check. Yogi isn’t always right. This prediction was easy. Almost eight months later, during the last week of June, CISA and the U.S. Coast Guard Cyber Command released a joint cybersecurity advisory to warn network defenders that “cyber threat actors, including state-sponsored advanced persistent threat actors, have continued to exploit CVE-2021–44228 (Log4Shell) in VMware Horizon and Unified Access Gateway servers to obtain initial access to organizations that did not apply available patches.”

A lot of them didn’t. And still haven’t. CISA doesn’t send out advisories for trivial risks.

Why does this keep happening? It’s been more than five years — practically a generation in information technology — since credit bureau giant Equifax allowed the personal and financial information of 145.5 million people to be compromised because it failed to install a patch to fix a vulnerability in the dispute portal of the Apache Struts platform — a patch that had been available for months.

If that wasn’t a wake-up call, what will it take?

Apparently longer than five years. And that’s in part because Log4j is so common and has been around for so long that “a lot of the people not patching are running some old app using Log4j that can’t be updated,” said Kurt Seifried, chief blockchain officer and director of special projects at the Cloud Security Alliance.

It also means that many likely don’t even know they’re using Log4j — they are among millions of users of open source software who aren’t keeping track of all the components in their codebases.

The Microsoft Threat Intelligence Center, in a Jan. 3 update to an advisory from Dec. 11, stated, in what was likely a massive understatement, “customers may not readily know how widespread the issue is in their environment.”

Or as Randori, an IBM-owned attack surface management platform, put it in April, “Log4j is here to stay; we will see attackers leveraging it again and again. Log4j [is] buried deep into layers and layers of shared third-party code, leading us to the conclusion that we’ll see instances of the Log4j vulnerability being exploited in services used by organizations that use a lot of open source.”

Predictions confirmed

And all those predictions were confirmed last week by the new Cyber Safety Review Board’s first report, which is a review of the Log4j event. The report, from the public-private initiative launched this past February in response to President Biden’s May 2021 Executive Order on Improving the Nation’s Cybersecurity, did contain a measure of good news — the board had not seen any evidence of Log4Shell exploits used to attack critical infrastructure — but the bottom line remains ominous. “The board assesses that Log4j is an ‘endemic vulnerability’ and that vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer.”

Which also makes the CISA warning an illustration of the reality that the problem with open source use is not the software itself — it is no more or less secure than proprietary or commercial software — but the failure to manage it.

Indeed, open source is overwhelmingly popular for very good reasons. It’s almost always free and can be modified to do what a user wants, as long as that user complies with any licensing requirements. Open source software components are the building blocks that power innovation because most software products are assembled from existing components, not written from scratch. It’s what has enabled tech innovation to move so fast.

The problem is a failure to track and maintain the software supply chain — a bit like a vehicle manufacturer not knowing what parts are being used in the airbags or where they came from.

Tracking and managing open source is difficult — it’s essentially impossible to do manually. Most organizations are using tens of thousands to millions of open source components — the latest annual “Open Source Software and Risk Analysis” report by the Synopsys Cybersecurity Research Center (CyRC) found that virtually every codebase in use today uses open source, and that it makes up an average of nearly 80% of those codebases.

And most of those components are, as Randori noted, buried in a web of so-called “dependencies” that will remain hidden unless you look for them.

Tim Mackey, principal security strategist within the Synopsys CyRC, has used a simple Slack application with an Instagram interface as an example.

In a recent webinar, he noted that the app included eight declared dependencies, one of which originated from Slack itself, called Bolt.

But Bolt has 15 dependencies of its own. And one of those, called Express, has another 30. “When you peel back the onion on this, Slack/Bolt actually has 133 separate components in it,” Mackey said.

That’s 16 times the eight declared or “visible” dependencies of just one component of one app. And given that most organizations have hundreds to thousands of apps, devices, systems, or networks powered by software, it’s obvious that tracking and maintaining a software inventory, commonly called a software Bill of Materials (SBOM), is complex and unwieldy.

Parade of horribles

But the risks of failing to do so — the most obvious of which is getting breached — can lead to what is now a familiar parade of horribles: hijacking of systems, data leaks (and therefore theft of sensitive data), denial-of-service attacks, system crashes, identity theft, brand damage, loss of customers, compliance fines or other sanctions, legal liability, recovery costs and more.

Those risks can extend beyond internal as well. Phil Odence, general manager of the Black Duck® Audit business within the Synopsys Software Integrity Group, said that “almost every concern organizations might have about their own code is one they should have about code they’re acquiring through M&A [merger and acquisition].”

“You could say it’s even more of a concern, since bigger companies tend to acquire smaller companies, and smaller companies tend to be less focused on code hygiene — a generalization but on average true,” he said.

It’s also another explanation for CISA’s advisory that there is still a lot of Log4Shell patching to do. “It’s rare for a company being acquired to even know most of what components are in their code,” Odence said. “If a company doesn’t know they’re using Log4j or, more likely, doesn’t know everywhere they are using it, that’s the simplest explanation of why they’re not patching.”

Fortunately, there are numerous resources available to help with the impossible manual task, including software composition analysis, an automated tool that helps find open source components and their dependencies, and that can help with the creation and maintenance of an SBOM.

With an SBOM, an organization simply needs to conduct a database search to know if it’s using a library like Log4j. So SBOMs ought to be standard. As is preached at security conferences all the time, if you don’t know what you have, you can’t protect it.

There is also plenty of guidance available from federal agencies as well, including the Federal Trade Commission, such as

• Update your Log4j software package to the most current version found here: https://logging.apache.org/log4j/2.x/security.html
• Consult CISA guidance to mitigate this vulnerability.
• Make sure your company’s practices do not violate the law. Failure to identify and patch instances of this software may violate the FTC Act.
• Distribute this information to any relevant third-party subsidiaries that sell products or services to consumers that may be vulnerable.

Still, here we are, eight months out from the revelation of a threat that, on the digital level, is as serious as Covid-19, with a “vaccine” available that’s not being universally being used.

Even though, unlike a biological virus, you don’t have to make an appointment, drive anywhere, or wait for hours for a vaccine. Patches and updates, once available, are immediately available to everybody.

Seifried is somewhat pessimistic that there will be universal immunity to Log4Shell anytime soon, even given potential fines or other sanctions for weak security.

“Most of the fines are far less than the cost of doing security, and most of the real cost is externalized,” he said, noting the recent news about a weakness in the remote keyless entry system of all Honda vehicles made since 2012. GitHub reported that the weakness could allow an attacker “to permanently open the car door or even start the car engine from a long distance.”

“Worst case for Honda is that they’ll have to provide free software upgrades/servicing at dealerships and a nice card saying, ‘We’re really sorry,’” he said.