Unhappy anniversary: A year later, Log4Shell vulnerabilities persist

As most people in business — especially the software security business — recall, it wasn’t a very Merry Christmas or Happy New Year at the end of 2021. Not because of a Grinch trying to destroy the holiday spirit in a global Whoville. Because of Log4Shell, a group of vulnerabilities in the Apache Software Foundation’s (ASF) open source logging library Log4j, one of the most popular in the online world, with billions of users.

Anyone who used Log4j, even if it was buried several levels deep in their software supply chain, was vulnerable.

And a year later, while the screaming headlines have turned elsewhere — collapse of cryptocurrency firm FTX, anyone? — and the initial panic has subsided, it’s still an unhappy first anniversary.

According to vulnerability scanning vendor Tenable, as of October, based on data from more than 500 million tests, the large majority (72%) of organizations were still struggling to fully remediate their vulnerability exposure to Log4Shell.

That’s no surprise to Tim Mackey, head of software supply chain risk at Synopsys. He said Log4Shell is likely to have “as long a tail as we saw with Heartbleed,” the catastrophic bug in the OpenSSL cryptographic library that was introduced into the software in 2012 but didn’t become public until 2014.

Mackey said it took four more years, until 2018, before the annual Synopsys report “Open Source Security and Risk Analysis” didn’t find any codebases that used a vulnerable version of OpenSSL.

The Log4Shell defects, called “remote code execution” vulnerabilities, can allow cyberattackers to gain remote control over computers running applications in Java — one of the world’s most popular programming languages.

Worse, exploiting Log4Shell is easy. Michael White, technical director and principal architect with the Synopsys Software Integrity Group, wrote in a blog post at the time that it’s “trivial to execute.”

“Log4j is very broadly used in a variety of consumer and enterprise services, websites, and applications — as well as in operational technology products — to log security and performance information,” White wrote, noting that an estimated 3 billion devices run Java. “An unauthenticated remote actor could exploit this vulnerability to take control of an affected system,” he added.

Pre-holiday panic

The news set off a panicked scramble, for good reason. The ASF rolled out multiple updates for Log4Shell and other vulnerabilities that were applied by millions of organizations trying desperately to protect their assets against cybercriminals who, by the week before Christmas, were launching attacks seeking to exploit it at the rate of more than one every second. For those without calculators handy, that’s in the 100,000-per-day range.

U.S. Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly famously called Log4Shell “one of the most serious [vulnerabilities] I’ve seen in my entire career, if not the most serious.”

The National Vulnerability Database gave Log4Shell a severity rating of 10.0 — out of 10.0.

So, a year later, an online world where almost everybody, or even a large majority, had updated their vulnerable versions of Log4j to make them immune to Log4Shell would qualify as a happy anniversary.

Not so much. Instead, as documented by Tenable, the large majority are still struggling with updating.

It’s complicated

Why? One major reason is that updating a software component like Log4j is much more complicated than simply tapping the “update” icon for an app on your smartphone and waiting a few seconds for it to install.

Kurt Seifried, chief blockchain officer and director of special projects at the Cloud Security Alliance said this past July that “a lot of the people not patching are running some old app using Log4j that can’t be updated.”

Or as Mackey put it, telling users they should just update Log4j “is the type of recommendation made by someone who hasn’t had to manage software in production. Every piece of software has unique functionality, and every update changes that functionality in some important way,” he said.

“Simply replacing one version of a component with another in an application that was never tested with the changed version creates a scenario where the application could misbehave simply because the assumptions the application developers made aren’t true any longer.” The result, he said, could be “garbage data being written to logs,” which could make things worse instead of better.

“If a malicious group knew that an update could result in corrupted log files, they might be able to craft an attack where the fingerprints of their attack are masked by the corrupted log data,” he said.

That means the owner of an application with an older version of Log4j needs to test it with the updated version to see if the application still works correctly with it. “If it doesn’t, they might need to change some of their code to reflect how the update works. This process creates a ‘supported configuration,’” he said, which could lead to yet another complication.

“If a software operator were to perform the Log4j update, then the application owner could claim that their application wasn’t deployed in a supported configuration and deny the software operator support for any problems caused by the change in Log4j. This is one reason why all software operators should have an active relationship with their software providers,” he said.

Another difficulty is, as Tenable Chief security Officer Bob Huber put it in a blog post, “vulnerability remediation is not a ‘one and done’ process. While an organization may have been fully remediated at some point, as they’ve added new assets to their environments, they are likely to encounter Log4Shell again and again.”

Huber did note that, based on the company’s data, most organizations had reduced their exposure, from about 10% of assets vulnerable in December 2021 to about 2.5% two months ago.

Still, it raises an obvious question: Why allow any assets to be added to an organization’s environment without testing them first for vulnerable versions of Log4j?

Comprehensive assurance

Because again, it’s not that simple. Screening new assets — desktop computers, laptops, servers, storage devices, network devices, phones, tablets, virtual machines, cloud instances, and containers — for known high-severity vulnerabilities “isn’t always practical,” Mackey said.

But, he added, that doesn’t mean companies can’t do anything to mitigate the risk. “It should be part of a more comprehensive software assurance process,” he said. “Procurement teams should implement ‘trust-but-verify’ paradigms backed up by contract language. For example, a contract might contain a clause stating that procured products shall be free from any CVSS [Common Vulnerability Scoring System] 7.0 or higher vulnerabilities on the date of shipment, and that all products containing software must include a software Bill of Materials (SBOM) for the software present in the device at its point of manufacture,” he said.

“Since an SBOM is machine-readable, querying for a vulnerable version of Log4j is easily automated. And if a vulnerable version is listed in the SBOM then the shipment could be refused based on that contract clause.”

How are organizations supposed to create an SBOM when a single application can have hundreds of so-called “dependencies” — components that other components rely on to function — that go multiple levels deep?

That’s where there is good news — an automated testing tool called software composition analysis (SCA) can help create SBOMs, which would be next to impossible to do manually. “SCA figures out which open source components were used in assembling an application,” said Jonathan Knudsen, head of global research within the Synopsys Cybersecurity Research Center. “If the application contains any component with a dangerous vulnerability such as Log4Shell, the organization can delay deploying the application until the vendor addresses the problem.”

In other words, the key to mitigating the threat from Log4Shell is to know where it could be. One of the long-time mantras at security conferences is that it’s impossible to protect, or fix, what you don’t know you have.

“It comes down to establishing visibility of the components in your software applications,” said Mike McGuire, security solutions manager with the Synopsys Software Integrity Group. “For software builders, this means working on open source software detection and continuous monitoring for vulnerabilities in the development pipeline. For software users, it means obtaining SBOMs and sometimes even a means to analyze the binaries provided to them by their vendors.”

It’s worth it

The time, effort, and money to do all that are worth it, on a number of levels. According to the Federal Trade Commission (FTC), failure to identify and update Log4j may violate the FTC Act. If you’re not sure which version to use, you can find the most current version here.

There is also CISA guidance on how to mitigate the vulnerability.

It’s worth all that to protect your organization. Log4Shell remains very dangerous.

Mackey called it “one of the most serious, and similarly, most complex issues I’ve seen — easily one of my top 10. Mostly, this is because very few development teams are actively choosing to use a specific logging mechanism and are simply using something that’s expedient. After all, people pay money for features and don’t pay money for code that writes log data.”