Nerd For Tech
Published in

Nerd For Tech

Get your vaccine for the Log4Shell software pandemic

here’s a major, worldwide vaccination push going on that has nothing to do with COVID. Although you could call Log4Shell, a group of vulnerabilities in the open source Apache logging library Log4j, the digital equivalent of the Omicron COVID variant.

Because it’s a pandemic. Except it didn’t have to spread. At the time it became public in early December, it was already present in hundreds of millions to billions of systems, services, websites and devices. If you used Log4j anywhere in your organization, you were vulnerable.

Hence the frantic effort to find it and patch or mitigate it — indeed, some referred to it as getting a vaccine. With good reason. The Log4Shell defects, called “remote code execution” vulnerabilities, can allow cyberattackers to make your organization very sick by gaining remote control over computers running applications in Java — one of the world’s most popular programming languages.

Worse, exploiting Log4Shell is easy. Michael White, technical director and principal architect with the Synopsys Software Integrity Group, wrote in a recent blog post that it’s “trivial to execute.”

“Log4j is very broadly used in a variety of consumer and enterprise services, websites, and applications — as well as in operational technology products — to log security and performance information,” White wrote. “An unauthenticated remote actor could exploit this vulnerability to take control of an affected system.”

The damage from attacks on Log4j so far range from crypto mining to, as the Federal Trade Commission (FTC) put it in a warning last week, “a loss or breach of personal information, financial loss, and other irreversible harms.”

U.S. Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly called Log4Shell “one of the most serious I’ve seen in my entire career, if not the most serious.”

White views it as “a classic iceberg. There are 3 billion devices that run Java and Log4j is a very common logging library,” he said.

Indeed, the National Vulnerability Database gave the Log4Shell vulnerability a severity rating of 10.0. That’s 10 out of 10 — the worst possible. So, combine very severe with very easy to execute, and that equals very, very bad. If you can’t trust your software, which likely runs just about every element of your business, you’re in trouble. Software risks are business risks.

Digital vaccinations

In response, Apache has rolled out multiple updates — digital vaccinations, if you will — for Log4Shell and other vulnerabilities, and there have likely been millions — perhaps dozens to even hundreds of millions — of those patches applied by organizations trying desperately to protect their assets against cybercriminals who, by the week before Christmas, were launching attacks seeking to exploit it at the rate of more than one every second. For those without calculators handy, that’s in the 100,000 per day range.

Government is taking it seriously as well, with a vaccination mandate. CISA issued Emergency Directive 22–02 on Dec. 17, requiring all civilian federal agencies (national security systems, the Department of Defense, and intelligence agencies are exempt) to find all Log4j components in their systems and patch, mitigate or remove them by Dec. 23.

The FTC was equally stern, warning that it “intends to use its full legal authority to pursue companies that fail to take reasonable steps to protect consumer data from exposure as a result of Log4j, or similar known vulnerabilities in the future.”

At one level, this sounds like it shouldn’t be too hard. The upside of a digital pandemic is that you don’t have to make an appointment, drive anywhere, or wait for hours to get vaccinated or buy a test kit. Patches and updates, once available, are immediately available to everybody. Sort of on the level of, “Your door is unlocked. Lock your door!” and everything will be OK.

But it’s not that simple. Numerous experts predict that the Log4j nightmare could go on for years, since the library is so deeply buried in many organizations’ software supply chains.

That’s the downside of digital infections. Too many organizations don’t know they’re vulnerable until they start experiencing symptoms — a cyberattack. While an organization may not knowingly be using Log4j, it’s likely that one or more of the components of its software are relying on it.

The Microsoft Threat Intelligence Center, in a Jan. 3 update to an advisory from Dec. 11, said, in what is likely a massive understatement, “customers may not readily know how widespread the issue is in their environment.”

Because of that, and “due to the many software and services that are impacted and given the pace of updates, this is expected to have a long tail for remediation, requiring ongoing, sustainable vigilance,” according to the advisory.

So, a month since it started, how is the vaccination campaign going?

It’s impossible to know in the private sector. In the public sector, at least at the federal level, it’s going pretty well according to CISA, although a statement from the agency was long on generalities and short on specifics. It said federal agencies covered by the directive had either remediated or mitigated “the majority of affected applications identified that support ‘solution stacks’ that accept data input from the internet.”

A majority, of course, could be anywhere from 50% to 100%.

The statement also noted that CISA had received status reports from all “large” agencies. But it didn’t define large nor how many or what percent of the total that meant. It did say those large agencies had “addressed the risk from thousands of internet-connected assets,” and that “CISA continues to work with each agency to drive further progress toward remediating all assets at risk.”

White said the fact that, in both the public and private sectors, “nobody knows what is affected or what percent success rate we have at remediation highlights a whole problem in itself. Ideally, I should be able to say, ‘Sure, we have 500 of those systems, 30% are granted an exemption, 20% need a code change, 50% are already fixed’, etc.”

Time for an SBOM

Whatever the status, the irony is that much of the present panic could have been avoided. There is a way to prevent the time-consuming nightmare of finding and fixing the vulnerabilities in Log4j or any other software component with a known vulnerability. It’s been around for years. It’s called a software Bill of Materials (SBOM) — a rigorous inventory of every component in an organization’s software supply chain.

With an SBOM, an organization simply needs to conduct a database search to know if it’s using a library like Log4j. That sort of thing ought to be standard. As is preached at security conferences all the time, if you don’t know what you have, you can’t protect it.

But creating and maintaining an SBOM requires the right skills and tools. Without them, the task can be daunting to impossible. Just about every modern software project is largely assembled, not written, from components that already exist — either commercial or open source. Each of those components could depend on dozens of other components or libraries to function. A single application could easily have hundreds of such “dependencies” going multiple levels deep.

The good news is that there are automated tools available to help with the task, including software composition analysis, which helps find open source components and their dependencies. It’s just that an emergency digital pandemic is not the best time to start building one.

As Joe Jarzombek, director for government and critical infrastructure programs at Synopsys, put it, “Many federal CISOs [chief information security officers] might be fretting, ‘If I only had SBOMs for my network-connected assets, then I’d know where we might be affected by Log4j.’”

Jarzombek added that suppliers might know about vulnerabilities like Log4j when a software component is created but, “many assert that certain known vulnerabilities are not exploitable — that is until they are, as with Log4J, because of a failure to conduct proper dependency analysis.”

But as bad as the Log4j pandemic is, a silver lining would be if it becomes a catalyst to make SBOMs mainstream. Indeed, President Biden’s “Executive Order on Improving the Nation’s Cybersecurity,” issued May 12, 2021, specifically calls for every software product to come with an SBOM.

That, White said, should mean that, “software component information should flow more seamlessly between product vendors and their customers. Instead of calling every single supplier to confirm component information, the task now becomes a straightforward database search.”

For now, organizations should make getting the vaccine for Log4Shell their top priority. The FTC provides directions:

  • Update your Log4j software package to the most current version found here:
  • Consult CISA guidance to mitigate this vulnerability.
  • Make sure your company’s practices do not violate the law. Failure to identify and patch instances of this software may violate the FTC Act.
  • Distribute this information to any relevant third-party subsidiaries that sell products or services to consumers who may be vulnerable.

Do it now. If you fail to address Log4j, you will likely be in trouble with more than government agencies. Cybercriminals, much like a virus, don’t grant extensions or waivers.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Taylor Armerding

I’m a security advocate at the Synopsys Software Integrity Group. I write mainly about software security, data security and privacy.