Some 20/20 hindsight on the chaotic disclosure of Log4Shell
By now, the “what and how” surrounding the chaotic disclosure of Log4Shell, one of the most severe and widespread software vulnerabilities to hit the digital world in recent history, is reasonably well-established.
In an industry where “coordinated disclosure” of such vulnerabilities is supposed to prevent hackers from finding out about a software bug before there is a patch available, there is wide agreement that in this case it wasn’t coordinated. A lot of hackers knew about it before their potential victims did. Paul Roberts, editor-in-chief of The Security Ledger, did a podcast on it titled “Unpacking Log4Shell’s Uncoordinated Disclosure Chaos.”
The “why”? Not so well established. But first, a brief summary of the “what.”
On Nov. 24, the day before Thanksgiving, a member of Alibaba Group Holding Ltd.’s cloud-security team sent an email to the Apache Software Foundation (ASF) reporting a software vulnerability in ASF’s open source logging library, Log4j (written in Java, one of the most popular programming languages), that he said could have “major impact.”
Indeed, it could allow hackers to achieve “remote code execution” — the remote takeover of a computer. And since there are hundreds of millions to billions of computers using Log4j, “major impact” now sounds like a major understatement.
But it set off a mad scramble by the volunteers of the nonprofit ASF, who set about trying to do the right thing — create a patch to fix the vulnerability — fast.
Unfortunately, two critical, and critically bad, things happened over the next couple of weeks.
First, as reported by Bloomberg and others, Alibaba notified the ASF group on Dec. 8 that “someone had just revealed the details of the vulnerability on a Chinese blogging platform.” That “someone” uses a pseudonym and didn’t respond to a request for comment from Bloomberg. But the post made Log4Shell known to the world, and the world’s bad guys began to move quickly.
While the Apache team did complete an update/patch less than 24 hours later, by then hackers had begun “mass exploitation” of Log4Shell. By mid December, attacks on Log4j were coming at the rate of more than one per second, or around 100,000 per day. The National Vulnerability Database gave Log4Shell a severity rating of 10.0 — out of 10.0.
Second, the patch the team created to fix the flaw was flawed itself, meaning attackers had more time during which even patched applications and systems were vulnerable.
Now, several weeks later, with Log4Shell still considered an emergency — U.S. Cybersecurity and Infrastructure Security Agency (CISA) Director Jen Easterly called it “one of the most serious (vulnerabilities) I’ve seen in my entire career, if not the most serious” — the 20/20 hindsight analysis of Log4Shell’s disclosure in a very uncoordinated way has ramped up a bit.
Mark Curphey, founder of the Open Web Application Security Project, which focuses on open source cybersecurity projects, and also cofounder and chief technology officer of Open Raven, told Bloomberg the event illustrates the need to take the security of open source software more seriously.
The fact that it’s free (with some licensing constraints) means those who use it aren’t “customers” so they don’t get direct notification if a vulnerability is discovered.
“Reusable code means reusable vulnerabilities and software supply chains feed the world’s technology consumers. There lies a perfect storm,” Curphey said.
Or, put another way, since open source software is deeply embedded in every business, if it’s at risk, then your business is at risk.
And the stakes of a vulnerability like this means the hindsight analysis has also raised the question of why, when a template for coordinated disclosure of vulnerabilities has been around for decades, was this one so chaotic?
Roberts noted in his podcast that in 2008, the late security researcher Dan Kaminsky discovered a serious bug in the Domain Name System internet technology, but “worked quietly for months” with both public and private entities “to get patches written and distributed — all before details of the vulnerability were made public. Vendors worldwide were able to take steps that largely mitigated the risk of attack before any details of the flaw became publicly known.”
Obviously, that didn’t happen with Log4Shell. Roberts and his guest, Mark Stanislav, vice president of information security at Gemini, both said one major reason was that neither the Alibaba cloud team nor the ASF team sought the help of a Computer Emergency Response Team (CERT).
“When we have these internet-scale issues, the vulnerability reporter — in this case the Alibaba cloud team or the Apache Foundation — generally should reach out to a higher-level entity like the CERT/CC [Coordination Center] at Carnegie Mellon,” Stanislav said, adding, “there’s been 30-plus years of coordinated incident response teams in information security now, so we have experience.”
“There are also other CERTs around the world, usually tethered to a country, municipality, or territory,” Stanislav said, “and that matters because those organizations have their hooks into hundreds if not thousands of vendors across the earth, and they can help on coordinated disclosure to make sure these issues come out in a measured, pragmatic and thorough way.”
Travis Biehn, technical strategist with the Synopsys Software Integrity Group, agrees. “Discoverers should engage with CERT and all parties should coordinate to protect the vulnerability information,” he said, but added that he didn’t think the ASF process was a complete failure.
“There’s perhaps a failure in that whether an issue requires the help of a CERT is ambiguous. And there may be some difficulty in getting a CERT involved in every bug report.”
But part of the “why” that didn’t happen is that, at least for now, engaging with a CERT is not the way the ASF handles the disclosure of any vulnerability, severe or trivial. Mark Cox, vice president of the ASF Security, said the organization’s vulnerability handling process “does not include doing prenotifications to vendors directly or via coordination centers in advance.”
“Where an issue affects a library that is so widely used, there is no practical way to prenotify affected vendors — there are too many parties to hold an embargo, too many parties to identify and notify, making it not only impractical but also greatly increasing the chance of early disclosure through leaks,” he said.
Art Manion, vulnerability analysis technical manager in the CERT Division of the Software Engineering Institute at Carnegie Mellon, said that argument has some merit, but he still thinks it should have been changed for an event of this magnitude.
“Apache’s disclosure policy, like other mature open source projects, is public, well-documented, relatively simple, and fair,” he said. But for Log4Shell, “I would have advised an exception to the policy. Get CERT/CC or another coordinator involved, convene an ad hoc group — under embargo — to plan the response, and choose teams and individuals who are able to reduce risk significantly for large proportions of the vulnerable population.”
If there had been coordination with major internet players who all agreed to the embargo, “the inaccurate mitigation advice and incomplete fixes could have been noticed and dealt with before they were public,” Manion said.
“Then, on public disclosure day, a significant number of users of service providers would already be patched, and major operating systems would have had updates.”
This, he acknowledged, amounts to “20/20 hindsight. I don’t know the details of what coordination did take place, and even this idealized coordination plan could be broken by premature public disclosure or active exploitation.”
Still, as noted earlier, it was an anonymous blogger who leaked details of Log4Shell before the ASF team had completed a patch. So while the Alibaba team may have intended to keep word of the vulnerability just between them and ASF, somehow it leaked.
Sammy Migues, principal scientist with the Synopsys Software Integrity Group, said while it appears that the Alibaba team reported the vulnerability only to the ASF on Nov. 24, “anecdotal reports are that it started getting exploited almost immediately after that disclosure, and possibly before,” Migues said.
“Once it’s being actively and easily exploited, everyone needs to know. Perhaps the question is, ‘Who do you tell when everyone needs to know now?’ It took hours for the news to filter through news channels, chat channels, social media channels, etc. before it entered the zeitgeist of the day,” he said.
The frantic rush to roll out a patch after both Alibaba and the ASF were blindsided by the publication of the Log4Shell details also likely led to the mistakes.
“The initial patch, as we now know, didn’t actually solve the problem,” Stanislav said. “And some of the initial mitigations that were released didn’t really mitigate the issue to the depth that maybe was believed. I think that’s one of the layers here — the speed with which you have to act when you don’t have coordinated disclosure.”
If nothing else, one upside of the magnitude of the damage that could be caused by Log4Shell is that it caught the attention of President Biden, who met with major tech executives last Thursday to discuss how to improve the security of open source software.
It will also be prompting some discussions at ASF. “We aim for agility and measure the time to get updates out for open source software in days rather than months,” Cox said, “so our policy includes a common process for disclosing and announcing the fix and publishing that data quickly once an issue is public.”
“We’re always eager to discuss and explore options, especially those that could help end users get patched quicker, and we are including CERT/CC in those discussions.”
If that happens, Manion said there are multiple ways the Carnegie Mellon CERT can help with coordinated disclosure. They include:
- Assistance, particularly in multi-party cases, with identifying and managing participants.
- Mediation and negotiation, particularly when researchers and vendors are not getting along.
- Technical verification. Is the report really a vulnerability? Can the CERT reproduce the claimed behavior? Does the mitigation or remediation work?
- Some degree of neutrality. “We aren’t the researcher or the vendor; we seek risk minimization for all. If we had to choose sides, it would probably be users,” Manion said.
- Some protection against legal threats, such as a large vendor taking legal action against a researcher.
- A lot of experience in helping to avoid common, and even uncommon, pitfalls.
All of which sounds like cooperation with a CERT could lead to better coordination that is much less chaotic.