Reminder: A weak link in your software supply chain makes the whole chain weak
There’s a downside to being popular. Especially in the online, digital business world, which these days is pretty much the entire business world.
Because if something that almost half the world is using has vulnerabilities that can be exploited by criminal hackers, it’s obviously a much bigger problem than if that product or service had just a small share of the market.
In the past few weeks, that problem applied to Jenkins, an open source automation server for building, deploying, and automating software projects. Jenkins is one of the most popular and powerful tools for what is known as continuous integration and continuous delivery/deployment (CI/CD). Basically, it helps software developers do what their organizational leaders want the most — build, test, and deploy applications faster.
No wonder it’s popular. And its growth curve looks like enough to make an investor salivate. According to the Linux Foundation, “Monthly Jenkins pipeline jobs defined grew 79% during the period June 2021 — June 2023, from 27,105,176 jobs per month to 48,625,398 jobs per month.”
Unfortunately, popular doesn’t mean perfect. That’s not a knock on Jenkins any more than on any other digital vendor. It is no outlier — virtually every piece of software out there is, or was, imperfect. That includes open source software, which comprises the large majority of most modern codebases.
The Synopsys Software Integrity Group has tracked the open source software ecosystem for nearly a decade and its annual Open Source Security and Risk Analysis (OSSRA) report has documented that open source is in virtually every modern codebase and makes up about 77% of the software in those codebases. Which makes open source much more popular than Jenkins.
Which should be a warning to organizations that are likely using thousands to millions of open source software components. If you don’t keep track of them, along with who created them and is maintaining them (or not) you’re asking for trouble.
Vulnerable here, vulnerable there
As has also been said many times but still bears repeating, software risk is business risk. Indeed, if you can’t trust your software, your customers can’t trust you. And as the most recent OSSRA report also found, of 1,067 codebases analyzed by researchers, 84% had at least one vulnerability, and 74% had high-risk vulnerabilities.
In this recent case, cybersecurity vendor Sonar’s Vulnerability Research team announced that it had discovered two security vulnerabilities in Jenkins.
The first, ranked as “critical” (the worst) by the Common Vulnerabilities and Exposures (CVE) catalog maintained by the MITRE Corporation and labeled CVE-2024–23897, “allows unauthenticated attackers to read a limited amount of arbitrary files’ data, and ‘read-only’ authorized attackers to an entire arbitrary file from Jenkins’ server,” according to a Sonar blog post.
The post added that “attackers could leverage this vulnerability by reading Jenkins secrets, to escalate privileges to admin and eventually execute arbitrary code on the server.”
Arbitrary code execution means an attacker can run any command or code on a target machine or process — in other words, take control of it to do malicious things.
A second vulnerability, ranked “high severity” (second worst) and tracked as CVE-2024–23898, could allow “cross-site WebSocket hijacking [that could] allow an attacker to execute arbitrary CLI [command line interface] commands by manipulating a victim to click on a link,” according to the Sonar blog.
“Manipulating a victim” is also known as a social media or phishing attack, designed to trick a victim into clicking a malicious link.
Good news, bad news
The good news is that nobody should be a victim of either vulnerability now, unless they’re not paying attention. Both have been fixed with updates — Jenkins version 2.442, and LTS 2.426.3. The company has also posted security advisoriesproviding detailed explanations of the vulnerabilities, what an attacker would have to do to exploit them, and how to mitigate or do a workaround if installing an update immediately isn’t feasible.
The bad news is that, as is too often the case, not everybody pays attention, which can lead to major damage. As the cliché puts it, a chain is only as strong as its weakest link. In the digital world, as numerous experts have said for years, if a single link in your software supply chain is vulnerable, the whole chain is vulnerable. In this case, organizations using Jenkins that don’t install the updates are vulnerable. And it’s a sure bet that with the vulnerabilities being public, criminal hackers will be looking to exploit them, hoping to find organizations too careless or clueless to have installed the updates.
They often don’t have to look very hard. Some of the worst supply chain vulnerabilities remain unpatched by thousands or even millions of users. One of the most notorious was Log4Shell, a group of vulnerabilities in the open source Apache logging library Log4j, (one of the most popular in the world) discovered at the end of 2021. At the time it became public it was already present in hundreds of millions to billions of systems, services, websites, and devices. If Log4j was a link in your software supply chain, you were vulnerable.
And perhaps still are. The latest estimates are that a third or more of Log4j users still haven’t installed updates to fix the vulnerabilities — even though they have been available now for years.
Even though the damage from attacks on Log4j range from crypto mining to, as the Federal Trade Commission put it in a warning at the time the vulnerabilities were discovered, “a loss or breach of personal information, financial loss, and other irreversible harms.”
The number of potential victims from the Jenkins vulnerabilities isn’t even close to those at risk from Log4j but is still in the millions. The Linux Foundation reported that there are about 25 million developers globally. Given Jenkins’ 44% market share, “this means approximately 11,264,000 developers use Jenkins for CI.”
There are no statistics yet on how many of those developers have failed to install the available patches, but “unpatched vulnerabilities are a reality of application development,” said Mike McGuire, senior software solutions manager with the Synopsys Software Integrity Group.
“Our own research found that the average age of a vulnerability [in analyzed codebases] is more than two years. While this may be acceptable for less severe, unexploitable vulnerabilities, it’s not the case for Log4Shell, which can still be found in 11% of Java codebases,” he said, adding that “this is a critical vulnerability that impacts nearly every network-connected Java application and is extremely easy to exploit. Unfortunately, many teams lack the visibility to even realize that their applications are impacted by vulnerable versions of Apache Log4j.”
You can do better
But that bad news leads to what ought to be some more good news. The ways to improve that visibility are now well established, especially for open source software components. An automated tool called software composition analysis(SCA) can help find open source components, any known vulnerabilities in them, and any potential licensing conflicts.
An SCA tool can also help to create and maintain a Software Bill of Materials (SBOM), an inventory of all the components in a software project. SCA and SBOMs won’t make an organization bulletproof — nothing will — but it does help make organizations aware of vulnerabilities when they are discovered, which means they can be prepared to apply an update as soon as it is available.
President Biden has recognized the value of SBOMs — in his May 2021 “Executive Order on Improving the Nation’s Cybersecurity” he explicitly called for organizations, both public and private, to create and maintain SBOMs — something the software security industry has been recommending for years. The Biden order even called for federal agencies to stop buying any software products that lacked an SBOM. But the original deadline for that to happen has come and gone. No word on when such a ban might take effect.
But the Jenkins’ vulnerabilities are just the latest — there will be more — examples of why it shouldn’t take a presidential order to give organizations an incentive to implement software supply chain security.
That security is as important as your own personal security. It’s a bit like giving your house key to the plumber. If somebody steals it from him, it doesn’t really matter that it wasn’t stolen directly from you — you’re now at risk.
