Analysis of a Supply Chain Attack
On November 20th, 2018, a backdoor was discovered in the npm module event-stream, injected by GitHub user right9ctrl via the malicious flatmap-stream module, which was created by GitHub user hugeglass. The backdoor would only be activated if the code was included in BitPay’s open source wallet Copay, or any forks that did not modify the project’s description, and would exfiltrate the wallet’s private key. The official Copay wallet versions 5.0.2 to 5.1.0 were released with this backdoor in place, and building directly from the source code between October 25th, 2018 and November 26th, 2018 would have produced a backdoored version of Copay.
Let’s take a look at what happened.
The flatmap-stream module backdoor existed only in the minified version of the code that was uploaded to npm, and was never uploaded to GitHub, so a simple inspection of the index.js file or the resulting minified code on GitHub would not reveal anything suspicious. The package.json manifest for the repository was fairly boilerplate, and specified the author of the repository as “Antonio Macias”. At the time of the discovery, there was only one commit in the repository, but npm had three versions published. A copy of the npm page was retrieved from the Google cache after npm removed the malicious package. According to npm, the package had been downloaded 867,232 times, though it is unclear how many of these downloads were the malicious version or the previous version which did not contain the malware.
The backdoor that was included in the flatmap-stream package was distributed by including flatmap-stream as a dependency in a popular node module, event-stream. The event-stream repository was previously managed by GitHub user dominictarr, who stopped making major contributions to the repository in 2015. GitHub user right9ctrl emailed dominictarr and offered to maintain the repository, and dominictarr gave right9ctrl write access after an attempt at transferring ownership was prevented by a fork right9ctrl created. right9ctrl made their first contribution to the repository on September 4th, 2018, and continued to make small bug fixes and run maintenance tasks on the repository until September 20th, 2018. On September 9th, 2018, right9ctrl added the flatmap-stream dependency, which at that time was at version 0.1.0. This version did not contain the backdoor, as a diff of the 0.1.0 and 0.1.1 versions on npm reveals the first stage of the backdoor was appended to the minified code with version 0.1.1.
On September 16th, 2018, right9ctrl removed the flatmap-stream import and replaced the code with functions baked into event-stream, and in a following commit removed the flatmap-stream dependency entirely, while simultaneously bumping the major version from 3 to 4. This major version bump prevents packages depending on event-stream from updating to the cleaned version due to semver, while also making master clean again. Almost three weeks after this change is made to event-stream, the malicious version 0.1.1 of flatmap-stream is pushed to npm by hugeglass on October 5th, 2018. event-stream was configured to pull this updated version in when flatmap-stream was originally added to version 3, ensuring future builds that imported event-stream version 3 would be compromised. This goes seemingly unnoticed until October 29th, 2018 when an issue is opened in the nodemon project reporting a deprecation warning at startup. It turns out that the code injected into flatmap-stream used an API deprecated in Node 10, crypto.createDecipher.
This attack was first described as such in an issue opened on event-stream on November 20th, 2018, and received much more attention following a Hacker News post on November 26th, 2018. It was at this point I was made aware of the issue and began investigating. As is often the case, the internet hive mind was already moving at breakneck speed to dissect the attack, and the issue on event-stream turned into a productive and efficient conversation that revealed the nature of the attack.
The Investigation
It was clear from the original issue opened to event-stream that the payload was AES encrypted, and so the first task in assessing what the attack actually did was to decrypt the payload. Participants in the issue made good progress reversing the malware and discovering that the decryption call supplied the key as process.env.npm_package_description .
By discovering what values could be in that variable it would be possible to create a set of potential keys and eventually decrypt the payload. npm documentation suggests that this value is defined by the packagedescription key in the package.json manifest, but other participants pointed out, and demonstrated, how this value can also come from a project’s README.md.
A complete archive of npm metadata is available and it is suggested that, worst case, a near-comprehensive list of package descriptions could be generated for a bruteforce.
Eventually, GitHub user maths22 discovers the correct key is “A Secure Bitcoin Wallet”, and suggests the target could be a fork of BitPay’s copay wallet for Dash. A full copy of the stages of the payload are available here. In the final stage of the malware, we see that the private key is retrieved, encrypted with a public key, and uploaded over HTTP.
The copay-dash fork did not change the original package description, meaning this attack would work on the original Copay wallet as well. Given the relative popularity of Copay I believe it is safe to assume the target of this attack was Copay.
An issue is opened on Copay, where the Copay team confirmed the affected versions and promptly removed the backdoor.
A search of GitHub indicates an upper bound of 190 projects that could be affected by this attack. My search didn’t seem to include the word “A” like I asked, so this estimate is overly broad. I’m not sure why the search did not return only exact string matches.
It is at this point unclear how much, if any, cryptocurrency was stolen as part of this attack. The malware exfiltrated private keys; it did not send the balance of the wallet to a different address. While this makes it hard to assess the scope of the attack from this analysis, it means that users who were impacted have a chance to move their coins before the attackers do. If you stored funds in Copay versions 5.0.2 to 5.1.0 (or any derivative forks), immediately send those funds to an address controlled by a different key.
Indicators of Compromise
The issue opened to BitPay includes a comment highlighting the IOCs, including:
- The hostname
copayapi.host - The IP addresses
51.38.112.212,145.249.104.239, and111.90.151.134
It is possible that these IP addresses are not the property of the attackers and rather compromised servers that are being used by the attackers.
What can we learn?
Using dependencies always requires some degree of trust.
When importing a new dependency, engineers should do their best to understand the quality of the repositories they are including into their projects. This isn’t a catch-all, as it is impractical to audit every dependency your dependency depends on, and all their dependencies, etc. But still, importing unmaintained packages should not be considered safe.
The software development ecosystem would benefit from a tighter relationship between source code and published packages.
In this attack, the code hosted by npm differed from the code visible on GitHub. This contradicts the assumption most developers would make, that if the module is built from that repository, it contains the code from that repository. A better system would be able to validate that the code hosted by a dependency repository is generated from the source repository at all times.
Focusing on fixing the issue in times of crisis helps everyone.
It can be tempting to point out ways in which this all could have been prevented, and to point fingers at those who might seem responsible. The fact is, however, that there was no single point of failure in this attack. When the discussion was focused on fixing the issue at hand, and tabling the conversation around blame and long term fixes, the conversation became a lot more productive, and the community quickly tackled the issue.
In conclusion
BitPay did not do anything outside the norm in software engineering that resulted in this attack. They used a popular dependency, something most engineers would never think twice about. This type of attack will continue to be a problem until security controls around dependency management are improved. It’s on all of us to help build a future where the economics of security make this attack less practical. Attackers will only invest their resources in places where their costs are less than the reward, and by improving the security of dependency management we can make the supply chain attack too costly to be worth their time.
Thanks for reading! Follow me on twitter to stay updated on future articles.
