Supply Chain Security Speculation

Everything thrown at the wall that seemed to stick

Bloomberg accuses the PLA of hardware tampering supply chain attacks. If this is at all true, it is a pretty big deal. If it is completely false, it is still a pretty big deal (but thats between Bloomberg’s lawyers and SuperMicro, the company allegedly shipping the hacked server boards.) Supply chain attacks are a scary vulnerability because the root of trust has to start somewhere, and if it starts in a no-name Chinese subcontractor factory…it’s maybe not the ideal foundation. I’ve attempted to collect as much actual information as I can based on the Bloomberg statement:

The illicit chips … were connected to the baseboard management controller

Before the wild speculation though, it must be mentioned that the story is short on evidence and high on flat out denials.

Update: more evidence from an earlier Ars Technica article seems to support the Bloomberg report.

Update: Amazon is pretty emphatic that everything Bloomberg said about them and Supermicro is wrong.

Update: In 2016 Apple did have security issues with Supermicro, but the circumstances are far from clear. It looks like maybe Apple is bluffing Supermicro about a bad firmware, then ghosts. If they actually did find a problem, engage in a coverup, then dump the whole problem on the .gov, it explains the weird messaging going on.

Update: Apple comes out swinging with another “nope!”

Update: ServeTheHome has a good write up on BMCs, but I think they may be attributing too much technical coherence to the Bloomberg article. The hypothetical attack – altering the password verification routine – is not particularly practical for an attacker. A backdoor with direct memory access, and just a few operations (read, write, jump) would be simpler, more robust, and much more useful.

Update: put Supermicro in the emphatic “nope” column.

Update: worth mentioning that Bloomberg (and these reporters) have a couple erroneous infosec stories that should have been retracted, but weren’t.

Something is rotten in the state of supply chain attack reports

All of the named companies in the report flatly deny pretty much every statement in Bloomberg’s article. These denials are not “non-denial” denials, but directly refute specific statements of fact in Bloomberg’s report, as well as explicitly denying the core premise of the supply chain attack.

Bloomberg claims that the circa 2015 modchip, about “the size of a grain of rice,” was discovered by a third party security auditor. I can think of people who are capable of detecting this sort of modchip hack. I cannot think of a reason why a due diligence audit of a server would go down to that level.

On the other hand, Baseboard Management Controllers (BMC) and the Intelligent Platform Management Interface (IPMI) protocol are a horrendous tire fire for cyber security. That’s why Amazon’s statement about the audit rings true to me.

The pre-acquisition audit described four issues with a web application (not hardware or chips) that SuperMicro provides for management of their motherboards. All these findings were fully addressed before we acquired Elemental. The first two issues, which the auditor deemed as critical, related to a vulnerability in versions prior to 3.15 of this web application (our audit covered prior versions of Elemental appliances as well),

Auditing multiple versions of the same server is already a lot of work, scouring them for camouflaged grain of rice sized backdoors seems a little excessive. The four issues:

• Two critical issues in the BMC web server (accessible over IPMI)
• Two non critical ones (probably about encryption or lack thereof) that were mitigated by Amazon’s planned deployment

These findings ring true to me, this is what a typical infosec due diligence analysis is going to do — look at the interfaces and ports, see what functionality there is, what bugs there are, and what needs to be hardened/fixed.

Stripping the boards and hunting for tiny camouflaged rogue modchips is pretty intense for an audit. However, if the modchip was buggy and alerted the auditors to dig deeper, then it is certainly possible. Things that could tip the auditors off:

• firmware errors when reflashing the modchipped unit (checksums?)
• unusual network traffic (e.g. beaconing) generated by the modchip
• anything else weird and unusual that raises redflags

Supply chain attacks exist. Is this article accurate? It feels a little off, but I don’t know.

What do we know?

There’s not much we can speculate about the modchip because the Bloomberg description of whatever it does is gibberish. It is safer to simply examine what is known about Supermicro’s server boards.

• Supermicro boards have third party BMC hardware to handle IPMI
• There are at least three hardware providers: ASPEED, ATEN, and Nuvoton
• ASPEED and Nuvoton use AMI software. ATEN has their own software stack
• All Supermicro IPMI controllers appear to provide an extensive range of functionality that would be useful for an attacker

See the full range here, but the highlights include:

• Keyboard Video Mouse (KVM) over IP
• SSH
• Serial over LAN (SOL), and SSH over SOL
• Web server (default login: ADMIN:ADMIN)
• Remote power management…

Servers get hacked via exposed BMC without a modchip all the time, just scan for the IMPI web console and use the default password. There are other ports to check for as well:

• TCP 80, 443: web interface
• TCP 3520, 5900: KVM access
• TCP 623: menu access, allowing full control of the hardware

Good supply chain attack?

To compromise a server with a tiny modchip, a backdoor into the BMC would be pretty good. For example, a simple ICMP shell that beaconed out and provided basic commands to interact with the system would work in many places. The modchip’s backdoor would have to be more complex if the idea was to breach a hard target, but the BMC is certainly a good place to start.

So, what’s the deal?

For me, Bloomberg’s article could go either way. The logic of backdooring the BMC makes a lot of sense. Whether it happened in this case — given all the categorical denials — I have no idea.

The real takeaway from this is that IPMI is a raging tire fire, BMCs are Satan spawn, and never ever expose IPMI interfaces to the Internet. Unless you want hackers, because that’s how you get hackers.