On October 4, 2018 Bloomberg News released an article: The Big Hack: How China Used a Tiny Chip to Infiltrate U.S. Companies. In this article Bloomberg reporters Jordan Robertson and Michael Riley claimed that the Supermicro embedded spying hardware, a chip no larger than a grain of rice on behalf of the Chinese government. The article made rather incendiary claims without substantive evidence to validate them.
According to the article, the discovery was made when Amazon hired a third-party security company to perform a security evaluation of products from elemental technologies. A company whose technology they were in talks to acquire. Elemental provides video encoding technology that is used by both government agencies and Media companies and ran on hardware they procured from Supermicro. Supermicro is a San Jose California based manufacturer of computer hardware with strong links back to Taiwan and China. They wrote that Amazon discovered and then disclosed findings to U.S. authorities of spy chips on the hardware used by Elemental.
The article squarely puts blame on China who leveraged Supermicro’s Chinese sub-contractors as well as the sub-contractor’s sub-contractors to implant the chip. The article also noted that subsequent to this discovery, Amazon, who also uses Supermicro in its data centers, further investigated the issue and found even more complex embedded spy chips on their servers.
The news shocked the industry. Many corporations shed themselves of thousands of Supermicro servers precipitously and there was a dramatically negative impact on Supermicro’s stock price.
I have personally had experience with these elemental servers. Both as the architect of ESPN’s security infrastructure, as well as architecting and developing network, security and operations for an entire elemental technologies infrastructure used by one of our startup clients. The elemental video encoding and processing hardware was a key part of their operation. If the platform was leaking data to China, the expectation is that we would see and have a record of the communications or at the least communication attempts in our logs.
Though possible, I doubt that Supermicro made special hardware for Elemental. The equipment we operated looked like standard 1U (1 Rack Unit 1.75″ Height x 19″ Wide x Up to 36″ Max depth ) Supermicro server.
So if there was a spy chip there are a number of things that can happen to all or select data it processes.
- Nothing.
- Data can be corrupted.
- Data can be stolen.
- Data can be deleted.
- A combination of the above.
If the data is corrupted, then it would be obvious within some period of time. Data deletion would be obvious as well.
If the data is stolen, then one fundamental function has to occur — it must use a network to communicate this data back to whomever is in control of the hardware embed. The network could be an RF (Radio Frequency) Network or a wired network.
Unless it is a bespoke compromise, an RF network would be challenging to implement. It would require a nearby receiver or transceiver both limiting its useful range and requiring custom frequencies for each operating region. Moreover, an RF doppler may be used to track the signal source, which would lead to the transmitting server or servers, not to mention that data centers have proven to be notoriously bad for RF signals. This approach is viable but not very practical. Also, RF transmissions are not easily hidden from spectrum analyzers — not to mention that the signal, even if encrypted, can be seen on the open airwaves by anyone. RF is just impractical for something as well monitored as Frequency spectrums.
So for argument’s sake — lets rule out RF leaks and backdoors.
Realistically, the server’s network is the most viable backdoor option. On Supermicro servers there are two network paths — the primary network interfaces and the IPMI management interface. In our use-case, data leaks via the primary interfaces of the Supermicro servers were completely contained. The only allowed communications were to and from the content storage systems along with one-way access from the management and monitoring networks. No other data flows were allowed. If the system attempted to communicate to China or other Chinese government controlled intermediary systems, it would have been to an arbitrary destination and we would have seen blocked outbound communications attempts. I do not recall any incidents for blocked communication attempts to arbitrary destinations.
The second approach, which many believe is most relevant to this scenario, is backdoor access via Supermicro’s Intelligent Platform Management Interface (IPMI). IPMI was spearheaded by Intel in the late 1990s with collaboration from a who’s who of computer manufacturers and networking companies. Realistically it has not evolved much since then — at least not in security terms. You can think of IPMI as a purpose-built device or an IoT within your server. As long as the server is connected to power, IPMI is accessible, even if the server is powered off or is without an operating system. IPMI is used for access, management and monitoring of the server hardware providing functions like remote keyboard, console video, serial port and USB access as well as the means to update and upgrade the computer’s Bios.
On Supermicro servers the IPMI is built on the Baseboard Management Controller or BMC. Supermicro’s implementation of the BMC to support IPMI can either be built-in the server motherboard or as an add-on module. Most recently though IPMI has been predominantly built in. The BMC board is typically built around a chip that functions very much like an IoT. It has its own ARM processor, memory and flash storage and in Supermicro’s case its own integrated web server. This IoT is intended to have a symbiotic relationship with the server hardware itself, though the server can function without it. If not available, only IPMI functions of remote hardware level access to keyboard, video, serial and USB would be unavailable.
Below is the diagram for one of these IoT IPMI chips. In the case of Supermicro, it has functional physical network interface (Dedicated NIC (Optional)), DRAM and Flash, though I have not seen a Supermicro server with onboard VGA or RGB video for the BMC. However it does have the shared LAN feature where one of the server’s primary interfaces can be used for the IPMI function.
But even with hardware level access, the purported spy chip has very limited function and capacity and ultimately would require 1) corruption of the operating system and 2) collection of more robust exploits from remote external systems to take effect. Both of these requirements need to evolve constantly to remain functional and relevant through OS upgrades and updates.
So yes it is possible for someone to use the IPMI platform as a backdoor and gain access to your system and data. But it seems a lot of suns, moons and stars have to align for it to work. Having said that, here is the one thing that even the most junior network, security or system administrator knows — Rule 1: you DEFINITELY DO NOT make IPMI networks Internet accessible and Rule 2: you don’t use the shared LAN feature. In many cases IPMI networks are plugged into not just separate vlans but separate switches. You must make your hardware management platform a closed loop network without any Internet level access. Why?
Even without the embedded chinese back doors, IPMI is an incredibly insecure platform. In fact it is horribly insecure and is still vulnerable to exploits that were addressed over twenty years ago. Again, the backdoor embed aside, IPMI provides access to a company’s technology holy grail. So everyone I have ever known treats IPMI as if it is inherently insecure — BECAUSE IT IS INHERENTLY INSECURE.
The bloomberg article with its anonymous sources is very short on technical detail about this issue. Their story has been refuted by Amazon and Apple, both one of the largest consumers of Supermicro servers. But you would expect denials from these companies who stand to lose customer confidence from the incident. However their denials have been unusually detailed and specific and backed by both the NSA and Homeland Security, who have publicly questioned the merits of the story.
There is no doubt in my mind that Chinese technology manufacturer’s hardware lacks integrity. Technologies from Huawei and ZTE are built on undoubtedly insecure and even bugged hardware. So much so that by law they may not be used by U.S. government agencies and critical infrastructure. I need to see more detail from the people who are making these assertions about Supermicro, though. The lack of detail around these incendiary claims is puzzling. Perhaps I’m missing something but if the servers are compromised, fundamental and foundational security practices should prevent backdoor access with good architectural practices or the lowly firewall, the most basic tool in a security team’s arsenal. However, I do have some questions for Bloomberg:
— Who was the third-party security company Amazon hired?
— How did Amazon’s security auditors get the original manufacturers schematics to know what was and was not part of the original manufacturers design? These are typically trade secrets and I doubt that Elemental Technologies had enough volume or sway to warrant custom designs or even access to schematics.
— If they were implemented by subcontractors, who QAed the systems and did the QA practice not account for hardware fuzzing and abnormal behavior?
— Given the denials, why has Bloomberg not released more detailed information about the issue?
— Why have none of the dozen or more anonymous sources for the Bloomberg article come forward?
There can be logical explanations to all of the above questions to validate the article and its claims. Since Supermicro has over 600 motherboard models, I am trying to identify the units in question for a forensics exercise that will hopefully get to the bottom of the issue. Ultimately, compromised hardware has and continues to be something that all security teams must account for.
About the Author
Babak Pasdar is an ethical hacker and a globally-recognized expert in Cyber-Security, Cloud, and Crypto-currency. He has a reputation for developing innovative approaches and methodologies for the industry’s most complex security problems. Before Acreto, Pasdar brought the first proxy-in-the-cloud platform to market, even before the word “cloud” was coined. He called in security in the “Grid”. Named one of New York’s Top Ten Startup Founders over 40, he has built and successfully exited two Cyber-Security technology companies and his innovations have been widely adopted by the industry.
About Acreto IoT Security
Acreto IoT Security delivers advanced security for IoT Ecosystems, from the cloud. IoTs are slated to grow to 50 Billion by 2021. Acreto’s Ecosystem security protects all Clouds, users, applications, and purpose-built IoTs that are unable to defend themselves in-the-wild. The Acreto platform offers simplicity and agility, and is guaranteed to protect IoTs for their entire 8–20 year lifespan. The company is founded and led by an experienced management team, with multiple successful cloud security innovations. Learn more by visiting Acreto IoT Security on the web at acreto.io or on Twitter @acretoio.
