In a recent blog, Babak Pasdar highlighted a Bloomberg report that claimed China had embedded hardware spy chips on servers from Supermicro. Supermicro provides data-center servers used by many companies from small startups to the likes of Amazon and Apple. Bloomberg claims that the spy chips were discovered by a security auditor hired by Amazon AWS. This audit was part of an acquisition due diligence of Elemental Technologies, a platform specializing in multi-screen video processing.
Bloomberg claims that Amazon and Apple are among the organizations impacted by the alleged Chinese spy chip. And one-by-one they have all denied that the story has merit. However, Bloomberg, a model agency in news reporting, has refused to offer any additional information or alternatively to pull the story.
There is a lot about this story that doesn’t pass the smell test. If Supermicro servers have been compromised, it is a huge story. Though not a household name like Dell or HP, Supermicro is one of the top data center server platforms on the market. It is considered to be a good product with global availability at a fair price.
In the article Bloomberg makes a pointed accusation yet offers evidence that at best is vague. In the previous blog, we asked several questions:
- Who was the Security audit company that discovered the spy chip?
- How did they get access to schematics to do chip by chip validation of the hardware? Schematics that in any scenario would be considered trade secrets.
- If the spy chips were secretly installed by a Supermicro contractor as the article claims, who QA’ed the hardware and why was the chip not discovered during the QA process?
- Given the emphatic and detailed denials by both companies and the U.S. government, why has Bloomberg not released more detailed data to back up their claims?
The implications are that China has backdoor access to countless systems, hosting applications and data, impacting thousands of companies and millions of individuals. The integrity of corporate, government and critical infrastructure is at stake — as well as personal data for large swaths of the population.
Is This Realistically Possible?
Bloomberg provided very little detail, and what they did provide was at best vague and not evidence-worthy. Based on the information they did provide, the industry take-away is that this vulnerability is via the server’s IPMI interface. IPMI is an always-on IoT embedded in a server to manage the hardware, even if the server is powered off.
As presented, the IPMI platform can theoretically be manipulated to function as a back door, providing access to the server’s network, system memory and the system bus. You can learn more about this in Pasdar’s previous blog on this issue here.
Having said that, for Bloomberg’s vague spy chip explanation to work, you need a Supermicro motherboard with an on-board IPMI, and then many, many, many things have to line up for the compromise to work.
First, an Internet accessible IPMI connection with stateful outbound access is needed — something no self-respecting organization with even a moderately experienced infrastructure team would have. The chip Bloomberg presented in their article is just physically too small to store and execute the necessary code to fulfill its purpose, so it would also need to connect and download software from an external server. Hackers will never use an external server they own that references back to them. It would lead authorities right to them and there would be no plausible deniability. The server is most likely another compromised system on the Internet. Moreover, the external server’s address isn’t hard-coded into the chip. Compromised servers are disposable since the compromise may be discovered and addressed at any point — or the system moved or decommissioned.
If this occurs, the entire effort of the compromise would be a complete waste. A process like fast-fluxing or something similar would be used to enable the spy chip to connect to an ever-changing botnet network of external servers. Fast-fluxing was specifically developed to control botnets without compromising the bot-master’s identity. It is a technique where the spy chip and the external server would meet to communicate at a particular fully qualified domain name (FQDN) at a particular time. Many Different FQDNs spanning many different domains may be used to deliver content to the spy chip based on the then valid compromised IP addresses hosting the malware.
The spy chip then needs to integrate into the server’s OS, on-the-fly, during the boot process. This requires injecting the appropriate code for the specific OS used on the server. The OS could be one of dozens, if not hundreds of possible options since the Supermicro B1DRi motherboard that Bloomberg claims is compromised, is certified compatible for many different OSes and associated versions. This includes 32-bit Red Hat, SUSE, Ubuntu and FreeBSD as well as many versions of 64 bit Red hat, Fedora, SUSE, Ubuntu, Solaris, FreeBSD, Centos and Windows. Further, it also supports multiple hypervisor versions of VMWare, KVM and Xen Server, not to mention Amazon AWS’s proprietary hypervisor. Each one of these OSes needs a different code. Even each version of the same OS may require an altogether different code to be injected into the compromised system. Consider how quickly the spy chip would have to act to intercept local boot code, determine the OS brand, distro and version from a smattering of code flying on a computer’s bus, perform the fast-flux operation and fetch the appropriate compromise code from the appropriate server.
All of this — which is a lot — needs to happen for the spy chip to work.
Breaking Down the Claims
Now let’s break down Bloomberg’s claims further. In the article they present a graphical image of a Supermicro motherboard and strip away components until the spy chip can be seen. The motherboard they present is a Supermicro B1DRi with an AOC-GEH-i4M add-on module. As shown on the Supermicro web site, the B1DRi is designed to host up to two Intel E-2500 v3/v4 CPUs and up to 256Gb of 288 pin DDR4 memory and can be mounted to a sled with its own hard-disks. However it is not a standalone server and needs to be mounted in a Blade Enclosure to function.
The enclosure provides power, hosts a network switch and most importantly has a shared IPMI management board plugin. If the spy chip works through the IPMI, how can Bloomberg show the spy chip placed on the motherboard, when the IPMI for the board is an external module in the enclosure? It looks like the IPMI must be individually linked to each server blade to manage that blade. The IPMI IoT is an external module plugged into the enclosure and to be used, it needs to be individually assigned to each of up to 16 server blades in the enclosure. If that is the case then there is a 1 in 16 chance of compromising a server and even then, it would be opportunistic and inconsistent depending on which blade the IPMI may be set to manage on boot.
Now let’s discuss the chip Bloomberg presented in the article. If the insanity of the logistics to effectuate this hack is not enough to make you call Bloomberg’s story Bullshit, then their presentation of the spy chip should. The chip presented IS NOT A SPY CHIP, it is an RF Balun. A standard, off-the-shelf Surface Mount Device (SMD) that converts between balanced signals and unbalanced signals, hence the name Bal-Un. If you look at the Stesys or Farnell sites, they are two of the many component providers who sell them. You too can have one for a mere $1.67.
And if the pictures were supposed to be mere examples of what a spy chip might look like and the type of motherboard it could be embedded on, they certainly did not present it that way.
Also, consider that a motherboard is an incredibly complex piece of equipment. These types of motherboards need to be extremely high performance and extremely compact at the same time. This makes them extremely dense. They are almost always multi-layer boards where traces connecting the various electronic components exist on as many as a dozen different layers. And these systems are delicate, their operation requires the various electronic components to operate harmoniously. Frankensteining hardware to the system would be at the very least — challenging.
The majority of people within a company involved in R&D, design, procurement, manufacturing and testing of the motherboards are often sequestered into groups with access that is limited to specific functional domains. Very few people have complete access to the designs and schematics for the entire board. And this almost never includes subcontractors or some small security company out of Canada doing technical due diligence for a mundane acquisition. Furthermore, the people charged with manufacturing are typically not the same people who do quality assurance (QA). The job of QA is to test every permutation of every function. We have to believe that QA’s most fundamental tests would catch something as overt as communications where the spy chip tries to identify, fetch and inject packets on-the-fly.
The number of people that would need to be turned or paid off would be staggering. As many as 30–50 people would need to be engaged throughout the supply chain spanning multiple companies and countries. An amateurish and incredibly messy way to run a covert op.
How Everything Comes Together
Because of the vague assertions, it is tough to argue definitively that any one aspect of the article is wrong, however when you put it all together:
- We don’t know of many security companies that do reverse engineering on PCs as part of their due diligence.
- Schematics are trade-secrets and almost never available for complex multi-layer motherboards. How could the security company have had access to schematics?
- The sheer number of people that need to be involved in implementing the spy chips is staggering and doesn’t make sense for this type of effort.
- The QA process, one known to be particularly meticulous, never caught the issue.
- The ridiculous complexity of the hack where the sun, the moon and the stars have to align for it to work.
- Not only is this compromise overt and easy to identify, but the vast majority of organizations have built-in defenses against this attack vector — especially Apple and Amazon.
- The need for an Internet accessible IPMI network.
- The need for the chip to fast-flux, connect to a remote system and pull-down compromise code while the system is booting.
- The complexity of pulling a different code set on-the-fly for each of the hundreds of unique operating system and revision combinations.
- The B1DRi motherboard being part of the blade system without any on-board IPMI, which can only be managed one blade at a time.
- The vagueness of the charges and lack of any supplemental follow up, while Bloomberg continues to sit silent.
- And trying to sell us that an off-the-shelf $1.67 RF Balun is a spy chip.
For these reasons, many of us believe the Bloomberg story just doesn’t have a leg to stand on. Bloomberg has made explosive allegations. They have had a drastic negative impact on Supermicro’s stock price — down 50% as of this writing. Their story is barely, if at all, viable. The information they provided was amateurishly vague. Their silence in the face of the backlash speaks volumes. And yet they continue to stand by their story and not recant. Add Bob Flores and Babak Pasdar to the growing list of skeptics.
If you have evidence, then present it and if you were conned it is understandable — but please stand up and own it.
About the Authors
Bob Flores, a former Chief Technology Officer of the Central Intelligence Agency (CIA), was responsible for many of the technologies used by the always at-risk government agency. Over his 30 year career in the US intelligence community, Bob served in a variety of areas, including the National Clandestine Service, Directorate of Intelligence, and the Directorate of Support.
Babak Pasdar is an ethical hacker and a globally-recognized expert in Cyber-Security, Cloud, and Crypto-currency. He has a reputation for developing innovative approaches and methodologies for the industry’s most complex security problems. Before Acreto, Pasdar brought the first proxy-in-the-cloud platform to market, even before the word “cloud” was coined. He called in security in the “Grid”. Named one of New York’s Top Ten Startup Founders over 40, he has built and successfully exited two Cyber-Security technology companies and his innovations have been widely adopted by the industry.
About Acreto IoT Security
Acreto IoT Security delivers advanced security for IoT Ecosystems, from the cloud. IoTs are slated to grow to 50 Billion by 2021. Acreto’s Ecosystem security protects all Clouds, users, applications, and purpose-built IoTs that are unable to defend themselves in-the-wild. The Acreto platform offers simplicity and agility, and is guaranteed to protect IoTs for their entire 8–20 year lifespan. The company is founded and led by an experienced management team, with multiple successful cloud security innovations. Learn more by visiting Acreto IoT Security on the web at acreto.io or on Twitter @acretoio.
