The China SuperMicro Hack: About That Bloomberg Report

According to Jordan Robertson and Michael Riley in Bloomberg Businessweek, China has recently engaged in bulk supply-chain sabotage, corrupting thousands of servers on computers that end up in the server rooms of major U.S. companies such as Amazon or Apple, government systems and other locations around the planet.

The attack reportedly focused on the Baseboard Management Controller (BMC), essentially a second small computer built into the server. The BMC implements the Intelligent Platform Management Interface, a tool enabling a remote administrator to reset the computer, reinstall the operating system, and perform other tasks without needing physical access. It is effectively “god,” able to take over the computer entirely.

Robertson and Riley report that the Chinese People’s Liberation Army (PLA) has quietly been corrupting the BMC on servers built by the San Jose-based company SuperMicro, installing a backdoor into the computers belonging to at least 30 U.S. companies as a result. I will leave it to others to discuss the policy implications—but assuming the article is true, the technical implications are frightening.

The article describes a backdoor installed in the BMC that would “phone home” to a Chinese-controlled server. From there, if the PLA wanted, it could to take over the computer completely, just like anyone with physical access to the computer could—for example, patching the operating system or performing other such activities that would enables the PLA to directly control the operating system. Or, alternately, the PLA could simply do nothing, and wait for other, more valuable servers to instead contact the control server, taking them over instead. The report suggests that Amazon saw their affected servers “phoning home” yet took no further action.

It’s worth noting that Apple, Amazon and SuperMicro have all provided Bloomberg with very strong denials. That kind of pushback from those kinds of companies (or rather those kind of company lawyers) is a good reason to approach this story with a degree of skepticism until more information comes out. But keeping that in mind, the attack described in the article is actually plausible. SuperMicro specializes in computer server hardware and offers two advantages over other vendors: an ability to customize servers for mass installations and a somewhat lower cost than the competition. SuperMicro sells over $2 billion a year worth of servers, both under their own label and as the basis of “appliances” sold by others.

Robertson and Riley report a scheme in which Chinese intelligence bribed, threatened or cajoled at least four separate subcontracted manufacturing facilities in China to modify the design of SuperMicro server motherboards to include a small chip—smaller than a grain of rice—that would insert the backdoor into the BMC.

This scheme is less crazy than it might seem. Modern circuit boards are filled with small support chips, and the backdoor chip would appear to be just another faceless component to all but the most detailed examination. And while the Bloomberg article doesn’t go into the mechanics of how this would work, there’s one likely culprit: the serial EEPROM chip or a serial FLASH chip, which is used to store program and other instructions used during the startup process. The BMC itself loads at least some data from such a chip, which itself needs only two wires to communicate—so it would only take two connections for a rogue chip to mask the contents of a SEEPROM or SPI FLASH, replacing the contents and thereby corrupting the BMC by installing the backdoor code.

Serial EEPROMs can be nearly microscopic: ST Microsystems makes 128 Kb Serial EEPROMs that are less than one millimeter on a side. It would be straightforward to disguise one of these chips as a different component and simply have the disguised chip functionally replace an original serial EEPROM by changing the circuit board’s design so that the original EEPROM is simply no longer used. A small patch in the board design and an additional component in assembly is all it would take to replace a serial EEPROM with a second corrupted version, thereby corrupting the BMC’s programming.

A more sophisticated chip could intercept a Serial FLASH and perform targeted substitutions while occupying a similar near-microscopic space. There are other possible candidates for such sabotage, but serial memory seems the most tempting to me.

I expect we will see independent confirmation of this attack within a few weeks. Although Robertson and Riley only name two companies affected by the attack—Apple and Amazon—they write that the attack reached “almost 30” companies. Corporations are going to start testing their SuperMicro servers for unusual communications or internal connections—and if anything is discovered, at least one analysis will probably be made public. It will only take one company discovering one corrupted server and forwarding that server to a security firm hungry for press.

The Bloomberg story also explains a previous mystery: in 2016, Apple quietly removed all SuperMicro servers from their products due to an unspecified “Security Incident.” At the time the rumor was that SuperMicro provided a sabotaged BIOS—that is, the bootstrap program used to start the computer, another “god mode” target for compromise. Apple denied then that there was any security incident—just as they are denying one now.

This incident once again illustrates the “Coventry problem,” referring to Winston Churchill’s apocryphal decision not to prevent the bombing of Coventry in order to keep secret that British intelligence had decrypted the Enigma machine. Robertson and Riley describe a U.S. intelligence apparatus that knew of these ongoing attacks, but could not effectively notify the affected companies nor provide useful recommendations. If the intelligence community had warned these companies, it would probably have revealed to the Chinese that the U.S. was aware of these activities, as well as potentially compromise an ongoing FBI investigation described in the article.

Now, however, the secret is out. With the attack public, the NSA should be able to safely warn the victims without compromising sources and methods. The backdoor reportedly includes a “phone home,” a small request to a Chinese-controlled server. The NSA can likely use this phone home process to identify the victims of the attack, and undoubtedly has already done so. The process I have in mind would simply involve starting with one server, identifying any unique feature about that server and then scanning the entire Internet for all such servers. This requires no special abilities and is the sort of activity that independent researchers do all the time. Once the NSA has identified all these control servers, it could well be just a matter of looking through records to identify all U.S. computers that contact these hostile servers.

Then there is the question of whether the NSA is aware of other supply chains compromised in similar manners. If so, a quiet nudge may be a good idea. This style of backdoor can be very hard to find until one knows where to look, but is reasonably discoverable once the searcher pointed in the right direction.

Robertson and Riley described a very scary security incident, one potentially compromising dozens of U.S. companies and tens of thousands of computers. Even if this proves a false alarm, it is a sobering wake-up call. The Chinese PLA has the capability of corrupting the god watching from within our machines.