The SolarWinds Breach: Why Your Work Computers Are Down Today

The information security news cycle went into overdrive yesterday afternoon. First, Reuters revealed that the Commerce and Treasury departments suffered significant intrusions. The Washington Post soon followed up with multiple sources attributing the attack to the Russian foreign intelligence service, the SVR—in particular, a portion of the SVR known as Cozy Bear—although there is no official attribution yet. Within a few hours, FireEye and Microsoft announced that this was a “supply chain attack” involving SolarWinds Orion software, and the Cybersecurity and Infrastructure Security Agency (CISA) issued an emergency directive. Today, it turns out that the attackers also compromised the Department of Homeland Security. SolarWinds revealed to the Securities and Exchange Commission that the breach may affect 18,000 customers.

There’s a lot going on, so here’s a quick guide for readers looking to get up to speed on the news.

What Is SolarWinds Orion?

SolarWinds Orion is part of the SolarWinds suite of network and computer management tools. One of the biggest problems with managing a large network arises from the scale: A network can have dozens of critical computers and hundreds or even thousands of computers overall. The SolarWinds solution suite includes not only monitoring capabilities, so that users can tell when a critical computer goes down, but also the ability to automatically restart services. As a consequence, this software is likely to be installed on the most critical systems in the enterprise—those that block the ability to get work done when they go down.

What Happened to SolarWinds Orion?

It appears that, in March 2020, someone managed to modify the SolarWinds Orion software during the build process—that is, the process that translates the human-readable code and merges it into a form that a computer can execute. This timing is based on both the Microsoft and FireEye analyses, as well as the reported versions affected by SolarWinds.

This modification included a sophisticated and stealthy Trojan program, designed to remotely control any computer that installed SolarWinds Orion. When customers installed the latest update, the Trojan program would start running on the victims’ computers. This is considered a software “supply chain attack”: The intended victims received a polluted copy of the Orion software directly or indirectly from SolarWinds.

What Did the Trojan Do?

The Trojan itself was a sophisticated and stealthy backdoor analyzed by both FireEye and Microsoft. Though the program produced some indications to tell if a computer was infected, it first waited 12 to 14 days before taking any action—a period of quiet designed to thwart analysis, as the malicious payload wouldn’t even start until the computer had been running for a long time. Then it started asking for a command-and-control server. Once again, this routine included checks to thwart analysis.

Only then would the Trojan start communicating with a remote server belonging to the attacker, with this communication disguised to look like the normal sort of web traffic generated by a benign automated tool. From there, the attacker now had effectively full control over the victim machine, including the ability to install additional software and perform other tasks.

What Then?

This is where the attacker gained extra capabilities. Among the “critical” enterprise servers that must remain up for the business to function are the authentication and Active Directory servers. These servers are incredibly important, as they identify users to other systems, say what permissions a user has to access data, and change the configuration of other machines. The attacker could use these abilities to move throughout the victim enterprise, gaining control, creating new accounts and accessing whatever data or resources were desired.

This would allow the attacker to become an “authorized” user with nearly unlimited reach, present effectively everywhere in the victim’s network. Notably, it seems that the attacker used these abilities to create new accounts and install new remote control software. Microsoft’s analysis of the attacker’s behavior showed that even if the SolarWinds backdoor were removed, the attacker might maintain access throughout the targeted network.

So What Did CISA say?

CISA ordered all nonmilitary governmental systems running the Orion software to both stop running the software and, critically, disconnect these computers from the rest of the network by noon today. This is simply the first step in a remediation process through which the network administrators seek to restore operations.

But the attacker’s ability to ingrain itself in the network further amplifies the problem faced by those rebuilding the networks. If the SAML (a protocol for federated authentication) or Active Directory (a tool for managing a Windows network) server is affected, there is now the significant possibility that the attacker used the initial compromise to spread throughout the entire network.

Which means that more than a few networks are going to have to take drastic measures. To quote the movie Aliens: “Take off and nuke the entire site from orbit—it’s the only way to be sure.” That is, they will need to start from scratch by reinstalling systems and then re-adding authorized users, rather than trying to ensure that all attacker accounts were removed successfully.

So Is This Why My Work Is Down?

If you work in government or in a private industry that has to worry about espionage and can’t do any work because the “network is down,” this is probably why. The systems running Orion are the most important computers for actually getting work done. After all, if they weren’t the important computers, they wouldn’t need the automated monitoring. So disconnecting all the computers that are important enough to need monitoring effectively turns off the entire enterprise.

What Now?

Christmas is now officially cancelled for three groups. The first is for the IT staff working for the perhaps 18,000 SolarWinds customers affected by the breach, who are going to have to spend the next weeks rebuilding their networks and going over everything with a fine-toothed comb looking for various backdoors. This is going to be a lot of work to sort out. The only good thing is that most of the customers don’t have secondary backdoors to worry about, because the biggest problem faced by the attacker was simply the target-rich environment. Each effort at exploitation increases the risk of discovery, and in the end, there are only so many people who can conduct these attacks.

The second group is the U.S. intelligence community. This attack started in March with the first exploitation starting in April. Either they didn’t know about it—a failure in the “defend forward” philosophy—or they did know about it, in which case they also failed to defend forward. There are going to be tough questions that the intelligence community will need to answer internally.

The final group is the Russian government. This was an amazingly valuable intelligence feed, capturing U.S. government communication leading up to the transition as well as critical insights into U.S. financial controls. Now the feed has gone dark and Russia has lost a hugely powerful asset. But then again, these are a bunch of Russian spies, so in the immortal words of every sysadmin: “fsck those guys”.