SolarWinds and the Holiday Bear Campaign: A Case Study for the Classroom

Author's Note: Have you been looking for a detailed-but-accessible case study of the Russian cyberespionage campaign that targeted SolarWinds (among others)? The following piece is adapted from my newly-released eCasebook “Cybersecurity Law, Policy, and Institutions” (v.3.1), which is available free and in full (270+ pages) in pdf format here. My aim is for this excerpt to be useful especially for teachers and students who want an account that takes the technical aspects quite seriously but is written in a way that non-technical readers can digest.

Southwest Parkway is a wide and winding road that leads away from Austin towards the Texas Hill Country. Along its length are neighborhoods, schools, and long stretches of open landscape. It is not where one might expect to find the epicenter of a major cybersecurity episode. But Southwest Parkways also is where one can find the unassuming headquarters of SolarWinds, a name that burst into the headlines in December 2020.

SolarWinds specializes in network-management tools—that is, software that large enterprises use to monitor and control conditions throughout their information technology environment. Its products are in widespread use around the world, including a wide array of prominent private sector entities and government agencies. Among its most successful products is a network-monitoring system called Orion. Orion is an “on premises” platform, meaning that it does not reside in the cloud (that is, on remote servers controlled by SolarWinds itself). Rather, customers upload Orion as a software package installed on their own networks and run from there. And this has consequences for the process by which Orion periodically is updated. Like any software, Orion periodically requires updates both for security purposes and to improve performance. Thus, Orion customers periodically receive and routinely install from SolarWinds software updates, much as all of us periodically accept vendor-provided updates for the operating system on our phones. In both contexts, the provider “digitally signs” the update in a way that can be verified technically, ensuring that the update really is coming from the provider. Trusting that the provider took all necessary safety precautions, and often lacking the means to conduct an independent security check in any event, most of us accept these verified updates as a matter of course. This is true for us as individuals with our phones, and it is true to some extent for many a large organization, including those using Orion.

Therein lay an extraordinary opportunity for espionage. If a would-be spy could “trojan” an Orion update—that is, if one could find a way to embed malicious code somewhere within an otherwise legitimate update—then customers by the thousands would open their virtual gates and let that code into their networks. And given the particular function of Orion—spanning across a user’s IT infrastructure—the resulting backdoors, if employed discretely enough, might then pave the way for deployment of further malware directly into those now-compromised networks. The end result could be an intelligence bonanza.

The opportunity would have been tempting for any foreign intelligence service engaged in collection against the U.S. government. And at least one such service did spot it: Russia’s Foreign Intelligence Service, better known in English as SVR (Sluzbha Vneshney Razvedki).

SVR has a well-deserved reputation for its ability to conduct espionage through cyber means. Hackers associated with SVR sometimes are referred to as “Advanced Persistent Threat 29” (APT29), under the anodyne labeling system frequently used in the information security industry as a way to track government hackers without having to expressly attribute particular campaigns or entities to the actual government involved. Others have used the label “Cozy Bear,” following the more-entertaining naming system popularized by Dmitri Alperovich and the security firm CrowdStrike. With Crowdstrike’s nomenclature, groups believed to be linked to the Russian government are named some variation of “Bear.” A group thought to be associated with Russia’s military intelligence agency (GRU), for example, are known in this system as “Fancy Bear.” And so, when a possibly new group of hackers linked to Russia emerges, a new name may follow. And in this case, when the SolarWinds story began to break in December 2020, the initial framing offered by Dmitri Alperovich was the seasonally appropriate “Holiday Bear.”

Since that time, attribution has focused firmly and reliably on SVR, but I will still refer periodically to Holiday Bear. This will remind us that analysts wrestling with attribution amidst unfolding attacks often are drawing on forensic and contextual clues that may be specific to specific groups within larger organizations.

What follows is a detailed account of the complex sequence of operations SVR conducted as part of the Holiday Bear campaign. As we shall see, exploiting SolarWinds was a central part of the campaign, but there is far more to the story than that (indeed, the intense media focus on SolarWinds has had the unfortunate effect of deflecting attention from the shortcomings of other companies and government agencies).

Step one: accessing the SolarWinds “build environment”

It is one thing to recognize that SolarWinds customers might not detect a trojaned Orion update, but quite another to compromise the update system in the first place. The task SVR first faced, accordingly, was to sort out how it could penetrate without detection the “build environment” (aka “development environment”) used by SolarWinds engineers to draft and tinker with Orion’s code. Then SVR would need to find a way to inject malicious code into an Orion “build” without detection. These were tall orders.

SVR managed both tasks, but we do not (yet) know precisely how. There are plenty of plausible explanations, however. Perhaps it spearphished an employee’s credentials (that is: tailoring a fake message for that particular employee and thereby tricking them into opening a malware-laced file and thereby gaining access to the person’s legitimate login credentials). Perhaps it simply engaged in password “spraying” until it hit upon something that worked (famously, a security researcher in 2019 had discovered the password for an unrelated SolarWinds server stored in a GitHub repository; the password was “password123”). Perhaps SVR hacked its way into the build environment, taking advantage of vulnerabilities or configuration errors in the software used by SolarWinds engineers to perform their development work. All of these are common routes to initial exploitation, and any could have been the culprit here. At any rate, SVR managed it. No later than early September 2019 (and possibly as early as January 2019), SVR established access to the build environment.

Step two: injecting malware into an Orion update

SVR’s next step was to test-drive its access by inserting some innocuous code into the build environment, to see if this could be done without detection. In fall 2019, SVR dipped its toes into the water, inserting a modest batch of innocuous code. It worked; the addition was not detected. Exhibiting remarkable patience, SVR continued with similar experiments for months before at last taking advantage of this access to inject actual malware into an Orion build. It took that step in February 2020.

To accomplish this, SVR used malware now known to us as “SUNSPOT.” SUNSPOT was thoughtfully designed for its task. Once deployed into the build environment, it determined whether a particular development tool (Microsoft Visual Studio) was in use by the SolarWinds engineers at that time. If so, SUNSPOT then checked to see if that tool was being used for an Orion build in particular. If so, SUNSPOT monitored to determine precisely when the developers used that tool to access a particular part of the Orion source code, and at that moment SUNSPOT injected a file with the plausible-sounding name “SolarWinds.Orion.Core.BusinessLayer.dll” into the Orion build. That .dll file was, in fact, a custom malware package known to us now as “SUNBURST.”

It might help at this point to draw out the analogy to the Trojan Horse. If the eventual distribution of a corrupted Orion update was the cyber equivalent to that giant wooden horse being brought within the walls of the city of Troy, then SUNBURST was the equivalent of the small squad of Greek warriors hiding within the horse, prepared to sneak out and quietly open a side gate in those walls and thus allows their fellows to pour in undetected. From that perspective, SVR’s success in loading SUNBURST into the Orion build was equivalent to loading that squad of Greek warriors into the horse prior to presenting the horse at the city’s gates; it remained to be seen if the Trojans would take the horse within their walls or if the squad hidden within could manage to open a gate from the inside without getting caught.

In the event, it unfolded exactly as SVR hoped. SolarWinds did not detect the compromise of their latest Orion update, and hence proceeded to digitally sign it and roll it out to customers though the existing, trusted remote-update mechanism. That process began in March 2020 and continued for a few months. The horse was now within the walls for a vast array of customers. The spread might have gone further, in fact, but at that point SVR apparently concluded that it had achieved sufficient distribution. Rather than risk someone at SolarWinds detecting the compromise of its build environment and unspooling the entire campaign, SVR at that point did what it could delete its access to the build environment, covering its tracks as well as it could.

At this point, some 18,000 customers had downloaded the trojaned Orion update. Among them were several U.S. government agencies, including the Departments of Treasury, State, Commerce, Energy, and Homeland Security.

Step three: deciding where to take advantage of SUNBURST

In theory, SVR could have blindly exploited all infected systems to the maximum extent possible. But doing so would have increased the opportunities for detection, with the potential to expose and thus put an end to the entire campaign. Rather than run such risks, SVR instead chose to proceed slowly and narrowly, with an emphasis on remaining undetected while carefully identifying which of the infected customers best fit their specific collection priorities.

In every instance, SUNBURST took no action at all for approximately two weeks once installed on the customer’s premises. This put some distance between its eventual actions and the upload process. Then, when it did finally come alive, its first action was to perform a complex series of checks designed to determine whether various security products were in operation on the system. Where such products were detected, SUNBURST disabled them if possible, but otherwise simply shut itself down without taking further action. Only if and when it passed these safety checks would SUNBURST go on to its next step: communicating with an external command-and-control (C2) server. In Trojan horse terms, the warriors in the horse had waited two full weeks, and after making sure no one was watching they moved at last moved to open up a side gate.

To minimize the risk that its communications with a C2 server would not draw attention, SVR attempted to disguise this malicious traffic. This was made easier by the fact that on-premises instances of Orion routinely had to communicate with SolarWinds as part of what SolarWinds calls the “Orion Improvement Program” (much as an iPhone user might click yes when asked if it is ok for our phone to share information with Apple about how we are using Siri, in order to help Apple improve the product or better tailor it to our usage patterns). SVR accordingly designed SUNBURST’s communications with C2 servers so as to appear, on casual inspection, to be part of that legitimate traffic flow. The ruse appears to have been highly effective. Among other things, this illustrates that victims typically were not limiting Orion’s external communications in a way that would limit them to specifically pre-approved (“allow-listed”) domains, or that would at least flag for attention any communications to a non-pre-approved domain.

Critically, the initial communication did not automatically result in SVR dispatching additional malware. The first task, instead, was to enable SVR to decide whether to exploit that particular victim or, instead to cover its tracks by uninstalling SUNBURST from that location. Towards that end, SUNBURST dispatched location information that SVR in turn used to make a judgment about the identity of the infected system. In most cases, it appears, the system in question proved not to be of interest, and SVR uninstalled SUNBURST frequently. But not in all cases. SVR had hit the jackpot in plenty of instances, and now it was time to move on to the active-exploitation stage.

Step four: injecting the tools needed to act effectively within targeted systems

Notably, SUNBURST did not itself contain the tools needed to engage in lateral movement within the compromised networks, to conduct data exfiltration, and to engage in other aspects of active exploitation. It was a slim design, reflecting the premium SVR placed on detection avoidance at the initial stage. SUNBURST contained no more than was necessary to reach the point that it could open a backdoor to a C2 server. Put simply: in order to actively exploit an infected system, SVR needed to use SUNBURST to inject additional malware.

But how to do so without that fresh injection of malware being detected at the victim’s perimeter by their network’s antivirus capability? This is a classic problem that hackers have long faced. Of course, one might try to overcome it by developing sophisticated, never-seen-elsewhere custom malware that might not be detected by security software that scans for known indicators of compromise. That’s not a realistic option for many attackers, and even where it is, it remains tempting to instead make use of “retail” malware that already exists and has a long track record of reliability and effectiveness. If such commonplace tools can be injected without detection, even sophisticated entities like SVR will make use of them.

These considerations over time have led to the development of a category of malware tools known as “droppers” and “loaders.” These are programs that hide within themselves some otherwise-detectable malware payload. The idea with a dropper is that the attacker might be able to install the dropper on a targeted system without detection, at which the dropper will unpack and install the core malware payload. Loaders are similar, but rather than containing the malware payload within themselves all along, they provide a (more secure) means to download the payload surreptitiously from an external server. Both have the effect, at any rate, of reducing detection risk at the stage when a malware payload must transit a system’s perimeter, thus making it more attractive to rely on retail malware.

That’s the route SVR took with the SUNBURST-infected systems that it targeted for active exploitation. Specifically, SVR developed what is now called TEARDROP. TEARDROP was a novel dropper that SVR could load into compromised systems via SUNBURST, with little risk of detection at victims’ network perimeter. SVR also deployed a novel loader, RAINDROP, for the same purpose.

In this way, SVR successfully injected the toolkit it needed for active exploitation of its priority targets, despite the fact that the toolkit it chose to use was, very much, a well-known one: Cobalt Strike.

Cobalt Strike is a commercial product sold by the Minnesota company HelpSystems as a tool for use in attack simulations and penetration testing. That is to say, it is meant to be a pro-security product, emulating what a real attacker might do. The Cobalt Strike BEACON capability, for example, provides a range of capabilities comparable to what a sophisticated, genuine attacker might employ. All of which is quite useful from the defense-improvement perspective, when actually used for testing purposes. But real attackers also find these tools useful. As the firm Intel471 recently wrote, “criminals love” Cobalt Strike, and it has “become a very common second-stage payload for many malware campaigns….”

SVR apparently loves it too, for rather than using TEARDROP and RAINDROP to inject bespoke (SVR-made) tools at this stage in the operation (as they had done at every prior stage), they instead went with what amounted to a somewhat customized version of Cobalt Strike BEACON.

Some have suggested that this was a significant mistake on SVR’s part, since there are techniques available to detect the presence of BEACON on a network. Yet it does not appear that any of the targeted entities actually detected BEACON in use before otherwise learning of the SolarWinds campaign. SVR did take steps to hide what was occurring, after all. To execute a file, for example, it would temporarily replace a legitimate file with a malicious one, execute that file (at times doing so by temporarily modifying a legitimate scheduled task so that it would issue the command to execute the file), and then restore the original. And, in any event, SVR may have had the notion that using such common tools might actually make it more difficult for investigators to attribute the operation to it, and by extension make it less likely that the larger campaign would be undone by exposure of any one instance (relatedly, SVR ensured that each deployed instance of BEACON varied from the others in terms of file names and other otherwise-matchable details, thus reducing the chance that detection by one victim would result in the sharing of indicators-of-compromise that could be used effectively by other victims).

Step five: swimming upstream to the cloud

At this point, SVR was well-established within victims’ own local (on-premises) servers and systems. And if that was where to find the richest intelligence, the table would have been fully set for espionage success. But for a growing number of organizations—including government entities—the on-prem environment is not where one will find most of the information an intelligence collector would value. Increasingly, organizations rely on cloud services for email and documents. This is true, for example, for organizations that rely on the Microsoft 365 (“M365”) product suite, which encompasses, among other things, the Microsoft Office software suite (Outlook for email, Word for documents, PowerPoint for slides, and Excel for spreadsheets), or that use Microsoft’s Azure cloud services (just as the same would be true for an organization that instead used the cloud services of other providers, such as Amazon’s AWS cloud services).

Not surprisingly, this was true for many of the entities SVR had compromised via Orion, especially with respect to using M365. In those case, accordingly, SVR had more work to do even after getting BEACON into the on-prem environment. To get the intelligence it actually sought—to realize the purpose of the whole effort—it next needed to swim upstream from its on-premises foothold into the M365 cloud environment.

This brings us to an important and underappreciated aspect of the “SolarWinds” story: it is just as much a Microsoft story. The SolarWinds framing is understandable, of course, given the breadth of the compromises that resulted from the breach of that one company’s own security. But from SVR’s perspective, what ultimately mattered was accessing the M365 cloud environment. Orion was one (very attractive and scalable) pathway to get to the doorstep, but it was not the only such pathway. Indeed, in a security advisory released on January 8, 2021, CISA revealed that it was “investigating instances in which [SVR] may have obtained initial access” not via Orion but, rather, “by Password Guessing…, Password Spraying…, and/or exploiting inappropriately secured administrative or service credentials…instead of utilizing the compromised SolarWinds Orion products.” Soon thereafter, the firm Malwarebytes revealed publicly that it too had been “recently targeted by the same threat actor,” and that it could “confirm the existence of another intrusion vector that works by abusing applications [other than Orion] with privileged access to Microsoft Office 365 and Azure environments.”

How did SVR make the leap to the cloud from within customer systems? It appears it used multiple pathways, one of which is known as the “Golden SAML” approach. This method overcomes the critical identity authentication process that normally precludes unauthorized users from accessing cloud-based accounts. More specifically, it is a method that works for systems that use “Active Directory Federation Services,” or ADFS, to connect users with cloud-based services like M365. The basic idea with ADFS is that when a user goes to log in to the cloud service, that service directs the request to a particular server (the ADFS server) to validate the user’s identity. If the ADFS server approves the credentials, it issues a token to the cloud service that confirms the user’s legitimacy and enabling access. For obvious reasons, then, the security of the ADFS server itself is critical. In particular, the private encryption key and signing certificate for that server are critical. Which brings us to the heart of the Golden SAML approach: if the attacker gains access to the private key, this in turn leads to the ability to issue tokens, and that then opens the door to accessing the accounts associated with that ADFS server.

That was one way that SVR began strolling through the 365 accounts of its victims. One of the other methods later identified by FireEye was similar: the creation within Azure Active Directory (Azure AD) of entirely new domains approved to function as a trusted third-party “identity provider”—that is, as a provider of authentication services such as those described above.

Whichever particular pathway was taken, however, the fundamental approach was the same: leveraging the client’s own credentials so as to circumvent cloud identity-authentication safeguards from the inside. At this point, it was simply a question of whether anyone would notice unusual patterns of account access and other account activity. For a long time, no one did.

Step six: don’t get caught

What changed? SVR’s luck ran out. It was extracting material from a variety of private victims, not just government agencies. One of these, as it happened, was the cybersecurity company FireEye, and at some point FireEye detected what was happening.

On December 8, 2020, Kevin Mandia published the jaw-dropping news that someone had penetrated FireEye’s systems and gotten away with their pen-testing tools. The sophistication of the attack suggested it was the work of a nation-state actor, in Mandia’s opinion, as did the fact that the attacker also apparently attempted to learn things about FireEye’s government customers. Five days later, Mandia announced as a follow-up that FireEye had “identified a global campaign that introduces a compromise into the networks of public and private organizations through the software supply chain.” In particular, Mandia explained, the campaign exploited the update mechanism for SolarWinds Orion.” It was a stunning turn of events, with holiday-wrecking implications for thousands-upon-thousands of SolarWinds customers.

Given the scale and potential impact of the operation, it was clear early on that the episode constituted a “significant cyber incident”—that is, an incident of sufficiently serious nature to warrant formal federal government interagency coordination efforts—under the U.S. government’s National Cyber Response Plan. In practical terms, this meant that the federal government would form an interagency “Cyber Unified Coordination Group” (UCG) as the focal point for cooperation and deconfliction among FBI (as the lead agency for what the government calls “threat response”), CISA (as the lead agency for “asset response”), and ODNI with NSA support (as the lead agency for “intelligence support and related activities”). By January 5, the UCG was ready to go on the record with a (very) limited degree of public attribution, declaring their collective belief that the attacker was “likely Russian in origin.” This was a far cry from blaming the Russian government directly, but it was a beginning. Meanwhile, private-sector experts were more explicit about the attribution. Dmitri Alperovitch and David Cross used the label “Holiday Bear” in an interview on Patrick Gray’s Risky Business show (a weekly must-listen for everyone interested in cybersecurity) as a way to capture their view that a Russian government actor was responsible, even it if was not yet clear which precise actor it was.

Eventually, the US government would state in no uncertain terms that it was the SVR, specifically, that conducted the operation: “Today the United States if formally naming the Russian Foreign Intelligence Service (SVR), also known as APT 29, Cozy Bear, and The Dukes, as the perpetrator of the broad-scope cyber-espionage campaign that exploited the SolarWinds Orion platform and other information technology infrastructures. The U.S. Intelligence Community has high confidence in its assessment of attribution to the SVR.”