Steven Adair wasn’t too rattled at first.
It was late 2019, and Adair, the president of the security firm Volexity, was investigating a digital security breach at an American think tank. The intrusion was nothing special. Adair figured he and his team would rout the attackers quickly and be done with the case—until they noticed something strange. A second group of hackers was active in the think tank’s network. They were going after email, making copies and sending them to an outside server. These intruders were much more skilled, and they were returning to the network several times a week to siphon correspondence from specific executives, policy wonks, and IT staff.
Adair and his colleagues dubbed the second gang of thieves “Dark Halo” and booted them from the network. But soon they were back. As it turned out, the hackers had planted a backdoor on the network three years earlier—malicious code that opened a secret portal, allowing them to enter or communicate with infected machines. Now, for the first time, they were using it. “We shut down one door, and they quickly went to the other,” Adair says.
His team spent a week kicking the attackers out again and getting rid of the backdoor. But in late June 2020, the hackers somehow returned. And they were back to grabbing email from the same accounts. The investigators spent days trying to figure out how they had slipped back in. Volexity zeroed in on one of the think tank’s servers—a machine running a piece of software that helped the organization’s system admins manage their computer network. That software was made by a company that was well known to IT teams around the world, but likely to draw blank stares from pretty much everyone else—an Austin, Texas, firm called SolarWinds.
Adair and his team figured the hackers must have embedded another backdoor on the victim’s server. But after considerable sleuthing, they couldn’t find one. So they kicked the intruders out again and, to be safe, disconnected the server from the internet. Adair hoped that was the end of it. But the incident nagged at him. For days he woke up around 2 am with a sinking feeling that the team had missed something huge.
They had. And they weren’t the only ones. Around the time Adair’s team was kicking Dark Halo out of the think tank’s network, the US Department of Justice was also wrestling with an intrusion—one involving a server running a trial version of the same SolarWinds software. According to sources with knowledge of the incident, the DOJ discovered suspicious traffic passing from the server to the internet in late May, so they asked one of the foremost security and digital forensics firms in the world—Mandiant—to help them investigate. They also engaged Microsoft, though it’s not clear why. (A Justice Department spokesperson confirmed that this incident and investigation took place but declined to say whether Mandiant and Microsoft were involved. Neither company chose to comment on the investigation.)
According to the sources familiar with the incident, investigators suspected the hackers had breached the Justice Department server directly, possibly by exploiting a vulnerability in the SolarWinds software. The Justice Department team contacted the company, even referencing a specific file that they believed might be related to the issue, according to the sources, but SolarWinds’ engineers were unable to find a vulnerability in their code. After weeks of back and forth the mystery was still unresolved, and the communication between investigators and SolarWinds stopped. (SolarWinds declined to comment on this episode.) The department, of course, had no idea about Volexity’s uncannily similar hack.
As summer turned to fall, behind closed doors, suspicions began to grow among people across government and the security industry that something major was afoot. But the government, which had spent years trying to improve its communication with outside security experts, suddenly wasn’t talking. Over the next few months, “people who normally were very chatty were hush-hush,” a former government worker says. There was a rising fear among select individuals that a devastating cyber operation was unfolding, he says, and no one had a handle on it.
In fact, the Justice Department and Volexity had stumbled onto one of the most sophisticated cyberespionage campaigns of the decade. The perpetrators had indeed hacked SolarWinds’ software. Using techniques that investigators had never seen before, the hackers gained access to thousands of the company’s customers. Among the infected were at least eight other federal agencies, including the US Department of Defense, Department of Homeland Security, and the Treasury Department, as well as top tech and security firms, including Intel, Cisco, and Palo Alto Networks—though none of them knew it yet. Even Microsoft and Mandiant were on the victims list.