It was one of the largest cyber-espionage attacks of recent times: hackers compromised several United States government federal agencies as well as big tech companies, and were inside networks for months before anyone spotted them.
These attackers were later revealed to be working for the Russian foreign intelligence service (SVR), and they started their attack in an unexpected way, by targeting a software company called SolarWinds. The hackers accessed builds of the company’s Orion software, and then placed malware into software updates sent out to SolarWinds customers between March and June 2020.
The software is used by thousands of organisations around the world. Applying security updates and patches is generally regarded as good cybersecurity practice to protect against software vulnerabilities being exploited to facilitate cyberattacks, so organisations around the world installed the Orion updates from a source they trusted. But it was that action itself that allowed the attackers in.
“It became clear early on the threat actor employed novel and sophisticated techniques indicative of a nation-state actor and consistent with the goal of cyber espionage via a supply chain attack. In addition, the operational security of the threat actor was so advanced, they not only attacked SolarWinds but were able to leverage the Sunburst malicious code and avoid detection in some of the most complex environments in the world,” SolarWinds said in its investigation after the attack.
SEE: A winning strategy for cybersecurity (ZDNet special report)
Among those compromised by the supply chain attack were the US Treasury Department, the Department of Homeland Security, the US Department of State, as well as cybersecurity companies including Microsoft, FireEye and Mimecast. In total, somewhere around 100 companies were targeted by the attackers.
Attackers had been active in the network for months before the attack was discovered in December 2020, when FireEye and Microsoft found intrusions into their networks.
The attack on SolarWinds was disclosed just weeks before Sudhakar Ramakrishna was set to take up his new position as CEO of the company in January 2021.
Due to the magnitude of the situation, he chose to get involved with the company’s attempt to investigate and resolve the incident right away.
“It was a stressful time for all involved,” he told ZDNet. “When the business is in a state of turmoil and crisis, there isn’t time to sit on the sidelines. The decision to jump in and start working with the team was simple.”
The first thing that had to be done was to examine what exactly had happened, how it had remained undetected for so long, and how to ensure it can never happen again.
Part of that involved bringing in the services of Krebs Stamos Group – a cybersecurity consultancy set up by former US government cybersecurity chief Chris Krebs, and Stanford University professor and ex-Facebook chief security officer Alex Stamos. The UK’s National Cyber Security Centre (NCSC) was also involved in helping SolarWinds in the aftermath of the incident.
But one policy Ramakrishna wanted to introduce from day one was the concept of ‘Secure by Design’ – building products with security more than anything else in mind. Many organisations and software developers say they take security seriously, but when there’s deadlines to meet or products to repeatedly roll out updates for, software security can often get left on the sidelines.
“The notion of secure by design, I had it in my mind and in practice at some level well before I joined SolarWinds,” Ramakrishna explains. “Between the time I came to know about the breach and the time I joined, I started formulating my thoughts in terms of how do we organise around secure by design, what does that mean and what are the various elements of that? Then essentially went about business on day one in terms of implementing that as a process.”
Much of this secure by design philosophy applies directly to the software build system, with the process now designed around cybersecurity as the priority.
One of the reasons that cyber attackers were able to conduct the supply chain attack was because of the static nature of the software-building process, where everything is done within one pipeline of development. While that’s useful for developers, it also provides a handy target for the attackers.
Now, SolarWinds uses a system of parallel builds, where the location keeps changing, even after the project has been completed and shipped. Much of this access is only provided on a need-to-know basis. That means if an attacker was ever able to breach the network, there’s a smaller window to poison the code with a malicious build.
“What we’re really trying to achieve from a security standpoint is to reduce the threat window, providing the least amount of time possible for a threat actor to inject malware into our code,” said Ramakrishna.
But changing the process of how code is developed, updated and shipped isn’t going to help prevent cyberattacks alone, which is why SolarWinds is now investing heavily in many other areas of cybersecurity.
These areas include the likes of user training and actively looking for potential vulnerabilities in networks. Part of this involved building up a red team, cybersecurity personnel who have the job of testing network defences and finding potential flaws or holes that could be abused by attackers – crucially before the attackers find them.
Importantly, the rest of the company doesn’t know what tactics and techniques are going to be used in tests against the network and staff – because cyber criminals and hackers don’t declare exactly how they’re going to conduct campaigns, either.
“They are paid to attack our internal systems, our behaviors and our internal practices. That improves the overall security consciousness of the company and that improves the overall security posture of the company,” Ramakrishna explained.
Analysis is performed to examine which techniques and vulnerabilities are successfully used to launch attacks – but crucially, nobody is made an example of. All of the information gathered from red teaming is put back into teaching everyone how to identify cyberattacks, phishing emails and other malicious activity to help drive good cybersecurity hygiene.
SEE: How do we stop cyber weapons from getting out of control?
But Ramakrishna and SolarWinds know that implementing new cybersecurity procedures isn’t just a one-time initiative, it’s something that needs to be repeatedly revisited as threats change, new vulnerabilities emerge, and offensive hacking techniques evolve.
“Increasingly, this will simply become part of the fabric of the company and we won’t have to talk about it in explicit terms as much as just believing in it and working on it on a daily basis,” he says, as SolarWinds works to ensure that something like the supply chain attack can’t happen again by making the network more robust and taking a more proactive approach to detecting potential malicious activity.
The company also hopes to take the lessons it has learned and help its worldwide customer base improve their cybersecurity.
“We are evolving and helping them digitally transform much faster into the future,” said Ramakrishna. “My hope also is that things like the build system that we have created will become more and more standards in the industry that others can leverage as well”.
By sharing what happened, SolarWinds hopes that other organisations can also learn lessons and improve their own cybersecurity strategies, because anyone can potentially be the victim of a cyberattack, particularly if those behind it have vast resources, such as the state-backed operation that breached SolarWinds.
“No one is immune, so you cannot think that it will not happen to you. It could happen to you, so just be vigilant about things and constantly learn,” said Ramakrishna.
“Don’t try to fight it alone or don’t wish the problem goes away because the problem is not going to go away,” he added.
SEE: Clueless hackers spent months inside a network and nobody noticed. But then a ransomware gang turned up
SolarWinds is implementing secure by design in its software build process and recommends that all organisations ensure they have cybersecurity frameworks in place to help manage security at every step of the way when conducting business, no matter what that may be.
Most victims of cyberattacks don’t speak out about them, and some will never publicly acknowledge they fell victim. But for Ramakrishna, the best way of showing other businesses what threats are out there and how to protect against them is to openly talk about what happened at SolarWinds – and he hopes that others can learn about what happened to help protect their own networks.
“I believe the best and maybe the only way to be most safe and secure is by information-sharing more transparently more quickly,” he said. “If you are creating a situation where there is a lot of victim-shaming that goes on, then people do not step forward to highlight what they are learning”.
For SolarWinds, there’s also an element of maintaining trust. The company fell victim to one of the most infamous cyber incidents of recent times and Ramakrishna argued it was only right to be transparent with customers about what happened “I truly believe you owe it to them: how can you earn that without being transparent?” he says.
