One of the most irritating things about the SolarWinds attack was that the Russian crack went unnoticed from March to December 2020. During that time, the Russian government’s SolarWinds hack was opening the door to the secrets of numerous top American government agencies and tech companies. Even now, we’re still trying to get our minds around just how widespread and bad the SolarWinds cracks were.
The root causes of this crack were a dangerous set of software supply-chain failures. It’s too late for anything but damage control for SolarWinds, but The Linux Foundation has found several lessons to make sure your programs, whether open source or proprietary, avoid SolarWinds-style disasters.
Also: Best VPNs • Best security keys
David A. Wheeler, the Linux Foundation’s Director of Open Source Supply Chain Security, explained that in the Orion attack that the malicious code was inserted into Orion by subverting the program’s build environment. This is the process in which a program is compiled from source code to the binary executable program deployed by end-users. In this case, the security company CrowdStrike worked out that the Sunspot malware watched the build server for build commands and silently replaced some of Orion’s source code files with malware.
By entering the program before it’s even properly a program, this hack makes most conventional security advice useless. For example,
“Only install signed versions” doesn’t help because this software was signed.
“Update your software to the latest version” doesn’t help because the updated software was the subverted one.
“Monitor software behavior” eventually detected the problem, but the attack was quite stealthy and was only detected after tremendous damage was done.
“Review source code” is not a certain defense either. In Orion’s case, it’s not even certain that developers could have spotted the source code changes. The changes were carefully written to look like the expected code. In addition, since the attackers had control of the build environment, they could have inserted the attack without it being visible to software developers.
Finally, since Orion isn’t open-source software, no one could independently audit the code. Only the company’s developers could review it or its build system and configurations. As Wheeler said, “If we needed further evidence that obscurity of software source code doesn’t automatically provide security, this is it.”
So, what can you do? Well using open-source is a good start. There’s nothing magical about open-source methodology. Security mistakes can still enter the code, but at least you have the possibility of more eyes looking for problems before they blow up.
In addition, Wheeler pointed out, companies must harden their build environments against attackers. For example, SolarWinds used extremely poor developer security practices. This included using the insecure ftp protocol for file transfers and publicly revealing passwords. Any build environment still using these is just asking for a security breach.
Also: Best Linux Foundation classes
Build systems are critical production systems, and they should be treated like such. If anything, they should receive a higher level of security requirements than production environments. This is code security 101.
Even when you’ve secured your build system to the best of your abilities it’s not a sure thing that it’s safe. In the long run, Wheeler thinks there’s only one true strong countermeasure for this kind of attack: Verified reproducible builds.
“A reproducible build,” wrote Wheeler, is one “that always produces the same outputs given the same inputs so that the build results can be verified. A verified reproducible build is a process where independent organizations produce a build from source code and verify that the built results come from the claimed source code.”
That’s more of a good idea than something you can do today. Very few programs today can be verified. The Linux Foundation and Civil Infrastructure Platform has been funding work, including the Reproducible Builds project, to make verified reproducible builds.
The Linux Foundation wants everyone to start implementing and requiring verified reproducible builds. Yes, this won’t be easy. Most software is not designed to be reproducible in their build environments. It may well take years to make software reproducible.
Wheeler wrote:
Many changes must be made to make software reproducible, so resources (time and money) are often needed. And there’s a lot of software that needs to be reproducible, including operating system packages and library level packages. There are package distribution systems that would need to be reviewed and likely modified. I would expect some of the most critical software to become reproducible first, and then less critical software would increase over time as pressure increases to make more software verified reproducible. It would be wise to develop widely-applicable standards and best practices for creating reproducible builds. Once software is reproducible, others will need to verify the build results for the given source code to counter these kinds of attacks.
Reproducible builds are much easier for open-source software (OSS) because there’s no legal impediment to having many verifiers. Closed source software developers will have added challenges; their business models often depend on hiding source code. It’s still possible to have “trusted rebuilders” worldwide to verify closed source software, even though it’s more challenging and the number of rebuilders would necessarily be smaller.
The information technology industry is generally moving away from “black boxes” that cannot be inspected and verified and towards components that can be reviewed. So this is part of a general industry trend; it’s a trend that needs to be accelerated.
Can’t happen? Why not? Auditors have access to the financial data and review the financial systems of most enterprises. Software companies need code and build auditors. Otherwise, we will certainly see more software build attacks spreading malware.
Too expensive? Think again. SolarWinds is already being hit by its first class-action lawsuit. More will follow. The company’s stock has also seen a 40% drop since news of its failure broke.
Wheeler also reminded us that “Attackers will always take the easiest path, so we can’t ignore other attacks.”
Specifically, since most attacks exploit unintentional vulnerabilities in code, we must prevent these unintentional vulnerabilities. These mitigations include changing tools and interfaces to avoid problems; detecting residual vulnerabilities before deployment; and educating developers on developing secure software. For example, there are free edX courses from Open Source Security Foundation (OpenSSF) on how to develop secure software.
Wheeler also noted that since “Applications are mostly reused software (with a small amount of custom code), so this reused software’s software supply chain is critical. Reused components are often extremely out-of-date. Thus, they have many publicly-known unintentional vulnerabilities. In fact, reused components with known vulnerabilities are among the topmost common problems in web applications. The LF’s LFX security tools, GitHub’s Dependabot, GitLab’s dependency analyzers, and many other tools & services can help detect reused components with known vulnerabilities.”
Again, simply using OSS doesn’t mean it’s safe. Wheeler added, “Vulnerabilities in widely-reused OSS can cause widespread problems, so the LF is working to identify such OSS so that it can be reviewed and hardened further.”
Even if you’re using proprietary programs, the code behind it may be open source based. Synopsys has found that 99% of commercial software programs include at least one open-source component. That’s fine, but 91% of those included out of date or abandoned open-source code. That’s not good at all.
Malicious code can also hide in the supply chain. For instance, Wheeler stated, “most malicious code gets into applications through library ‘typosquatting.’ That is, by creating a malicious library with a name that looks like a legitimate library.”
For users, that means they should start asking for a software bill of materials (SBOM) so they will know exactly what it is they are using. Yes, that’s yet another argument for open source. Orion, like all proprietary software, is a black box. No one except its builders know what’s in it. And with Orion, it appears they never knew either until it blew up in their users’ faces.
The US National Telecommunications and Information Administration (NTIA) has been encouraging SBOM adoption. The Linux Foundation’s Software Package Data Exchange (SPDX) format is an SBOM format that’s being widely adopted.
Armed with SBOM information, you can examine what component versions are used in your program. Of course, this requires you to pay attention to what’s in your programs. For example, Equifax’s infamous failure was due to its simply not paying attention when a public fix was issued for the Apache Struts library it used in its programs. Equifax tried to pin the blame on Apache, but when the dust settled, Equifax admitted it was entirely at fault.
When a program uses components with known vulnerabilities, that’s a big red flag, True, some vulnerabilities may not be exploitable, but, Wheeler states, far “too many application developers simply don’t update dependencies even when they are exploitable. To be fair, there’s a chicken-and-egg problem here: specifications are in the process of being updated, tools are in development, and many software producers aren’t ready to provide SBOMs. So users should not expect that most software producers will have SBOMs ready today. However, they do need to create a demand for SBOMs.”
Wheeler also wants to see software distributors embracing SBOM information. “For many OSS projects this can typically be done, at least in part, by providing package management information that identifies their direct and indirect dependencies (e.g., in package.json, requirements.txt, Gemfile, Gemfile.lock, and similar files). Many tools can combine this information to create more complete SBOM information for larger systems.”
Finally, Wheeler believes, “Organizations should invest in OpenChain conformance and require their suppliers to implement a process designed to improve trust in a supply chain.” OpenChain is an open-source blockchain secured distributed ledger technology for tracking open-source program components and licenses. It’s also an ISO/IEC standard: 5230:2020.
With all this in mind, you should put all the below on your software development and deployment list:
Harden software build environments
Move towards verified reproducible builds
Change tools & interfaces so unintentional vulnerabilities are less likely
Educate developers
Use vulnerability detection tools when developing software
Use tools to detect known-vulnerable components when developing software
Improve widely-used OSS
Ask for SBOMs in SPDX format. Many software producers aren’t ready to provide one yet, but creating the demand will speed progress
Determine if subcomponents used have known vulnerabilities
Work towards providing SBOM information if we produce software for others
Implement OpenChain
If you don’t, as Wheeler reminds us, “Those who do not learn from history are often doomed to repeat it.” Do you want your company to be the next SolarWinds? I don’t think so!
Related Stories:
