Hardly a week goes by without yet another major Windows security problem popping up, while Linux security problems, when looked at closely, usually turn out to be blunders made by incompetent system administration. But Linux can’t rest on its laurels. There are real Linux security concerns that need addressing. That’s where Google and the Linux Foundation come in with a new plan to underwrite two full-time maintainers for Linux kernel security development, Gustavo Silva and Nathan Chancellor.
Silva and Chancellor’s exclusive focus will be to maintain and improve kernel security and associated initiatives to ensure Linux’s security. There’s certainly work to be done.
As the Linux Foundation’s Open Source Security Foundation (OpenSSF) and the Laboratory for Innovation Science at Harvard (LISH) found in its open-source contributor survey, security is often neglected in open-source software development. True Linux has over 20,000 contributors, and as of August 2020, one million commits, but security is not one of their top-of-mind issues.
Unfortunately, it starts at the top. Linus Torvalds, Linux’s creator, really dislikes people who make improving security in Linux more trouble than it needs to be. In 2017, in his own inestimable style, he called some security developers “f-cking morons.” But Torvalds, while often colorful, also gave direction to security programmers.
From Torvalds’ viewpoint, “Security problems are just bugs. … The only process I’m interested in is the _development_ process, where we find bugs and fix them.” Or, as Torvalds said in 2008, “To me, security is important. But it’s no less important than everything *else* that is also important!”
Torvalds isn’t the only one who sees it that way. Jason A. Donenfeld, creator of Linux’s Wireguard Virtual Private Network (VPN), said on the Linux Kernel Mailing List (LKML) that “some security people scoff at other security people’s obsession with ‘security bugs.'”
He added: “The security industry is largely obsessed by finding (and selling / using /patching /reporting /showcasing /stockpiling /detecting / stealing) these ‘dangerous/useful’ variety of bugs. And this obsession is continually fulfilled because bugs keep happening — which is just the nature of software development — and so this ‘security bug’ infatuation continues.”
While Torvalds and Donenfeld recognize the importance of securing Linux, too many developers hear their disdain for security researchers while missing that they both regard fixing real security bugs as necessary work. The result? On average, open-source programmers use just 2.27% of their total contribution time on security. Worst still, most open-source developers feel little desire to spend more of their time and effort on security.
As David A. Wheeler, The Linux Foundation’s director of open-source supply chain security, said in the Report on the 2020 FOSS Contributor Survey: “It is clear from the 2020 findings that we need to take steps to improve security without overburdening contributors.”
The solution, the report authors suggested, was to devote money and resources to specific security purposes. This includes adding security-related tools to the continuous integration (CI) pipeline, security audits, and computing resources. In other words, make it easier for developers to add security to their projects.
Specifically, OpenSSF and LISH suggested:
- Funding security audits of critical open-source projects and require that the audits produce specific, mergeable changes.
- Rewrite portions or entire components of FOSS projects prone to vulnerabilities to produce a substantially more secure result (e.g., contribute a rewrite in a memory-safe language).
- Prioritize secure software development best practices.
- Companies should make secure software development training a requirement for hiring or continued professional development for their paid FOSS developers.
- Use badging programs, mentoring programs, and the influence of respected FOSS contributors to encourage projects and their contributors to develop and maintain secure software development practices.
- Encourage projects to incorporate security tools and automated tests as part of their continuous integration (CI) pipeline; ideally as part of their default code management platform.
By Google providing funds to underwrite two full-time Linux security maintainers signals the importance of security in the ongoing sustainability of open-source software. “At Google, security is always top of mind and we understand the critical role it plays to the sustainability of open-source software,” said Dan Lorenc, Google staff software engineer, in a statement. “We’re honored to support the efforts of both Gustavo Silva and Nathan Chancellor as they work to enhance the security of the Linux kernel.”
Chancellor’s work will be focused on triaging and fixing all bugs found with Clang/LLVM compilers while working on establishing CI systems to support Clang and LLVM compiler tools. Two years ago, Chancellor started contributing to mainline Linux under the ClangBuiltLinux project, which is a collaborative effort to get the Linux kernel building with Clang and LLVM.
The Linux kernel has always traditionally been compiled with GNU toolchains such as GCC and binutils. The more modern Clang and LLVM utilities enable developers to create cleaner and more secure builds. Linux distributions such as Android, ChromeOS, and OpenMandriva already use Clang-built kernels.
Chancellor has been working on the Linux kernel for four and a half years. “I hope that more and more people will start to use the LLVM compiler infrastructure project and contribute fixes to it and the kernel — it will go a long way toward improving Linux security for everyone,” said Chancellor.
Gustavo Silva’s full-time Linux security work is currently dedicated to eliminating several classes of buffer overflows by transforming all instances of zero-length and one-element arrays into flexible-array members, which is the preferred and least error-prone mechanism to declare such variable-length types. Silva is also working on fixing bugs before they hit the mainline, while proactively developing defense mechanisms that cut off whole classes of vulnerabilities. Before that, Silva led the effort to eliminate implicit switch fall-throughs in the Linux kernel Silva sent his first kernel patch in 2010 and is an active member of the Kernel Self Protection Project (KSPP). He is consistently one of the top five most active kernel developers since 2017 with more than 2,000 mainline commits. Silva’s work has impacted 27 different stable trees, going all the way down to Linux v3.16.
“We are working towards building a high-quality kernel that is reliable, robust, and more resistant to attack every time,” said Silva. “Through these efforts, we hope people, maintainers, in particular, will recognize the importance of adopting changes that will make their code less prone to common errors.”
“Ensuring the security of the Linux kernel is extremely important as it’s a critical part of modern computing and infrastructure. It requires us all to assist in any way we can to ensure that it is sustainably secure,” added Wheeler. “We extend a special thanks to Google for underwriting Gustavo and Nathan’s Linux kernel security development work along with a thank you to all the maintainers, developers, and organizations who have made the Linux kernel a collaborative global success.”
Google has recently been putting more resources behind security for all open-source software. The company recently proposed a framework, “Know, Prevent, Fix,” for how we can think about open-source vulnerabilities and concrete areas to address first, including:
- Consensus on metadata and identity standards: We need consensus on fundamentals to tackle these complex problems as an industry. Agreements on metadata details and identities will enable automation, reduce the effort required to update software, and minimize the impact of vulnerabilities.
- Increased transparency and review for critical software: For software that is critical to security, we need to agree on development processes that ensure sufficient review, avoid unilateral changes, and transparently lead to well-defined, verifiable official versions.
Going back to Linux in specific, funding Linux kernel security and development is a collaborative effort that needs support from everyone. To support work like this, discussions are taking place in the Securing Critical Projects Working Group inside the OpenSSF. If you want to be involved in the work, now’s your chance. It’s not just Google and top Linux developers, everyone who works with Linux needs to be involved.
