A few years ago Google completed its switch from an Ubuntu-based Linux desktop to Debian. Now Google has detailed how this change has led to rolling releases for Linux desktops with faster and smoother upgrades as well as faster security patching.
After over 15 years with Ubuntu as the base for Google’s internal devices, the company switched to Debian to avoid major OS upgrades every two years and spread the upgrade workload out over time.
Google announced it had completed the move in 2018 as ZDNet reported at the time. Margarita Manterola, a Google engineer, explained it was moving from Goobuntu, the Google build of Ubuntu, to gLinux, a rolling release based on Debian Testing, which is the beta for the next stable version of Debian.
SEE: The open source jobs are out there
It was a big change considering the number of devices involved in the migration off Goobuntu, but also not surprising because Ubuntu is based on Debian, making some aspects of the package-updating process similar.
Manterola and fellow Google engineers, Kordian Bruck and Sven Mueller, have now explained how the company arrived at rolling Linux releases for desktops in a blogpost that details how other large firms could implement the same upgrade system for desktops.
As they note, prior to the switch to rolling releases, every OS cycle created a “rather large version jump in major packages that could require significant changes to software configuration”. Google automated most but not all of the upgrade process.
Upgrading the Goobuntu fleet took the better part of a year and the timing of OS cycles meant the big-bang process was never-ending for core teams who were “close to burn out” after an upgrade.
“With a two-year support window there was only one year left until we had to go through the same process all over again for the next LTS. This entire process was a huge stress factor for our team, as we got hundreds of bugs with requests for help for corner cases,” they write.
“Once one upgrade was done there was a general sense of being “close to burnout” in the team that we barely could recover from until the next round of updates came about. Running off an LTS version also meant that some bugs encountered by users of our distribution might’ve already been fixed upstream, but those improvements might’ve never been backported to the LTS version.”
Google designed gLinux Rodete (Rolling Debian Testing) with the aim of killing the two-year upgrade cycle and rolling it out over time to reduce the load on engineers.
As they note, the software industry’s general move to CI/CD (continuous integration/continuous development) has shown that smaller incremental changes are easier to control and roll back. For similar reasons, Linux distributions like Arch Linux and NixOS have implemented rolling releases too. A rolling-release Linux is constantly being updated with the idea that users and developers are best served by giving them the latest updates and patches as they’re created.
Google chose Debian because of the availability of packages, the large community, and existing internal packages and tooling in the Debian format. Google explains why it opted for the beta of Debian rather than Stable.
“While the Debian Stable track follows a roughly two-year jump between releases, the Debian testing track works as a rolling release, as it’s the pool of all packages ingested and built from upstream, waiting for the next stable release to happen,” the Google engineers write.
Google eventually settled on weekly releases for OS updates but originally intended for them to be more frequent releases.
When it starts a new release today, the update team takes a snapshot of packages ingested from Debian at that time. Google then runs, accepts, tests and then “cautiously” rolls out the update to a dedicated testing and a 1% fleet-wide ‘canary’. The canary gives it a few days to detect problems with Debian packages or Google internal packages before rolling it out to the entire fleet.
Google also builds a workflow system called Sieve to manage the building of upstream packages from source. Sieve has tools to retry builds if the build process and testing fails.
One of the security benefits Google has gained is a reduced “trust envelope” it places in upstream Debian.
“During a security incident for example, we are able to rebuild quickly and have confidence in the build working with a temporary patch, as we have been building all packages before, that land in our distribution.
“Additionally, we also reduce the trust envelope that we have to place into upstream Debian and the binary build artifacts produced by their infrastructure. Instead once the source code is ingested and the binary built verifiably, we can cryptographically attest that the running binary originated from exactly that source code.”
Google says it has also “dramatically improved our security stance by operating our fleet closer to upstream releases.”
“While Debian provides a good source of security patches for the stable and oldstable tracks, we realized that not every security hole that gets patches, necessarily has a Debian Security Advisory (DSA) or CVE number,” the engineers note.
“Our rolling release schedule makes sure we patch security holes on the entire fleet quickly without compromising on stability, while previously security engineers had to carefully review each DSA and make sure the fix has made it to our fleet.”
SEE: How to enable Linux on your Chromebook (and why you should)
Google says it does plan to work more closely with upstream Debian and contribute more of its internal patches to maintain the Debian package ecosystem.
The company also urges others thinking of implementing rolling releases to “balance the needs of the company against upgrade agility”. What the change has confirmed for Google is that incremental changes beat big bang releases.
“Being in control of our own moving target and baseline has helped to slow down whenever we encountered too many problems and broke any of our team [Service Level Objectives]. Our journey has ultimately reinforced our belief that incremental changes are better manageable than big bang releases.”
