Red Hat historically has had the best record of all the Linux companies in finding and fixing Linux and open-source security bugs. Here’s how the Raleigh, NC-based company does it.
First, Red Hat Product Security is in charge of both finding and fixing security holes. It doesn’t do this alone. The team works with other Linux and open-source companies and developers. Security in the Linux world isn’t done in secret, but with the full cooperation of all involved programmers.
The security team is also a member of the Forum of Incident Response and Security Teams (FIRST). It uses the industry-standard Common Vulnerability Scoring System (CVSS) for each vulnerability — which is useful to describe how an attack works. All Red Hat product-related Common Vulnerabilities and Exposures (CVE)s are given a CVSSv3.x score. But Red Hat doesn’t use CVSS to prioritize vulnerabilities. Instead, it uses a four-point scale to describe the severity of a particular bug.
This scale is meant to help users determine the risk each bug poses to them. Red Hat’s four-point scale rates vulnerabilities are: Low, Moderate, Important, or Critical. Critical vulnerabilities pose the most severe risk to an organization. A Critical vulnerability could be exploited remotely over a network or the internet and could be automated (by a worm, for example). Red Hat also includes flaws that affect web browsers or browser plug-ins that users might be susceptible to if they visit malicious or compromised websites. Other open-source companies use similar scales.
Christopher Robinson, the Red Hat Product Security Assurance Team Lead, explained:
“CVSS does NOT equal Risk (CVSS!=Risk). Anyone who thinks otherwise is mistaken and setting themselves up for more work, pain, and stress than they realistically should have to go through. A risk is a potential for loss or damage if a threat exploits a vulnerability (which is a weakness in hardware or software).”
Robinson went on:
“CVSS is a great, yet imperfect tool. It does not account for all possible things … Even when CVSSv4 is released, it will still leave room for interpretation between an overall security risk and the specific risk for a specific implementation of software in a specific environment.”
What Red Hat’s scale added to this is:
A severity rating for the incident. This is more precise guidance from that organization about how they feel the vulnerability affects their particular implementation of that hardware or software. This gives us additional flexibility to escalate the visibility of a given issue as the need arises. It also allows us to provide additional context about features/protections/processes we use that might help alter the severity of the problem.
Robinson also wrote in a recent blog:
“We provide context to the CVEs that we felt could have the most impact from a real-world risk perspective, as well provide a filter to the fear, uncertainty and doubt that is sometimes shown around high profile vulnerabilities, which in turn introduces additional challenges to your operations.”
In the just-released Red Hat Product Security Report 2019, Red Hat said it’s seeing more customers than ever trying to grapple with ever-mounting security issues by using third-party scanners. But, he said, “While scanning tools can provide a useful ‘single pane of glass’ view of vulnerabilities across an enterprise-wide environment, they generally do a poor job of articulating risks specific to a technology or implementation.”
So, Red Hat Engineering and Red Hat Product Security both explain exactly what’s what with security issues and making Red Hat’s “upstream packages enterprise-ready by regression testing, hardening, and tweaking the package to meet our customers’ unique business demands and our release standards.”
To help improve this process, Red Hat made a fairly sizable change to Red Hat Enterprise Linux (RHEL) support life cycle. Because “RHEL is the foundation of all of our products and services, we felt it was important to expand the scope of what we supported.”
So, RHEL now includes patches and fixes for Important-rated issues, which typically cover the largest share of issues. Previously, Red Hat was more selective about which Important-rated issues were addressed in RHEL’s Extended Update Support.
Overall, in 2019, Red Hat reports:
- 2,714 security issues were reported to Red Hat Product Security (slightly down from 2018).
- 1,313 CVEs were addressed throughout 2019, a 3.2% increase from 2018.
- 968 Red Hat Security Advisories (RHSA) were issued, a record increase over previous years.
- 40 Critical advisories addressing 27 Critical vulnerabilities.
- 41% of Critical issues were addressed within 1 business day.
- 85% of Critical issues were addressed within 1 week.
Robinson added:
“Where the rubber meets the road, so to speak, are the RHSAs that are issued and ultimately applied to a system, be it physical, virtual, or container. We released 968 RHSAs, which is a three-fold increase from 2011. This speaks to the diversity and complexity of the portfolio of 2019. System administrators have their work cut out for them, no matter what types of systems they manage, which is part of the reason one might note the difference in number of CVEs fixed versus the number of issued security advisories. As time, content, and opportunity allows, we’ll bundle multiple CVE updates into a single RHSA (but then issue separate RHSAs based off of product and version) to try and help ease that administrative burden of getting downtime windows for maintenance. Obviously, as an awesome enterprise vendor, we do have solutions that can help with that (cough, cough, Insights, cough cough, Ansible, cough Satellite… just sayin’).”
Maintaining security across Red Hat’s software ecosystem — not to mention how it spans standalone servers, containers, and the cloud — is a big job. Fortunately, not just for Red Hat’s software, but for the entire Linux and open-source world, Red Hat is up to the job.
