Web browser closeup on LCD screen with shallow focus with light shining through https padlock. Internet security, SSL certificate, cybersecurity, search engine and web browser concepts
Getty Images/iStockphoto
The Let’s Encrypt project will revoke more than 3 million TLS certificates on Wednesday, March 4, 2020, due to a bug it discovered in its backend’s code.
More specifically, the bug impacted Boulder, the server software the Let’s Encrypt project uses to verify users and their domains before issuing a TLS certificate.
Bug impacted CAA checks
The bug impacted the implementation of the CAA (Certificate Authority Authorization) specification inside Boulder.
CAA is a security standard that was approved in 2017 and which allows domain owners to prevent Certificate Authorities (CAs; organizations that issue TLS certificates) to issue certificates for their domains.
Domain owners can add a “CAA field” to their domain’s DNS records, and only the CA listed in the CAA field can issue a TLS certificate for that domain.
All Certificate Authorities — like Let’s Encrypt — must follow the CAA specification by the letter of the law or face steep penalties from browser makers.
In a forum post on Saturday, February 29, the Let’s Encrypt project disclosed that a bug in Boulder ignored CAA checks. The Let’s Encrypt team explains:
“The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.”
The Let’s Encrypt team patched the bug on Saturday during a two-hour maintenance window, and Boulder is now verifying CAA fields properly before issuing new certificates.
It is very unlikely that someone exploited this bug, the project said.
Nonetheless, today, the Let’s Encrypt project announced it was revoking all the certificates that have been issued without proper CAA checks, following industry rules, as dictated by the CA/B Forum.
Only 3 million of 160 million active certs were impacted
Let’s Encrypt engineers said that of the 116 million TLS certificates that are currently active, only 2.6% are impacted by this issue, representing 3,048,289 certs.
Of these 3 million, one million are duplicates for the same domain/subdomain, putting the actual number of impacted certs at roughly 2 million.
“Because of the way this bug operated, the most commonly affected certificates were those that are reissued very frequently, which is why so many affected certificates are duplicates,” Let’s Encrypt engineers explained today in a special FAQ page dedicated to the incident.
Tomorrow, Let’s Encrypt plans to revoke all the impacted certificates, starting with 00:00 UTC, March 4, 2020.
After this date, all the impacted certs will trigger errors in browsers and other applications. As a result, domain owners will have to request a new TLS certificate and replace the old one.
Let’s Encrypt says it started notifying impacted domain owners by email, but not all users listed a valid contact method.
System administrators and webmasters who are currently using Let’s Encrypt certificates for their networks and servers can check a list of impacted TLS certificate serial numbers on this page, or they can visit the following website and check to see if their certificate was impacted by entering their certificate’s domain name.
Last week, the same Let’s Encrypt project announced it had issued its one-billionth free TLS certificate, making it the most successful CA to date. In its five-year-old history, the project has managed to remain free of major incidents of abuse, although some platform-specific bugs have been reported once in a while. This one marks the first time the project is forced to revoke certificates. However, taking into account that the project provides free certificates, many of its users are most likely to look the other way.
