0th Root Secure Network
Let’s Encrypt change affects OpenSSL 1.0.x and CentOS 7
Default certificate chain to include an expired root certificate
Let’s Encrypt is a Certificate Authority run by Internet Security Research Group (ISRG) that provides free and automated TLS certificates for secure communication over the Internet. In the context of the web, whenever the browser connects to a website over https, the browser verifies the TLS certificate of the website to make sure the communication is secure.
This process of verifying a TLS certificate requires the web browser, or the operating system, to maintain a trusted database of root certificates. And, starting with the TLS certificate provided by the website, its Issuer is verified by creating a certificate chain/path to a root certificate present in the trusted database.
Unfortunately, these root certificates in the trusted database have an expiry date. And, correspondingly any intermediate certificates issued by these root certificates will also align to that expiry date (except for the newly cross-signed ISRG Root X1 intermediate certificate explained below).
Cross-Signed Let’s Encrypt R3 and DST Root CA X3, intermediate and root certificates, expire on Sep 29, 2021 and Sep 30, 2021 respectively.
“Android intentionally does not enforce the expiration dates of certificates used as trust anchors.”
Root Store Inclusion
The success of any Certificate Authority ultimately depends on the platforms and devices where their root certificate is trusted. This allows for the TLS clients on the platforms and devices to successfully verify the certificates issued by them.
When Let’s Encrypt started out in 2015, their new root certificate ISRG Root X1 wasn’t trusted on any device or platform. Hence, the cross-sign of their intermediate certificate with 5 years of validity by DST Root CA X3 root certificate was a significant milestone, it was extended for another year in 2020.
Typically, a root certificate is used to sign a bunch of intermediate certificates that can issue certificates to subscribers. These subscriber certificates contain only the intermediate certificate mentioned as its Issuer, without any mention of the root certificate.
Hence, there can be multiple intermediate certificates with the same name and public key signed by different Certificate Authorities, which are called cross-signed certificates. This results in different certificate chains and paths for the same subscriber certificate.
New Cross-Sign
The root certificate ISRG Root X1 is now trusted by most platforms and devices. However, to maintain compatibility with older Android devices, Let’s Encrypt has decided to take an unique approach,
IdenTrust has agreed to issue a 3-year cross-sign for our ISRG Root X1 from their DST Root CA X3. The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3. This solution works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. ISRG and IdenTrust reached out to our auditors and root programs to review this plan and ensure there weren’t any compliance concerns.
The original plan was to transition to their own root in early January. However, with the change of plans to use a special cross-sign, the newly issued certificates (since May 4, 2021) use a longer chain with cross-signed ISRG Root X1 as an intermediate certificate.
The irony here is, Let’s Encrypt issued a new ECDSA root certificate ISRG Root X2 last year stating smaller certificate sizes as a big benefit. However, with the new cross-sign, additional 512 bytes of certificate data needs to be transmitted for every connection made.
OpenSSL 1.0.2 — Not Supported
Unfortunately, due to the way certificate paths are built and verified, not all implementations of TLS can successfully verify the cross-sign. This is the case with OpenSSL 1.0.2. Hence, programs running on RHEL/CentOS 7 that use OpenSSL will fail to verify the new certificate chain or establish TLS connection.
Upgrading to OpenSSL 1.1.x is not straightforward and will likely break some applications. With that said, there are a few options: either update the trust store on the client side (or) change the certificate chain on the server side.
The problems are very similar to the Sectigo root certificate expiry last year. Hence, removing the DST Root CA X3 root certificate from the trusted list on the client side (or) removing the newly cross-signed intermediate certificate from the chain on the server side should solve the problem.
However, there is a subtle difference. Last year, to avoid the problem, an issuing authority can decide to not include the problematic intermediate certificate in the chain. Here, a conscious decision has been made to have the default chain use the cross-signed certificate.
Hence, a clean solution would be to switch to the alternate chain and live with support only for Android 7.1.1 or later. However, if you have a mix of older android devices and clients with OpenSSL 1.0.x to support, switch to a different certificate authority, maybe?
Certbot
Since Version 1.6.0, Certbot ACME client supports the option --preferred-chain to choose the shorter Let’s Encrypt chain. This option can be used as shown below,
certbot renew --dry-run --preferred-chain "ISRG Root X1"Or, add the following to configuration under /etc/letsencrypt/renewal/,
preferred_chain = ISRG Root X1Due to a bug in certbot, version 1.12.0 will be required to choose the shorter chain. However, as on 6-Oct-2021, latest version available on EPEL for CentOS 7 is only 1.11.0. Hence, choosing the shorter chain is not as straight forward on CentOS 7.
Test Certificates
If you wish to reproduce the problem yourself, you can use the test certificates below,
Soon To Expire CA, Very Old X3
Main CA, Root X1 and cross-signature by Very Old X3
Main CA, Intermediate R3
Subscriber, localhost
Subscriber, test.example.com
- test.example.com.pem, test.example.com.txt
- fullchain: chain-long-example.pem, chain-short-example.pem
CA files for verification
Start a test server as follows,
nc -kCl --ssl-cert chain-long-example.pem --ssl-key chain-long-example.pem 4433(Remember to add a hosts entry for test.example.com to the corresponding IP address.)
OpenSSL 1.0.2k (CentOS 7)
The commands below try to establish a TLS connection to test.example.com on port 4433 at a future time — Oct 1, 2021. As you can see below, certificate verification fails if the root certificate is expired. However, without the expired root certificate in the trusted list, verification is successful.
$ faketime '1 Oct 2021' openssl s_client -connect test.example.com:4433 -CAfile x3+x1.pem -quiet
depth=3 C = US, ST = California, L = San Francisco, O = Soon To Expire CA, CN = Very Old X3
verify error:num=10:certificate has expired
notAfter=Sep 30 09:36:31 2021 GMT$ faketime '1 Oct 2021' openssl s_client -connect test.example.com:4433 -CAfile x1.pem -quiet
depth=2 C = US, ST = California, L = San Francisco, O = Main CA, CN = Root X1
verify return:1
...
Wget 1.14 (CentOS 7)
Unfortunately, due to this change, fundamental tools such as wget will fail to verify Let’s Encrypt certificates. I am hoping that the ca-certificates package will be updated and the expired root certificate will be removed, or some other solution will be provided.
$ faketime '1 Oct 2021' wget -q https://test.example.com:4433/ --ca-certificate x1.pem -O-
Hello World$ faketime '1 Oct 2021' wget https://test.example.com:4433/ --ca-certificate x3+x1.pem -O-
...
(test.example.com)|192.168.1.112|:4433... connected.
ERROR: cannot verify test.example.com's certificate, issued by ‘/C=US/ST=California/L=San Francisco/O=Main CA/CN=Intermediate R3’:
Issued certificate has expired.
...
Curl 7.29 (CentOS 7)
Curl doesn’t seem to be affected, as it is built with NSS instead of OpenSSL.
$ faketime '1 Oct 2021' curl https://test.example.com:4433/ --cacert x1.pem
Hello World$ faketime '1 Oct 2021' curl https://test.example.com:4433/ --cacert x3+x1.pem
Hello World$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
...
Final Thoughts
It is not yet clear, what the impact will be on CentOS 7. In fact, I am not even sure whether certbot ACME client will be able to fetch certificates on CentOS 7 servers since the API endpoint uses a similar certificate chain,
$ faketime '1 Oct 2021' openssl s_client -connect acme-v02.api.letsencrypt.org:443 -quiet
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMTConsidering RHEL 7 and other distributions such as OL 7 (Oracle Linux) and AMI 2 (Amazon Machine Image) could be impacted, the issue could be widespread. But, all this only if DST Root CA X3 root certificate isn’t removed by the ca-certificates package. Once it is removed, impact should be minimal.
References
- DST Root CA X3 Expiration (September 2021)
- Extending Android Device Compatibility for Let’s Encrypt Certificates
- OpenSSL Client Compatibility Changes
At 0th Root, we provide a solution 0th Root Secure Network — 0SNet to secure organization’s internal web apps with TLS client certificates. Do check out our product, it is easy to deploy and available as images on AWS, GCP and Azure.
