Fixing SSL Expired Error Due to DST Root CA X3 Expiration
Situation:
You have multiple servers that have been running without any issue for months or years, using certificates provided by letsencrypt.org, with certbot handling auto-renewals.
As of 10/1/2021 — you’re receiving SSL expired errors in certain applications, even though your certificates are most definitely not expired. Clients are yelling at you, your apps aren’t working (well some of them) — and you’re lost in a sea of comments on github issues.
If you’re like me, you don’t know much about certificates other than you need them for creating secure connections between your application, browsers and your service/servers.
We have multiple applications, across multiple OSes, multiple architectures, connecting to multiple services, accessing different APIs, blah blah blah you get the point.
The main issue we were having was that our Electron Apps could not connect to our APIs to do their business. It appears there is a fix for this in a new patch, but we cannot just willy-nilly update our apps to the latest version without doing some rigorous testing, and we need our apps to start working NOW.
We were able to fix this by doing the following on our servers:
First — we haven’t touched certbot in quite some time, as it’s been doing it’s business in the background without needing any attention. It’s now recommended to install and use via snap, as the previous version is no longer updated. Please note our servers are running Apache and nginx on Ubuntu > 18.04 — on AWS.
Step 1: Remove certbot that was installed via apt
sudo apt remove certbotStep 2: Verify snap is up to date, and install certbot via snap, and make sure the certbot command can be run.
sudo snap install core; sudo snap refresh coresudo snap install certbotsudo ln -s /snap/bin/certbot /usr/bin/certbot
Step 3: I ran the certbot dry run to make sure everything was working as expected. Please note that we have existing certificates already installed. If you do not have existing certificates, I don’t know why you’re reading this.
sudo certbot renew --dry-runStep 4: This is the actual fix for the problem:
sudo certbot --preferred-chain "ISRG Root X1"Certbot will then ask you what names you’d like to activate HTTPs for. Select the names or press enter for all of your domains. It will then tell you:
You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/***YOURCONFFILEHEREWhat would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Attempt to reinstall this existing certificate
2: Renew & replace the certificate (may be subject to CA rate limits)
Select option 2. Certbot will now do its thing, and install the new certificates.
If you see a success message, you’re almost there.
Step 5: Restart your webserver service using whatever method you want.
sudo service apache2 restartOR
sudo service nginx restartFinal step: Test your previously malfunctioning applications/endpoints. For me I deployed this fix on a development server first, and tested to make sure it was functioning on the devices I needed it to. After confirmation, I implemented the fix on the rest of our servers.
Please note: This worked for me and my specific use cases. It may not work for you, or may cause more problems. I also don’t believe that this is the ‘correct’ fix — I believe that the applications need to be updated to look past the initial expired certificate, and read the rest of the chain. This was just a stop gap I implemented to keep our support lines from catching on fire.
Best of luck in your endeavors.
Further reading:
