Adding a wildcard Let’s Encrypt certificate to your server without a web server

Whenever you start working on servers beyond a simple web server, you quickly get to the point where you need to use certificates to secure your TLS/HTTPS traffic. In this article I look at how you can add a Let’s Encrypt certificate to your server when you do not have a web server.

11 min readJan 27, 2024

Public Key Infrastructure (PKI)

PKI creates the trust and security on the Internet. Before PKI anyone could be anyone and anyone could snoop on anyones messages. PKI introduced the concept of authentication (proving who you are) and encryption (ensuring your messages are private and not tampered with).

If you want to learn more about PKI, I have written a previous article describing what it is, when it is used and how it works.

Whenever you have two servers that want to talk to each other, it is strongly recommended that their communications is secured. This means using TLS or HTTPS, secured with digital certificates and keys.

To secure communications you will need:

A private key for your server
A certificate for your server (which includes the public key to the private key)
A certificate for your Certificate Authority (CA) that signed the certificate

You can generate all these yourself. This is refered to as creating a self-signed certificate. Sounds great but there is a downside to this. A client that connects to your server will not know how to verify that your certificate is real.

To verify your certificate, the client needs the Certificate Authority (CA) certificate and needs to trust that that certificate is the one that you control. To do this you need to find a way to install your CA certificate on the client.

There is another way. By creating certificates using a well-known, trusted and publicly available CA, the client can validate the certificate directly with the CA itself.

There are a lot of certificate authorities that will sign your certificates for you and that are well-known. In this case, well-known means that their certificates (or, in the case of intermediate CAs, their parent’s certificates) are actually installed into operating systems so that client applications (like browsers) can automatically and quickly validate the server certificates.

For this article we are going to look at one of these CAs, called Let’s Encrypt.

Let’s Encrypt

Let’s Encrypt is a Certificate Authority (CA) that issues free certificates. Yes, truely free.

In 2013, Josh Aas and Eric Rescorla founded the non-for-profit Internet Security Research Group (ISRG). From their website:

Our mission is to reduce financial, technological, and educational barriers to secure communication over the Internet.

They are backed by Google, AWS , Cisco and others.

One of their products is a highly automated service to generate TLS certificates for free. The service is call Let’s Encrypt.

By automating the process and working with open source solutions, they keep costs to a minimum.

The Let’s Encrypt CA certificates are installed in all the major operating systems, meaning that their certificates are trusted without the need to copy CA certificates.

Note that Let’s Encrypt certificates expire after 90 days. This means that you need to refresh them to keep your server secure (which can be done automatically). This is ideal for experimenting as you are not paying for a 1 year certificate that you use for a few weeks.

Getting a Let’s Encrypt Certificate

As I mentioned, the process for getting certificates is highly automated, which makes things easier for us.

There are two ways to obtain a certificate, both rely on you proving who you are or, more importantly, whether you control the domain you want a certificate for.

The two ways are:

Via HTTP challenge
Via DNS challenge

HTTP Challenge

In this method, you install and use an application (certbot) to add a hook into your web server (eg: apache or NGINX). This allows Let’s Encrypt to connect to that hook, proving that you control the web server and hence the domain of the web server.

Once you are proven to be the owner of the domain, the certificates are generated and stored on your server.

I have used this method many times and it works well.

The main problem with this method is that your server has to have a web server running on it. This is not always the case. This is why there is a second method, DNS challenge.

DNS Challenge

When you do not have a web service you can use your DNS to prove who you are (or at least that you have control of the domain).

In this case the certbot application can generate a string of characters that you add to your DNS entry as a TXT entry. Let’s Encrypt then checks your DNS entry for that string and this confirms that you have control of the domain.

In the same way as the HTTP challenge, once you are proven to be the owner of the domain, the certificates are generated and stored on your server.

It is a very simple process. Now let’s look at the actual mechanics of doing it.

Wildcard certificates

Normally you would create a certificate for a very specific domain, such as: website.requillion-solutions.com. When you want to use a certificate across an entire domain with multiple servers, you need a wildcard certificate. These will act as a certificate for any sub-domain of the root domain, eg: *.requillion-solutions.com. This wildcard certificate could be used for:

www.requillion-solutions.com
website.requillion-solutions.com
mail.requillion-solutions.com
etc

It cannot be used for further levels of sub-domains, eg: www.my_site.requillion-solutions.com. For that you would need a wildcard certificate for *.my_site.requillion-solutions.com.

In this article I am looking at creating a wildcard certificate for my domain *.k8s.requillion-solutions.com.

Obtaining the certificate

As Let’s Encrypt work across many different operating systems and web server applications, they need to maintain good documentation on how to obtain a certificate. If you go to their web site, you will find instructions on how to obtain your certificates.

Anyone who follows me knows that I am using Ubuntu servers for all my infrastructure and so for this article I will describe how I obtain Let’s Encrypt certificates for the Ubuntu 22.04 operating system.

Install the utilities

First we will install the utilities required to manage our Let’s Encrypt certificates. Log in to one of your servers. For this example, I have a Kubernetes cluster with a k8s-master node, which I am going to use.

sudo apt update
sudo apt install letsencrypt -y

Request the certificate

Now we have the utilities, in particular certbot, we are going to ask it to obtain the certificate for us using:

sudo certbot certonly --manual --preferred-challenges=dns --email <your email address> --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d *.<your root domain>

This is a long command and needs a bit of explanation:

certbot — the utility that Let’s Encrypt uses to manage certificates on your server
certonly — we only need certbot to retrieve the certificates and save them on our server, not to install them for us
manual — certbot has various plugins to enable it to automatically update your DNS — I do not recommend this as you have to hand over your DNS credentials and so this flag tells certbot that I will do this update myself
preferred-challenges — in this case we are going to use the DNS challenge method
email — this allows Let’s Encrypt to get in contact with you when, for example, the certificates are about to expire
server — this is the API that certbot will use to obtain a certificate from Let’s Encrypt
agree-tos — this tells certbot that you agree to the Let’s Encrypt terms of service
d — one or more domains certificates are to be created for (in this case a wildcard domain due to the *. in front of the root domain) — in my case this would be *.k8s.requillion-solutions.cloud

It will then ask:

Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit
organization that develops Certbot? We'd like to send you email about our work
encrypting the web, EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o:

This is up to you and purely optional. enter y or n as appropriate and press return.

It will then ask:

NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o:

Again, this is up to you.

As we have asked to manually update DNS, it will then ask:

Please deploy a DNS TXT record under the name
_acme-challenge.<your root domain> with the following value:

iR-rfUOuBxB4l5bitFCRclO-acKJf1Gm0rKirbkwpGH

Before continuing, verify the record is deployed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue

Note that the string it gives you is random and the one above will not work as it has been made up by me.

Update DNS

I cannot tell you how to update your DNS entries as there are so many suppliers. Generally you need to:

Log in to your supplier’s site
Select the root domain and manage its DNS entries
Add a TXT record for the _acme-challenge subdomain and the value given above
Save the record

DNS entries take time to propagate across the Internet. You need to wait for that propagation to occur. You can check with www.nslookup.io. Ener your domain with the _acme-challenge subdomain. When the results come back with the TXT record value, you can continue with certbot.

Press return to confirm.

Obtain your certificates

Once your DNS entry has been verified, you will get the following response:

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/k8s.requillion-solutions.cloud/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/k8s.requillion-solutions.cloud/privkey.pem
   Your cert will expire on 2024-04-25. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

You can then check the certificates:

sudo ls -l /etc/letsencrypt/live/<your root domain>/

In this folder you will see the following:

-rw-r--r-- 1 root root 692 Jan 27 09:14 README
lrwxrwxrwx 1 root root  50 Jan 27 09:14 cert.pem -> ../../archive/requillion-solutions.cloud/cert1.pem
lrwxrwxrwx 1 root root  51 Jan 27 09:14 chain.pem -> ../../archive/requillion-solutions.cloud/chain1.pem
lrwxrwxrwx 1 root root  55 Jan 27 09:14 fullchain.pem -> ../../archive/requillion-solutions.cloud/fullchain1.pem
lrwxrwxrwx 1 root root  53 Jan 27 09:14 privkey.pem -> ../../archive/requillion-solutions.cloud/privkey1.pem

The README explains the use of these files:

This directory contains your keys and certificates.

`privkey.pem`  : the private key for your certificate.
`fullchain.pem`: the certificate file used in most server software.
`chain.pem`    : used for OCSP stapling in Nginx >=1.3.7.
`cert.pem`     : will break many server configurations, and should not be used
                 without reading further documentation (see link below).

WARNING: DO NOT MOVE OR RENAME THESE FILES!
         Certbot expects these files to remain in this location in order
         to function properly!

We recommend not moving these files. For more information, see the Certbot
User Guide at https://certbot.eff.org/docs/using.html#where-are-my-certificates.

Take note of the warnings.

Congratulations, you now have your certificates.

Testing the certificates

If you have been following, you will have a gateway server (gw) that is running NGINX. You can use this to test your certificates.

First ensure the public IP address of your gw server is accessible via the root domain (eg: k8s.requillion-solutions.cloud in my example).

Once you have set this up, you should be able to go to http://<your domain> within a browser and see the Welcome to nginx! message. If you see this, you can try going to the https://<domain> address. You will see an error message about the site not being accessible.

Now, you can install Let’s Encrypt certificates using a HTTP challenge and this will set up NGINX for you. For this test case, we already have the certificates and so we will set up NGINX manually.

Transfer the certificates

You need to securely transfer the two certificates to:

/etc/letsencrypt/live/<root domain>/fullchain.pem
/etc/letsencrypt/live/<root domain>/privkey.pem

First create the folders:

sudo mkdir -p /etc/letsencrypt/live/k8s.requillion-solutions.cloud

As the two files are text files, you can actually cut and paste them across. If you do this, make sure you include all headers and footers, including all hyphens!

Configure NGINX

Once you have the certificates in place, you can configure NGINX to use them.

If you have a vanilla NGINX installed, the default page you get is served via the default site configuration, which you can find here:

cat /etc/hginx/sites-available/default

As a test, you can insert the following virtual server configuration into that default file, either above or below the existing one:

server {
 listen 443 ssl;

 server_name <your domain name>;

 root /var/www/html;

 index index.html index.htm index.nginx-debian.html;

 location / {
      try_files $uri $uri/ =404;
 }

 # Redirect non-https traffic to https
 if ($scheme != "https") {
  return 301 https://$host$request_uri;
 } # managed by Certbot

 ssl_certificate /etc/letsencrypt/live/k8s.requillion-solutions.cloud/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/k8s.requillion-solutions.cloud/privkey.pem;
}

Things to note about what each line tell NGINX:

listen 443 ssl — this virtual server is to listen on port 443 and is to provide SSL (the original name for secure TLS connections)
server_name — the requested host name that NGINX uses to select this virtual server (for me this is k8s.requillion-solutions.cloud)
root — the location of any HTML we page files to be served to the user
index — what files to use if the user does not specify a file in the requested path
location — the path to be matched for displaying the index file
redirect block — this block redirects any http request to the same location but as an https request, ensuring a secure communications channel
ssl_certificate — the signed certificate with the server’s public key
ssl_certificate_key — the private key that matches the server’s public key

Test the configuration

Before applying the changed configuration, you should always test your NGINX configuration:

nginx -t

If your changes are valid, you should get:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Otherwise you will be told what and where the error may be.

Install the new configuration

Now you have a new configuration and certificates installed, you can now reload the configuration with:

systemctl restart nginx

Test the new configuration

NGINX normally restarts without any significant delay. This means that once you restart it, you should be able to go to https://<domain> and instead of getting a ‘not found’ error, you should see the Welcome to nginx! message.

Depending on your browser, you can then see the certificate details.

Ensuring automatic certificate refresh

I mentioned earlier that the Let’s Encrypt certificate expires after 90 days. You will be warned by email and be given the chance to renew it. When you got the certificates, the message told you how to do this:

sudo certbot renew

You can do this at anytime. It will only renew if you are getting close to the expiry time.

Whilst you can renew manually, I find that when the email reminder comes in, I tend to put it off until it is too late and my service goes off line!

To avoid this, you can add the renew command to a cron job.

Edit the crontab with:

sudo crontab -e

Add the following line to check each day:

0 12 * * * /usr/bin/certbot renew --quiet

The check will be made each day and will automatically renew the certificate when it gets to within 30 days of expiry.

Summary

In this article we created a Let’s Encrypt wildcard certificate using a DNS challenge. We added the challenge response to our DNS as a TXT record.

Having confirmed you have control over the DNS, certificates were then created and loaded on to your server.

Following this, we installed them into our gateway server and configured it to serve a virtual server over TLS. Once configured we were able to then test the site and prove the certificates worked.

Finally, we installed a crontab to periodically renew the certificate to ensure our service does not go off the air.

If you found this article of interest, please give me a clap as that helps me identify what people find useful and what future articles I should write. If you have any suggestions, please add them in the comments section.