Managing multiple Let’s Encrypt certificates with Oracle Cloud Infrastructure

14 min readOct 14, 2022

In my previous post I explained how you can use Let’s Encrypt and Oracle Cloud Infrastructure (OCI) serverless functions to obtain a publicly signed SSL certificate, and automatically manage its renewal lifecycle. The solution works as expected; I have a Let’s Encrypt certificate for my website automatically renewing 30 days before expiry. If you haven’t read my previous post I’d recommend taking a look before following the setup outlined below as it covers how the solution works, and some prerequisites.

Having multiple workloads running in various OCI regions I started thinking about a more elegant way to provision certificates across multiple regions. Certificates stored in the certificate service are only available to resources in the same region and would have required a function to be deployed in each region, and for each SSL certificate required.

I’ve since updated the solution to address this requirement. It is now possible to provision certificates across multiple OCI regions using a single OCI Function application. I’ve also taken the opportunity to implement other features such as:

Loading a list of certificates you want to manage from a JSON file stored in Object Storage.
Adding support for wildcard SSL certificates.
Adding support for Subject Alternative Names (SAN) in addition to the CN name.
Adding support for the use of DNS zones and Vaults that reside in different regions to the OCI Function.

Adding support to specify which vault, and region to use for a given certificate ensures that workloads with strict cryptographic key material requirements can still benefit from this solution.

If you’ve already followed the instructions from my previous post, the solution will continue to work as described. The only limitation being that it’ll only work for a single certificate. By following the steps below you can easily upgrade to issuing multiple certificates. If you haven’t set anything up yet that’s also fine as I’ll be covering the full install again here.

Setup & Configuration

First you’ll also need to install Docker, and the Fn Project. On OSX installing the Fn Project is easy with Homebrew:

$ brew install fn

To push the function to the OCI container repository you’ll also need an Auth Token which can be generated in OCI Identity by selecting your user, and clicking “Auth Tokens”.

Download the function source code from Github https://github.com/scotti-fletcher/oci-letsencrypt. As always I recommend inspecting any source code that you download from the Internet for security issues before use.

Now we’ll create a Functions application. An application is just a logical grouping of functions, and in this solution I only have one application called “acme-certbot” and one function called “lets-encrypt”.

After creating the application, follow the “Getting Started” instructions for “Local setup”. The instructions displayed will be unique for your OCI environment, however I’ll explain the steps as not all are required for this example.

In terminal, cd into the lets-encrypt directory containing the function files:

scott@scott-mac lets-encrypt % cd ~/Projects/oci-certbot/lets-encrypt 
scott@scott-mac lets-encrypt % ls 
Gemfile func.rb func.yaml	models.rb

Create a context for this application and select it for use.

$ fn create context acme-certbot --provider oracle $ fn use context acme-certbot

Update the context with the compartment OCID where you created the application and the Oracle Functions API URL for the region where the function lives.

$ fn update context oracle.compartment-id [your compartment ocid] 
$ fn update context api-url https://functions.ap-sydney-1.oraclecloud.com

Create a repository where your function container image will be stored.

$ fn update context registry syd.ocir.io/[your namespace]/acme-certbot

Log into the OCI container registry.

$ docker login -u '[your namespace]/oracleidentitycloudservice/scott.fletcher@oracle.com' syd.ocir.io

You should see a “Login Successful” message. You can now build and deploy your application. The first time you do this it might take a few minutes:

scott@scott-mac lets-encrypt % fn deploy --app acme-certbot Deploying lets-encrypt to app: acme-certbot 
Bumped to version 0.0.19 
Using Container engine docker 
Building image syd.ocir.io/abcd/acme-certbot/lets-encrypt:0.0.19 ...... 
Parts: [syd.ocir.io abcd acme-certbot lets-encrypt:0.0.19] 
Using Container engine docker to push 
Pushing syd.ocir.io/abcd/acme-certbot/lets-encrypt:0.0.19 to docker registry...
The push refers to repository [syd.ocir.io/abcd/acme-certbot/lets-encrypt] d15aa7ac2cf0: 
Pushed f9a15c3fd1e8: 
Pushed 111ae97432f0: 
Layer already exists fb57e582ca84: 
Layer already exists d55982f12cfa: 
Layer already exists 04aab8128d11: 
Layer already exists 05dc728e5e49: 
Layer already exists 0.0.19: digest: sha256:d86dcf6889fa8fe3a036da8e17fbb7733c50369a158c1daf2649eaaf7d232c40 size: 1783 
Updating function lets-encrypt using image syd.ocir.io/abcd/acme-certbot/lets-encrypt:0.0.19...

Now we need to deploy the second worker function. Change to the worker directory:

And we run the same deploy command, which will build and upload the function to the same application:

scott@scott-mac worker % fn deploy --app acme-certbot 
Deploying worker to app: acme-certbot 
Bumped to version 1.0.1 
Using Container engine docker 
Building image syd.ocir.io/abcd/acme-certbot/worker:1.0.1 ... 
Parts: [syd.ocir.io abcd acme-certbot worker:1.0.1] 
Using Container engine docker to push 
Pushing syd.ocir.io/abcd/acme-certbot/worker:1.0.1 to docker registry...
The push refers to repository [syd.ocir.io/abcd/acme-certbot/worker] 5e5b0e4eac3e: 
Pushed e671531a01e8: 
Pushed 111ae97432f0: 
Mounted from abcd/acme-certbot/lets-encrypt fb57e582ca84: 
Mounted from abcd/acme-certbot/lets-encrypt d55982f12cfa: 
Mounted from abcd/acme-certbot/lets-encrypt 04aab8128d11: 
Mounted from abcd/acme-certbot/lets-encrypt 05dc728e5e49: 
Mounted from abcd/acme-certbot/lets-encrypt 1.0.1: digest: sha256:6c4c70ae061117a62f0a5053ccec32755a8f6b92f5de0965bdb7a42b537a91de size: 1783 
Updating function worker using image syd.ocir.io/abcd/acme-certbot/worker:1.0.1... 
Successfully created function: worker with syd.ocir.io/abcd/acme-certbot/worker:1.0.1

Once created, you should be able to see both functions in the OCI console under applications and also in the Container Registry.

Now we need to create a Log Group to hold our Function specific logs. Create a log group, I’ve named mine “cert-bot-logs”:

In logs, click “Create custom log” and call it “cert-bot-activity”:

When prompted select “Add configuration later” and click create. You should see the log “cert-bot-activity” in the “cert-bot-logs”. Note down the OCID of the cert-bot-activity log as you will need this later.

You will also need a Master Encryption Key in each OCI vault that you want to use. For example if you need to keep certificates and keys in the Tokyo region for workloads based in Japan, and certificates and keys in Sydney for Australian workloads then you will need two vaults each containing a Master Encryption Key. If you don’t have such a requirement, then you only need one vault and one Master Encryption Key.

If you don’t already have a Master Encryption Key in OCI Vault then you will need to create one:

Note down the OCID’s of the vaults and Master Encryption Keys as you will need them later.

Now we’ll go back to our Function, and update the required configuration items. Configuration items are just environment variables that are made accessible to the function:

You need to configure the following items:

OCI_LOG_OCID. This is the OCID of the cert-bot-activity-log you created earlier.
VAULT_SECRET_NAME. This is the name of the secret that will hold your Let’s Encrypt account Private Key. You can use “acme-cert-bot-key”, or change it if you wish.
CERT_CONTACT. This is the email address associated with the certificates that will be issued.
USE_CONFIG. Entering TRUE here will indicate that the function should read the certificate config.json file from the Object Storage bucket. This was added for backwards compatibility where values for a single certificate were provided as environment variables as described in my previous post.
LETS_ENCRYPT_URI: This is the Lets Encrypt API endpoint.

If you have already followed the instructions in my previous post then you can remove all other configuration items as we’ll be specifying them in the config.json file.

Functions also emit logs, and it’s useful to see them for debugging purposes. To enable these logs, click “Enable Log”:

Now we need to create our config file. In the same region as your function, create an object storage bucket called “lets-encrypt”:

Now we need to create a file called config.json to hold our certificate configuration:

{
  "certificates": [
    {
      "cn_name": "sydney.dflect.me",
      "alt_names": [],
      "dns_zone_name": "dflect.me",
      "dns_region": "ap-sydney-1",
      "certificate_region": "ap-sydney-1",
      "cert_compartment_ocid": "OCID of compartment where you want certificate created",
      "auto_deploy": true,
      "vault_region": "ap-sydney-1",
      "vault_ocid": "OCID of the vault in Sydney",
      "vault_master_key_ocid": "OCID of the Master Encryption key from Sydney vault",
      "renew_days_before_expiry": 30
    },
    {
      "cn_name": "tokyo.dflect.me",
      "alt_names": ["tokyo1.dflect.me", "tokyo2.dflect.me"],
      "dns_zone_name": "dflect.me",
      "dns_region": "ap-sydney-1",
      "certificate_region": "ap-tokyo-1",
      "cert_compartment_ocid": "OCID of compartment where you want certificate created",
      "auto_deploy": true,
      "vault_region": "ap-tokyo-1",
      "vault_ocid": "OCID of the vault in Tokyo",
      "vault_master_key_ocid": "OCID of the Master Encryption key from Tokyo vault",
      "renew_days_before_expiry": 30
    }
  ]
}

In my config file I’ve configured two certificates to be managed, one using my vault in Sydney and one using the one vault in Tokyo. As my dflect.me DNS zone home region is Sydney, “dns_region”: “ap-sydney-1” is present in both configurations. You will also need to configure:

cn_name. This is the common name subject of the certificate to be issued. A fully qualified domain name (FQDN) or wildcard domain can be specified.
alt_names. This is an array of Subject Alternative Names that you wish to assign to the certificate.
dns_zone_name. This is the name of your DNS zone.
dns_region. This is the region where you created the DNS zone.
certificate_region. This is the region where you want the certificate to be created.
cert_compartment_ocid. This is the compartment where you want the certificate to be created.
auto_deploy. Setting this to true will make future renewed certificates the “CURRENT” version in the certificate service, triggering the certificate update on any associated load balancers.
vault_region. This is the region where you created the vault.
vault_ocid. This is the OCID of the vault where the Master Encryption Key you want use is located.
vault_master_key_ocid. This is the OCID of the Master Encryption Key you want to use, or created earlier in the respective vault.
renew_days_before_expiry. This defines how many days before the certificate is due to expire that you wish to renew the certificate. I’ve chosen that my cert should be renewed 30 days before expiry.

Note: If you are using a wildcard CN name e.g. *.dflect.me then you must not specify any alternative names otherwise Let’s Encrypt validation will fail and not issue a certificate.

Save the file as config.json and upload it to the lets-encrypt object storage bucket:

Before we test our function, we need to create a Dynamic Group, and Policies to allow our function to operate in our compartment. Depending on your use-case and in which compartments your Vault and DNS Zones reside you may need to adjust the policies. My examples below are scoped to my sandbox compartment.

Create a new policy, with the following policy statements. If you’ve chosen different Dynamic Group and Compartment names, you will need to update the policy statements accordingly:

Allow dynamic-group acme-certbot-dg to use log-content in compartment scott_fletcher_sandbox
Allow dynamic-group acme-certbot-dg to use dns in compartment scott_fletcher_sandbox
Allow dynamic-group acme-certbot-dg to manage leaf-certificate-family in compartment scott_fletcher_sandbox
Allow dynamic-group acme-certbot-dg to manage secret-family in compartment scott_fletcher_sandbox
Allow dynamic-group acme-certbot-dg to manage key-family in compartment scott_fletcher_sandbox
Allow dynamic-group acme-certbot-dg to use vaults in compartment scott_fletcher_sandbox
allow dynamic-group acme-certbot-dg to use objects in compartment scott_fletcher_sandbox
Allow dynamic-group acme-certbot-dg to use fn-invocation in compartment scott_fletcher_sandbox
allow service faas to {KEY_READ} in compartment scott_fletcher_sandbox where request.operation='GetKeyVersion'
allow service faas to {KEY_VERIFY} in compartment scott_fletcher_sandbox where request.operation='Verify'

These statements allow my function to:

Push logs to the cert-bot-activity log.
Update the DNS Zone to respond to the TXT record Let’s Encrypt challenge.
Import certificates into the Certificate Service.
Create and read your Let’s Encrypt account private key stored as a secret in the Vault.
Read the config.json file in our object storage bucket.
Invoke our worker function that will issue / renew certificates.
Use your Master Encryption Key when encrypting your Let’s Encrypt account private key.

Now with everything configured, we can invoke the function to create a certificate. This may take 1–2 minutes. You should see a “Completed Successfully” message returned:

scott@scott-mac lets-encrypt % fn invoke acme-certbot lets-encrypt "Completed Successfully"

Looking at the metrics we can see the lets-encrypt function was invoked once, with the worker function invoked twice (one per certificate)

If we take a look at our cert-bot-activity log we can see the certificate was created:

Looking at our DNS Zone records we can see a TXT records were created for all our CN names, and SAN’s:

Looking at our Sydney & Tokyo vaults, we can see the acme-cert-bot-key secret has also been added:

And lastly we can see the certificates are imported into the Certificate Service in their respective regions:

When the function is invoked again, we will see future versions added to the existing certificates with the most recent promoted to the current version:

Now we need to associate the certificates with their respective Load Balancer listeners. You can create a HTTPS listener, or edit an existing listener. Select the certificate that has been created:

Once the Load balancer work request has completed, you can now browse to your website. In my case it’s https://sydney.dflect.me

I can also view the certificate that was issued by Let’s Encrypt:

The awesome thing now, is that whenever the function is invoked, if a new certificate is required (30 days before expiry in this example) a new certificate will be generated, promoted to current, and automatically pushed to any associated resources. This means my Load Balancers will forever have valid SSL certificates.

The last thing to do is to configure a mechanism by which to invoke the function in OCI. There are a few ways you could achieve this:

Run it manually as required via the command line (as shown earlier)
Run it via a cron job from a compute instance, or other scheduler that you use
Run it automatically using a method to trigger the function from within OCI.

Because I don’t want to run it manually, and I don’t have a scheduler I’m going to use OCI Alarms to trigger my function once a day. My colleague @callanhp has a great article on how to do this https://redthunder.blog/2022/05/03/a-better-mechanism-for-periodic-functions-invocation/. It’s a good read and dives further into detail of how the approach works.

First we need to create a topic:

After creating a topic, create a subscription that will call our lets-encrypt function:

Now we need to create an alarm:

Because I’m running multiple compute instances I’m choosing a metric that will always fire.

Trigger the rule for when the value is greater than -1, meaning the alarm will always fire. You may need to enter 1, then press the down arrow key to get to the value of -1. You can validate the metric will work by looking at the graph. The blue line indicates the metric will fire the alarm:

Now all we need to do is configure the alarm to send a message to our notification topic. Note I have configured the alarm to repeat the notification every 24 hours. This will ensure our lets-encrypt function runs once a day:

To confirm the lets-encrypt function is running each day, look at the Function metrics:

Awesome, now we have a fully automated solution to allow us to issue and renew multiple Let’s Encrypt certificates in any OCI region, and automatically update Load Balancers.

Other Considerations / Thoughts

In case you haven’t read them in my previous post, here they are again.

When requesting Let’s Encrypt certificates, the cert chain that is provided doesn’t include all certificates to the trust root. If you’ve tried to manually import Let’s Encrypt certificates into OCI’s Certificate Service you will have likely received a “trust chain error”. I’ve handled this in the function by traversing the CA Issuer tree and building the correct cert chain that is imported. If you look in the source code you will see how it’s done.
I’d recommend enabling Cloud Guard in your OCI tenancy. It’s free and amongst a range of awesome Cloud Security Posture Management features it has detectors to identify when certificates on Load Balancers are expiring. If you are renewing certificates 30 days before expiry then setting your Cloud Guard detector to identify certificates that are due to expire in 20 days will ensure you are alerted if the function fails to run or execute successfully.
If you are looking to do end-to-end SSL encryption, then you can leverage OCI’s private certificates on backend sets. You can also use init scripts when provisioning compute instances to retrieve, install, and trust these certificates.
My IAM policies are scoped to everything in my sandbox compartment. If you wanted further control over what the function has access to, you could scope these to specific resource OCID’s.
I’ve chosen to set the auto_deploy configuration variable to true in config.json, meaning renewed certificates will automatically be the current version, triggering the auto update on associated Load Balancers. If you want to just retrieve the certificate and not have it automatically pushed to Load Balancers then set this to false. If you set it to false, then you will need to monitor when the certificate is renewed and mark it as “current” manually.
I’m storing the account private key as a secret in the Vault. This is because if you need to revoke a certificate you will need both the account private key and the certificate.
To handle multiple certificates I’ve chosen to split out the initial entry function “acme-certbot/lets-encrypt” from the worker function “acme-certbot/worker” which actually performs the certificate renewal task. This means you can issue and renew hundreds of certificates specified in the config.json file, if you actually need that many.

If you’ve found this useful, please share! If you have any questions you can reach out to me on LinkedIn https://www.linkedin.com/in/scotti-fletcher/

Originally published at http://redthunder.blog on October 14, 2022.

Managing multiple Let’s Encrypt certificates with Oracle Cloud Infrastructure

Setup & Configuration

Other Considerations / Thoughts

Written by Scotti Fletcher