Google Cloud CDN Best Practice Series: Live Traffic Migration
Pre-Deploying a SSL Certificate on Cloud CDN
UPDATE (Jan 27, 2023): DNS-01 auth is now available via command line tooling so that you can pre-issue a Google Managed SSL certificate WITHOUT having to point the resource name to the Google Load Balancer. This makes all the fancy maneuvering you once had to perform a moot issue. Just create a CNAME resource record in your DNS zone file with the nonce value. For historical sake, To learn how, see my latest article on deploying a wildcard certificate using the Google Cloud Certificate Manager service.
At the time of this writing, Google’s managed certificate process will issue certificates signed by the Google Trust Services only when the hostname is pointed to a Google IP address. ̶G̶o̶o̶g̶l̶e̶ ̶c̶a̶n̶’̶t̶ ̶p̶r̶o̶v̶i̶s̶i̶o̶n̶ ̶a̶ ̶G̶o̶o̶g̶l̶e̶ ̶m̶a̶n̶a̶g̶e̶d̶ ̶S̶S̶L̶ ̶c̶e̶r̶t̶i̶f̶i̶c̶a̶t̶e̶ ̶i̶f̶ ̶t̶h̶e̶ ̶D̶N̶S̶ ̶r̶e̶c̶o̶r̶d̶ ̶i̶s̶ ̶p̶o̶i̶n̶t̶i̶n̶g̶ ̶t̶o̶ ̶a̶ ̶n̶o̶n̶-̶G̶o̶o̶g̶l̶e̶ ̶I̶P̶ ̶a̶d̶d̶r̶e̶s̶s̶.̶ If you want to use a Google managed SSL certificate on the Google Cloud CDN hostname this can cause an issue if the configured hostname is already delivering live HTTPS traffic. This article discusses a workaround to generate a Let’s Encrypt SSL certificate as a bridge to allow you to migrate live HTTPS traffic to Cloud CDN before pointing the hostname to the Google Cloud CDN IP address.
Challenge
The Google certificate provisioning system can support both customer provided SSL certificates as well as manage Google signed SSL certificates. Google validates domain ownership by looking at the DNS record of the target hostname and determines if it is pointing to a Google IP address. If the hostname is pointed to a Google IP address, the Google Trust Services will issue the certificate and manage any subsequent renewals on behalf of the customer. When a Cloud CDN hostname is actively delivering HTTPS traffic already, you cannot point the CDN hostname over to a Google IP address without causing a degradation in the end user quality of experience. There will be a period of time when HTTPS requests will fail as there will be no valid SSL certificate on the Google load balancer supporting the Cloud CDN instance. It can take upwards of 30 minutes or longer for the Google certificate provisioning system to issue the SSL certificate once a domain is pointed over to it. Another factor that can affect the speed of the certificate issuance is your DNS resource record time-to-live value which will dictate how quickly the changeover to Cloud CDN will take; which in turn, can effect the time that it takes to issue the Google managed SSL certificate. Once the certificate is issued by the Google Trust Services, the SSL certificate is automatically deployed; nevertheless, there could be a period of time where HTTPS Cloud CDN request will fail without a SSL certificate in place when the domain is in transition.
Solution
To pre-issue a Google managed SSL certificate, you must use the alpha gcloud command line in order to generate the DNS-01 auth signature value. Make sure that you have the gcloud alpha enabled: version Google Cloud SDK 345.0.0, alpha 2021.06.11 or newer installed in your terminal.
Open Cloud Shell Terminal
First, navigate to your project in the Cloud Shell terminal. See Step 6 for directions on setting the project in your Cloud Shell terminal.
A) Verify GCLOUD version:
gcloud versionYou will want to make sure that you have the gcloud version Cloud SDK version is: 346.0.0 or above and have the alpha commands installed in your terminal.
If you do not have the alpha commands installed in your Cloud Shell terminal, you can run the following command:
gcloud components install alphaB) Enable IAM permissions for certificate manager
You must have the following Identity and Access Management permissions to authorize, create, and attach certificates to an Edge Cache service:
* certificatemanager.certs.create
* certificatemanager.certs.get
* certificatemanager.certs.list
* certificatemanager.dnsauthorizations.create
* certificatemanager.dnsauthorizations.get
* certificatemanager.dnsauthorizations.list
* certificatemanager.dnsauthorizations.useC) Generate a certificate request:
gcloud alpha certificate-manager dns-authorizations create [name-of-ssl-certifiacate] \
--domain="[hostname value]"The [name-of-ssl-certificate] value is the friendly name that you will refer to this certificate within GGCP. This is NOT the FQDN value.
example: foobar-example-com-ssl-cert
The [hostname value] variable should be replaced with the hostname value that you want to issue the SSL certificate for.
example: foobar.example.com
Input
gcloud alpha certificate-manager dns-authorizations create foobar-example-com-cert --domain="foobar.example.com"Output
Create request issued for: [foobar-example-com-cert]
Waiting for operation [projects/projectName/locations/global/operations/operation-15703233-5c5840590b-a7ff6466-dd11e] to complete...done.Created dnsAuthorization [foobar-example-com-cert].
NAME DOMAIN DNS_RECORD RECORD_TYPE DNS_VALUE
foobar-example-com-cert foobar.example.com _acme-challenge.foobar.example.com CNAME aab6acc8-1234-5678-967e-71.0.authorize.certificatemanager.goog.
D) Create CNAME resource record in zone file
The next step in the process is to create a resource record in your zone file. Google will look to verify the domain ownership by querying the _acme.challenge hostname and verifying the value in the query answer. To do this, you must have access to your domain’s zone file.
Create CNAME resource record using the _acme-challenge.foobar.example.com generated with the certificate request process step. The resource record will always be in the format of _acme-challenge.[hostname_value] . With this value in hand, you need need to create a CNAME record value with the acme-challenge hostname and the DNS Value is the nonce signature that is generated by the certificate request command. In the above example, the DNS Value is aab6acc8–1234–5678–967e-71.0.authorize.certificatemanager.goog. (with the trailing period). to prevent adding on any additional apex domain value.
E) Verify the SSL certificate has been issued.
Old Solution <out of date>
̶A̶s̶ ̶p̶r̶e̶v̶i̶o̶u̶s̶l̶y̶ ̶m̶e̶n̶t̶i̶o̶n̶e̶d̶,̶ ̶G̶o̶o̶g̶l̶e̶ ̶c̶a̶n̶n̶o̶t̶ ̶p̶r̶e̶-̶i̶s̶s̶u̶e̶ ̶a̶ ̶G̶o̶o̶g̶l̶e̶ ̶m̶a̶n̶a̶g̶e̶d̶ ̶S̶S̶L̶ ̶c̶e̶r̶t̶i̶f̶i̶c̶a̶t̶e̶ ̶f̶o̶r̶ ̶a̶ ̶h̶o̶s̶t̶n̶a̶m̶e̶ ̶t̶h̶a̶t̶ ̶i̶s̶ ̶s̶u̶p̶p̶o̶r̶t̶i̶n̶g̶ ̶l̶i̶v̶e̶ ̶t̶r̶a̶f̶f̶i̶c̶ ̶s̶i̶n̶c̶e̶ ̶i̶t̶ ̶c̶a̶n̶n̶o̶t̶ ̶b̶e̶ ̶p̶o̶i̶n̶t̶e̶d̶ ̶t̶o̶ ̶a̶ ̶G̶o̶o̶g̶l̶e̶ ̶I̶P̶ ̶t̶o̶ ̶v̶a̶l̶i̶d̶a̶t̶e̶ ̶d̶o̶m̶a̶i̶n̶ ̶o̶w̶n̶e̶r̶s̶h̶i̶p̶ ̶w̶i̶t̶h̶o̶u̶t̶ ̶i̶n̶t̶e̶r̶f̶e̶r̶i̶n̶g̶ ̶w̶i̶t̶h̶ ̶H̶T̶T̶P̶S̶ ̶d̶e̶l̶i̶v̶e̶r̶y̶ ̶(̶s̶i̶n̶c̶e̶ ̶t̶h̶e̶r̶e̶ ̶i̶s̶ ̶n̶o̶ ̶p̶r̶e̶-̶e̶x̶i̶s̶t̶i̶n̶g̶ ̶c̶e̶r̶t̶i̶f̶i̶c̶a̶t̶e̶ ̶i̶n̶ ̶p̶l̶a̶c̶e̶)̶. This document outlines a workaround whereby a Let’s Encrypt certificate is issued as a customer managed certificate as a bridge to allow customers to point their domain to a Google IP address such that the Google Trust Services can validate domain ownership and issue a Google managed SSL certificate. This procedure outlines issuing the certificate using a DNS TXT validation scheme.
Step 1: Create CAA Record
You should start by creating 2 Certificate Authority Authorization (CAA) resource records in their DNS. The CAA record provides the issuing certificate authority permission to issue the SSL certificate on your behalf. One is for Let’s Encrypt and the other is for the Google Trust Services. These entries provide a secondary level of authorization for issuing the customer SSL certificate, whereby the primary authorization is made with a DNS token value as outlined in Step 3.
Format:
+-------------------+---------+-------------+-----------------+
| Hostname | DNS TTL | Record Type | Value |
+-------------------+---------+-------------+-----------------+
| [FQDN - hostname] | 300 | CAA | pki.goog |
| [FQDN - hostname] | 300 | CAA | letsencrypt.org |
+-------------------+---------+-------------+-----------------+Example:
+-----------------------------------------------------+
| cdn.example.com 300 IN CAA 0 issue “pki.goog” |
| cdn.example.com 300 IN CAA 0 issue “letsencrypt.org” |
+-----------------------------------------------------+Step 2: Create DNS Challenge for Let’s Encrypt
Let’s Encrypt makes use of an application called Certbot that can call the Let’s Encrypt certificate authority and issue a DNS validation token. Note, that Certbot can also issue a HTTP validation, but this validation scheme is beyond the scope of this article. Certbot can run on a customer Web server to automatically deploy and update the certificate. For the purposes of this procedure, we are going to illustrate running Certbot in STANDALONE mode which allows you to generate a token and validate the token without having to deploy it on a particular Web server.
Install Certbot
Depending on your compute environment, there are different installation routines. Instructions for installation can be found here: https://certbot.eff.org/instructions. For this example, we are installing Certbot on Debian 9 with no specific Web server.
- SSH to Debian 9 instance
- Install Certbot
$ sudo apt-get install certbot3. Install Certbot in STANDALONE mode
$ sudo certbot certonly --standaloneStep 3: Generate DNS Challenge Token
Now that Certbot is installed, we next need to generate the DNS token value that you will use as an input for the DNS challenge value.
Format:
certbot -d [hostname_value] --manual --preferred-challenges dns certonly [--dry-run]`NOTE: adding the --dry-run directive will allow you to test the call to Let’s Encrypt without actually issuing a token and SSL certificate. This is helpful to test your call to Let’s Encrypt to make sure your values are correct. Let’s Encrypt enforces a request quota so if you make too many attempts, you may be blocked from issuing a certificate request.
Example:
certbot -d cdn.example.com --manual --preferred-challenges dns certonlyOutput:
What we are looking for is the value RED highlighted area in the picture above (partially blurred out). This is the token value that we need to enter in a DNS entry. Let’s Encrypt will validate the domain ownership by referencing the nonce value generated from Certbot. The customer will need to create a DNS TXT entry using this value. The hostname value will always be in the form of _acme-challenge.[hostname_value] where the hostname_value is the domain/subdomain name we are trying to issue the SSL certificate for. It is possible to issue the certificate with multiple hostname values. You just need to add a -d [hostname] for each additional hostname you want issued on the Let’s Encrypt certificate. After executing the command, you will be issued a DNS challenge per hostname, e.g. you will need to make multiple DNS TXT entries in the DNS zone file.
Format:
Example:
DO NOT click enter to validate the hostname in Certbot until you see the DNS challenge value in the hostname DNS zone file. If you click enter before the TXT file is propagated, Let’s Encrypt will not validate the domain and you will have to wait for Let’s Encrypt to make a second attempt. Let’s Encrypt will only make 2 attempts to validate the hostname before it will nullify the token. If this happens, you will need to regenerate a new DNS challenge token and modify the hostname’s DNS TXT entry with this new value.
Step 4: Validate the DNS Token/Challenge
Before proceeding, you should validate that the hostname DNS TXT file is visible. You can use your local terminal to perform a DIG and/or you can also use an online tool:
- https://www.whatsmydns.net/#TXT/_acme-challenge.cdn.example.com
- https://toolbox.googleapps.com/apps/dig/#TXT/
Example Output:
As noted above, it is very important that you did not already click to validate your Let’s Encrypt certificate already. If you did, Let’s Encrypt will make 1 more attempt before abandoning the verification step. If the TXT file is not visible, you will see a failed authorization procedure error.
Step 5: Extract Certificate and Key from Certbot Server
Once you have successfully validated the DNS challenge process, Let’s Encrypt will send your Certbot instance the certificate and private key for the hostname(s). There are two files that will be generated:
- Fullchain.pem > contains the public certificate and the Let’s Encrypt intermediate certificate
- Privkey.pem > contains the certificate private key
You need to download the certificates in order to re-upload them to Google Cloud. If you are unable to download the certificates, you can view the certificates in your terminal window and recreate the certificates locally. Because we are dealing with the hostname private keys, you should treat the customer keys with great care and immediately delete the keys after uploading them to GCP.
~$ sudo nano /etc/letsencrypt/live/cdn.example.com/fullchain.pem~$ sudo nano /etc/letsencrypt/live/cdn.example.com/privkey.pem
Save the two pem files separately — do not combine them into a single pem bundle. You will use both files in the next step.
Some people may get nervous downloading a private key to a local machine and want to limit the number of places where a private key may reside. It is possible to install the Google command line tools, gcloud, on the VM where you have certbot installed — give this VM Google Cloud compute API privileges and just upload it without moving keys to the local workstation. This process is beyond the scope of this document, but know that it is an option.
Step 6: Upload Customer Certificate
The easiest way to upload the certificate is to take the fullchain.pem and privkey.pem bundles and upload them via gcloud commands. Log into your Google Cloud console. In the upper right hand corner of the navigation menu, click on the Cloud Shell icon to launch Cloud Shell.
Navigate to the project you are looking to upload the customer certificate. If you selected the project in the Google Cloud console and then launch the Cloud Shell application, it should default to the current project. If you aren’t sure, you need to set your project before upload the customer certificate.
List all projects in your organization that you have access to:
gcloud projects listExample output:
+----------------+-----------------+----------------+
| PROJECT_ID | NAME | PROJECT_NUMBER |
+----------------+-----------------+----------------+
| covid-id-123 | covid-emergency | 4270496XXXX |
| prueba-perf | prueba-perf | 3220824XXXX |
| test-job-2a-0 | test-job-2a-0 | 72148XXXXXX |
+----------------+-----------------+----------------+Take the name of the Project ID value and run the following command to set the Cloud Shell project:
gcloud config set project [Project_ID]Example:
gcloud config set project covid-id-123Once you have set your Cloud Shell environment to the correct project you want to upload the SSL certificate to, execute the following command to upload the certificate.
Format:
gcloud compute ssl-certificates create certificate-name \
--certificate=certificate-file \
--private-key=private-key-file \
--globalExample:
gcloud compute ssl-certificates create cdn-example-letsencrypt \
--certificate=fullchain.pem \
--private-key=privkey.pem \
--globalStep 7: Add Let’s Encrypt Certificate to GCLB HTTPS Frontend
From within the Google Cloud Console, navigate to the Load Balancer section. From the left hand navigation in the upper left navigation, select the three line icon, Networking > Network services > Load balancing
Click on the load balancer instance that contains the property that you are looking to add the Let’s Encrypt certificate on.
Next, click on the Edit icon
Click on the Frontend configuration and the HTTPS instance that you want to upload the Let’s Encrypt certificate to.
Click “Add certificate” button. From the select menu, select the Let’s Encrypt certificate that you uploaded in Step 6.
Step 8: Add a Google Managed Certificate
The Let’s Encrypt certificate is a completely adequate certificate to support HTTPS delivery; however, it is not a managed certificate that will automatically renew. By deploying a Google managed SSL certificate, Google will take on the responsibility to renew and rotate your SSL certificate.
You can request a certificate be generated via the Google Cloud console or via command line. For the purposes of brevity, this document only discusses provisioning a Google managed SSL certificate via the Cloud console. From the Google Cloud console, you can request a certificate via the Frontend configuration tool of the Load balancer.
Click “Add certificate” button just as you did in Step 7. From the select menu, select the Create a new certificate. A new configuration panel will unfurl and you will have an opportunity to request a Google managed certificate.
Click the “Create” button.
In the “select a certificate” select menu, select the hostname value for the Google managed certificate you just requested.
Click “Done” on the Frontend configuration for the HTTPS frontend.
If you are using both IPv4 and IPv6 addressing, you need to repeat the step of adding the Google managed SSL certificate to the alternate HTTPS Frontend configuration.
Click “Update” to deploy the updated Load balancing configuration.
Step 9: Change Hostname DNS Record
Once the new load balancing configuration has been deployed, you can test the configuration by spoofing the hostname in your local HOSTS file to the Google GCLB IP address listed on your Frontend configuration within the Google Load balancer. Once you have changed your local HOSTS file, you should be able to load the target hostname in your browser where you can verify that it is now using the new Let’s Encrypt certificate. It can take a few minutes for the updated Load balancing configuration to take root.
Once satisfied with the new certificate on the target hostname, you can modify the A record in your DNS resource record to point to the Google IP address.
Once Google’s Trust Service sees that the hostname is pointed to a Google IP address, Google is able to provision a Google managed SSL certificate. Google will automatically process and deploy the Google managed SSL certificate for you.
Step 10: Remove the Let’s Encrypt SSL Certificate
Now that your hostname is being services by the Google Load balancer and the Google managed SSL certificate has been deployed, you can remove the Let’s Encrypt bridge certificate.
First, verify that the Google managed SSL certificate has been successfully deployed. From the Google Cloud console, there is a visual cue that will indicate to you that the Google managed SSL certificate has been deployed.
Navigate to the Load balancing section of the Cloud console: Networking services > Load balancing > [select the target load balancer]
Look for the GREEN checkbox next to the Fully Qualified Domain Name, FQDN, value- that is you visual indicator that Google has issued and deployed the Google managed SSL certificate.
You can also click on the FQDN value which will provide the details on the certificate. In particular, you want to look at the Domain status line which will indicate whether or not the certificate has been issued and deployed.
From the certificate details page, you can click on the “Delete” button to remove the certificate from the GCLB. Be certain that the Google Trust Services certificate is live.
Now that you have removed the Let’s Encrypt certificate from your load balancer instance, you must click “Update” to commit the modification to your GCLB instance.
Using GCLOUD Commands
First, navigate to your project in the Cloud Shell terminal. See Step 6 for directions on setting the project in your Cloud Shell terminal.
Get a list of all SSL certificates deployed on your project.
gcloud compute ssl-certificates list — globalExample output:
+-------------------+-------------------------------+
| NAME | CREATION_TIMESTAMP |
+-------------------+-------------------------------+
| us-qu-example-com | 2020-02-20T13:55:58.127-08:00 |
| www-example-com | 2019-12-27T06:38:05.448-08:00 |
+-------------------+-------------------------------+Get the name of your GCLB instance that has the Let’s Encrypt certificate bound to the HTTPS target
gcloud compute target-https-proxies listExample output:
Next we want to remove the temporary Let’s Encrypt certificate from the load balancer. To do this, we need to update the HTTPS Load balancer with JUST the names of the SSL certificates you want to have associated with the Load balancer going forward.
Format:
gcloud compute target-https-proxies update [target-proxy-name] \
--global \
--ssl-certificates=[certificate-list] \
--global-ssl-certificatesWhere the [target-proxy-name] is the value of the NAME from the target-https-proxies list command you just executed. The [certificate-list] value(s) come from the NAME command when you performed the ssl-certificates list gcloud command.
Example:
gcloud compute target-https-proxies update www-example-com-cdn-target-proxy-7 \
--global \
--ssl-certificates=cdn-prueba-com \
--global-ssl-certificatesNOTE: in my example gcloud command, I am only sending the name of the SSL certificate that I want to keep rather than the SSL certificate I are trying to remove from the Load balancer.
Now that we have disassociated the temporary SSL certificate from the Load balancer instance, we now need to delete the SSL certificate from our project. If you execute the gcloud compute ssl-certificates list --global command, you will see that the only SSL certificate we just removed from the load balancer is still present in your project.
To remove the temporary SSL certificate from your project, run the following gcloud command:
Format:
gcloud compute ssl-certificates delete [ssl-certificate-name]Example Output:
gcloud compute ssl-certificates delete us-qu-prueba-com
The following ssl certificates will be deleted:- [us-qu-prueba-com]Do you want to continue (Y/n)? YDeleted [https://www.googleapis.com/compute/v1/projects/sandboxproject/global/sslCertificates/us-qu-prueba-com].
You will be prompted with a confirmation Yes | No question that you will need to answer in order to proceed with the SSL certificate removal. You will get a confirmation message if the SSL certificate was successfully deleted from your project. If you attempt to delete a SSL certificate that is active on a Load balancer instance, you will not be able to delete the SSL certificate until you disassociate the SSL certificate from all Load balancer instances.
To verify that the SSL certificate was removed, you can execute the gcloud compute ssl-certificates list --global command to validate that the certificate was removed. The old SSL certificate should no longer be listed when you run the command.
gcloud compute ssl-certificates list --globalExample output:
+-----------------+-------------------------------+
| NAME | CREATION_TIMESTAMP |
+-----------------+-------------------------------+
| www-example-com | 2019-12-27T06:38:05.448-08:00 |
| www-prueba-com | 2019-10-15T03:24:05.798-08:00 |
| cdn-prueba-com | 2019-11-20T02:15:05.359-08:00 |
+-----------------+-------------------------------+If you have questions about Google Cloud and Cloud CDN, contact your Google Cloud Sales team or reach out to me via the Google Cloud Community Slack channel and post a note in the #cloud-cdn channel.
