Open in app

Sign in

Write

Sign in

Mutual TLS auth with AWS API Gateway Part 2 - check certificate revocation

Using Lambda Authoriser to validate client certificate against OCSP & CRL

Koustubha Kale
Contino Engineering
Published in
10 min readOct 2, 2020

--

In my last blog we looked at how to implement Mutual TLS auth for an API on AWS API Gateway and how it impacts the current open banking landscape in Australia. But what happens when a certificate is compromised and needs to be revoked? How do we handle this? For this blog let us have a look at how to check for certificate revocation using a Lambda authoriser.

Why do we need this?
Once you set up the truststore with API Gateway, it allows clients with trusted certificates to communicate with the API. But certificates can get revoked any time for a variety of reasons. This can be painful because it will break your API transaction and cause disruption to your users.

At the moment API Gateway does not check for certificate revocation, although I am sure this is a planned feature.

A brief background on certificate revocation
As I said, CAs might revoke a certificate before it’s expiry for reasons such as key compromise or buggy code. We have seen this happen before, e.g. last year and this year. As the certificate is stored/housed at the client end, the CA has no way of modifying or deleting it. But when the certificate is issued, the CA has the option of embedding information about where it publishes revocation information in the certificate itself. So if the certificate needs to be revoked, the CA marks the certificate as revoked in it’s records and publishes revocation information in the form of a Certificate Revocation List (CRL) or via Online Certificate Status Protocol (OCSP) server. And the client and server can both go and confirm that the certificate offered by the other has not been revoked before allowing connection.
You can check the OCSP and/or CRL information (if available) by inspecting a certificate.
Check for OCSP URL:

openssl x509 -in cert.pem -noout -ocsp_urihttp://ocsp.int-x3.letsencrypt.org

Check for CRL:

openssl x509 -in PcaClient.pem -noout -text |grep -i crlX509v3 CRL Distribution Points: 
                  URI:http://{your CA}/crl/d070c239-4dc1-4c93-b6dc-31d46fb2e024.crl

There’s a lot to know about certificate revocation but I am not going into the details in this blog. You might find this article worth reading if you are interested. I want to keep this blog focussed on how to implement revocation checking using a Lambda Authoriser.

What I am trying to achieve
Building on top of part 1, I would like the Lambda authoriser to:

  • keep working with my selfsigned certificates that have no OCSP or CRL information
  • validate certificate revocation for CAs that have OCSP information e.g. Let’s Encrypt issued certs
  • validate certificate revocation for CAs that have CRL information e.g. AWS Certificate Manager Private Certificate Authority issued certs

Given that API Gateway already does the certificate validation against the chain from truststore, as well as certificate expiry check, I want the Lambda authoriser to focus solely on checking certificate revocation. Here is a simple flow I have in mind for this Lambda Authoriser PoC:

Lambda Authoriser Certificate Revocation Check Flow

And here is the solution diagram for this PoC:

PoC architecture for mTLS Cert Revocation Check

Let’s create our three test certificates

1. Self signed cert:
Although we can build a custom solution to store and check revocation of our self signed certificates, in this blog I will keep the focus on checking the standard OCSP and CRL based revocation. So for our self signed certs, all I want to achieve is that the Lambda Authoriser should not reject our self signed certs. To that end, I would like to bring consistency in key usage and extended key usage amongst all three client certificates we are going to use in this PoC. This was deliberately ignored in part 1 for simplicity but we will address it now to reflect what would happen in the real-world.
This is the info I want common amongst all certs issued by our three issuers:

X509v3 extensions:    X509v3 Key Usage: critical        Digital Signature, Key Encipherment    X509v3 Extended Key Usage:        TLS Web Server Authentication, TLS Web Client Authentication

So let’s re-issue the self signed certificate with key usage and extended key usage attributes.

Let’s create a openssl config file config.cnf:

[ req ]distinguished_name = req_distinguished_nameattributes = req_attributesprompt = no[ req_distinguished_name ]commonName = OB mTLS ClientcountryName = AUlocalityName = MelbourneorganizationName = ContinoorganizationalUnitName = Open Banking ClientstateOrProvinceName = Victoria[ req_attributes ][ cert_ext ]basicConstraints=CA:FALSEsubjectKeyIdentifier=hashkeyUsage=critical,digitalSignature,keyEnciphermentextendedKeyUsage=clientAuth,serverAuth

Create a new CSR:

openssl req -config ./config.cnf -new -key MyClient.key -nodes -out new.csr

Create a new self signed cert with key usage and extended key usage attributes (Using the same Root CA we created in part 1):

openssl x509 -extfile ./config.cnf -extensions cert_ext -req -in new.csr -CA RootCA.pem -CAkey RootCA.key -set_serial 01 -out new_crt.pem -days 365 -sha256

Check our new self signed cert has the key usage and extended key usage attributes:

openssl x509 -in new_crt.pem -noout -textcheck for X509v3 extensions: in the output

Check our new self signed cert still works with our API. It should, since it’s signed by the same Root CA of which we had put the public cert in our truststore in part 1:

curl -v --key MyClient.key --cert new_crt.pem https://{your custom domain}/cds-au/v1/banking/accounts -H "Authorization:kk-mtls-selfsigned-1"* Connection #0 to host {your custom domain} left intact
{"data":{"accounts":[{"accountId":"string","creationDate":"string","displayName":"string","nickname":"string","openStatus":"OPEN","isOwned":true,"maskedNumber":"string","productCategory":"TRANS_AND_SAVINGS_ACCOUNTS","productName":"string"}]},"links":{"self":"string","first":"string","prev":"string","next":"string","last":"string"},"meta":{"totalRecords":1,"totalPages":1}}

2. Let’s Encrypt cert:
I chose Let’s Encrypt because a) it’s free b) it has a OCSP service. And certbot makes it really easy to issue and manage certificates.

Certbot is an easy-to-use client that fetches a certificate from Let’s Encrypt — an open certificate authority launched by the EFF, Mozilla, and others.

I chose DNS challenge and AWS Route 53 because it just works for my use case, I don’t need to host a web server which is the other method by which Let’s Encrypt checks domain ownership. certbot has dockerised clients which I found perfect for my use case. You will need Docker installed to use this. You can use any other method that suits you too..

# Create a directory to hold certsmkdir -p /path/to/folder/certs/letsencrypt/etc /path/to/folder/certs/letsencrypt/lib# Run certbot with dockerdocker run -it --rm --name certbot \
-v "/path/to/folder/certs/letsencrypt/etc:/etc/letsencrypt" \
-v "/path/to/folder/certs/letsencrypt/lib:/var/lib/letsencrypt" \
certbot/dns-route53 certonly --manual --preferred-challenges dns

Follow the prompts and you should have the freshly minted certs under `/path/to/folder/certs/letsencrypt/etc/live/{your domain}/`

Also grab the Let’s Encrypt root cert from https://letsencrypt.org/certs/trustid-x3-root.pem.txt

Note: Add an empty line to the end of the trustid-x3-root.pem.txt file. API Gateway truststore has trouble if each cert does not start on a new line.

Add Let’s Encrypt chain.pem & trustid-x3-root.pem to the truststore.pem file we created in part 1.

cat /path/to/folder/certs/letsencrypt/etc/live/{your domain}/chain.pem /path/to/folder/certs/letsencrypt/etc/live/{your domain}/trustid-x3-root.pem.txt >> /path/to/folder/with/truststore.pem

3. AWS Certificate Manager Private Certificate Authority (ACM-PCA) issued cert:

Chose ACM-PCA because a) it’s AWS and b) it’s free for first 30 days c) It has built in CRL functionality even for private certificates.

Please refer to AWS documentation to create ACM-PCA.

Once created, let’s get the ACM-PCA Root certificate & chain (In case you created some intermediaries).

aws acm-pca get-certificate-authority-certificate \
 — certificate-authority-arn {your ACM-PCA arn}\
 — output text > PcaRootChain.pem

Let’s add the ACM-PCA chain to our truststore

cat /path/to/folder/certs/acm_pca/PcaRootChain.pem >> /path/to/folder/with/truststore.pem

Ensure each cert in the truststore.pem file starts on a new line before uploading the cert to our truststore S3 bucket.

aws s3 cp /path/to/folder/with/truststore.pem s3://{your-truststore-s3-bucket}/truststore.pem

Once the file is uploaded to s3, get it’s version Id and update the truststore version in API Gateway Custom domain names > custom domain > domain details > Truststore version. Wait for status to become available.

Note: If there are any errors in truststore.pem file format OR if API Gateway can’t find the full chain of each certificate, it will display a warning.

Click on View Warnings to see which certificate it has trouble with and fix it in truststore.pem then re-upload it to s3.

Update the new s3 version Id in API Gateway and ensure there are no warnings before proceeding.

Create a openssl config file config.cnf:

[ req ]distinguished_name = req_distinguished_nameattributes = req_attributesprompt = no[ req_distinguished_name ]commonName = OB ACM-PCA mTLS ClientcountryName = AUlocalityName = MelbourneorganizationName = ContinoorganizationalUnitName = Open Banking ACM-PCA ClientstateOrProvinceName = Victoria[ req_attributes ][ cert_ext ]basicConstraints=CA:FALSEsubjectKeyIdentifier=hashkeyUsage=critical,digitalSignature,keyEnciphermentextendedKeyUsage=clientAuth,serverAuth

Let’s issue a client cert using our ACM-PCA:

# Create a Client Private Key
openssl genrsa -out PcaClient.key 2048# Create a CSRopenssl req -config ./config.cnf -extensions cert_ext -new -key PcaClient.key -nodes -out PcaClient.csr# Issue a cert using ACM-PCA AWS CLI
aws acm-pca issue-certificate \--certificate-authority-arn {your ACM-PCA arn} \
--csr fileb://PcaClient.csr \
--signing-algorithm "SHA256WITHRSA" \
--validity Value=90,Type="DAYS" \
--idempotency-token 1234

The output of the above command will give you your new certificate’s ARN.

{"CertificateArn": "arn:aws:acm-pca:your-region:{your-aws-account-number}:certificate-authority/d070c239-4dc1-4c93-b6dc-31d46fb2e024/certificate/ad189b77a65372cf5c5584eb804a16e9"}

Use it to retrieve the certificate:

aws acm-pca get-certificate \
--certificate-authority-arn {your ACM-PCA arn} \
--certificate-arn {your certificate arn} \
--output text > PcaClient.pem

Let’s build our Lambda Authoriser:

The Lambda Authoriser leverages certvalidator python library. It’s got excellent documentation along with examples.

As the Lambda Authoriser uses external libraries, you will have to build it as a lambda deployment package. I have created AWS SAM based code to make this easier here.

The Lambda Authoriser’s IAM execution role will need following policies attached:

The contents of allow-truststore-download managed policy should be similar to below to allow the lambda function to download the truststore.pem file from the s3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucketVersions",
                "s3:ListBucket",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::<your truststore s3 bucket name>",
                "arn:aws:s3:::*/*"
            ]
        }
    ]
}

After you have built and deployed the lambda function, attach it as authoriser to the API. Grant API Gateway permission to invoke your Lambda function.

Note the settings highlighted

All right, let’s test:

1. Test our self signed certificate still works:

curl -v --key MyClient.key --cert new_crt.pem https://{your custom domain}/cds-au/v1/banking/accounts -H "Authorization:kk-mtls-selfsigned-1"{"data":{"accounts":[{"accountId":"string","creationDate":"string","displayName":"string","nickname":"string","openStatus":"OPEN","isOwned":true,"maskedNumber":"string","productCategory":"TRANS_AND_SAVINGS_ACCOUNTS","productName":"string"}]},"links":{"self":"string","first":"string","prev":"string","next":"string","last":"string"},"meta":{"totalRecords":1,"totalPages":1}}

2. Test our Let’s Encrypt issued certificate works:

curl -v --key privkey.pem --cert cert.pem https://{your custom domain}/cds-au/v1/banking/accounts -H "Authorization:kk-mtls-letsencrypt-2"{"data":{"accounts":[{"accountId":"string","creationDate":"string","displayName":"string","nickname":"string","openStatus":"OPEN","isOwned":true,"maskedNumber":"string","productCategory":"TRANS_AND_SAVINGS_ACCOUNTS","productName":"string"}]},"links":{"self":"string","first":"string","prev":"string","next":"string","last":"string"},"meta":{"totalRecords":1,"totalPages":1}}

Let’s revoke our Let’s Encrypt cert:

docker run -it — rm — name certbot \
-v “/path/to/folder/certs/letsencrypt/etc:/etc/letsencrypt” \
-v “/path/to/folder/certs/letsencrypt/lib:/var/lib/letsencrypt” \
certbot/dns-route53 revoke — cert-path “/etc/letsencrypt/live/{your client domain}/fullchain.pem” — no-delete-after-revoke — reason cessationofoperation- — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — Congratulations! You have successfully revoked the certificate that was locatedat /etc/letsencrypt/live/{your client domain}/fullchain.pem- — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Now let’s check if openssl says it’s revoked:

openssl ocsp -issuer chain.pem -text -cert cert.pem -url http://ocsp.int-x3.letsencrypt.org -header “Host” “ocsp.int-x3.letsencrypt.org” -no_nonce -verify_other chain.pem— — — — — — — — —Response verify OKcert.pem: revoked— — — — — — — — —

Now let’s check if our Lambda Authoriser detects it as revoked and rejects our API request:

curl -v --key privkey.pem --cert cert.pem https://{your custom domain}/cds-au/v1/banking/accounts -H "Authorization:kk-mtls-letsencrypt-3"* Connection #0 to host {your custom domain} left intact
{"message":"Forbidden"}%

Yay! that proves that our OCSP checking works!!

Now let’s check our ACM-PCA issued cert:

curl -v --key PcaClient.key --cert PcaClient.pem https://{your custom domain}/cds-au/v1/banking/accounts -H "Authorization:kk-mtls-acm-pca-3"{"data":{"accounts":[{"accountId":"string","creationDate":"string","displayName":"string","nickname":"string","openStatus":"OPEN","isOwned":true,"maskedNumber":"string","productCategory":"TRANS_AND_SAVINGS_ACCOUNTS","productName":"string"}]},"links":{"self":"string","first":"string","prev":"string","next":"string","last":"string"},"meta":{"totalRecords":1,"totalPages":1}}

Now let’s revoke our ACM-PCA cert:

# grab the cert serial number
openssl x509 -noout -in PcaClient.pem -serialserial=AD189B77A65372CF5C5584EB804A16E9# Revoke the client certaws acm-pca revoke-certificate --certificate-authority-arn {your ACM-PCA arn} --certificate-serial AD189B77A65372CF5C5584EB804A16E9 --revocation-reason "CESSATION_OF_OPERATION"# This command returns no response on success

Note: You will have to wait for ACM PCA to update the crl file in ACM-PCA created crl s3 bucket. It can take up to 30 mins.

Once you see the timestamp on the crl file change in the s3 bucket, you can test again to see if our Lambda Authoriser is able to pick it up and reject our request:

curl -v --key PcaClient.key --cert PcaClient.pem https://{your custom domain}/cds-au/v1/banking/accounts -H "Authorization:kk-mtls-acm-pca-3"* Connection #0 to host {your custom domain} left intact
{"message":"Forbidden"}%

Yay! That proves our CRL checking works too!!

Hurray! We have successfully implemented certificate revocation checking using a Lambda Authoriser for OCSP and CRL, whilst ensuring our self signed certs with no OCSP or CRL info in them work too. Mind you this is just a basic implementation and there are several optimisations such as caching possible.

Hope you found both these posts about AWS API Gateway Mutual TLS auth support, informative and useful. Please leave your comments and feedback.

AWS
Mutual Tls
Api Gateway
Open Banking
Certificate Revocation

--

--

Koustubha Kale
Contino Engineering

Written by Koustubha Kale

20 Followers
Writer for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams