Using Hashicorp Vault as a Certificate issuer in Cert Manager

Configure vault PKI backend as a certificate provider in Cert Manager

In my previous article, I’ve explained how to use let’s encrypt as a certificate issuer. in this article, we will be understanding how to use Hashicorp vault as a certificate issuer using vault PKI and cert-manager. The PKI secrets engine generates dynamic X.509 certificates. Using this vault PKI backend one can get certificates without going through the usual manual process of generating a private key and CSR, submitting to a CA, and waiting for a verification and signing process to complete. Certificates are usually issued within seconds. In this article, we will understand how we can create our Root certificate and use the certificate for signing the certificate requests from the cert-manager.

Image Credits: Google Images

What is the entire story all about? (TLDR)

1. Creating our own Root CA.
2. Create a certificate from Cert Manager using Vault PKI as the certificate issuer.

Prerequisites

1. A Kubernetes Cluster ( AKS, GKE, Kind, Local cluster ).

Story Resources

1. GitHub Link: https://github.com/pavan-kumar-99/medium-manifests
2. GitHub Branch: cert-manager-vault

Installing Hashicorp vault using Helm Chart

We will install the official helm chart for the vault and unseal it manually. However, this is not the Ideal way for running a vault in Production. You may want to unseal the vault using a KMS Key ( If being installed in AWS ) or Google KMS key ( If being installed in GCP ).

Helm Install PKI Vault

Once the helm chart is installed, you should find a Stateful set created with the name vault and also a pod named vault-0 created. However the pod is not in a ready state, it is because the vault is not unsealed yet. Let us now unseal the vault by exec-ing into the Pod.

vault_pki_unseal.sh

Let us understand what the init command does

vault operator init -key-shares=1 -key-threshold=1

-key-shares=Number of key shares to split the generated master key into. By default it is 5 and 3 keys are needed to unseal the vault

-key-threshold=Number of key shares required to reconstruct the master key. This must be less than or equal to -key-shares. 3 by default.

Generating Root CA and Intermediate CA

Vault PKI backend can generate its own self-signed CA. But let us create our own CA then upload it to the vault PKI backed for signing our certificates.

We now have our certificates generated, let us now configure our vault PKI backend to add these certificates as the CA. We will also have to create the vault roles and the vault issuing and CRL URLs.

Configure Vault’s PKI Backend

Let us understand the commands one by one

vault login $VAULT_ROOT_TOKEN = Used to login to vault with the root token.

vault secrets enable pki = Used to enable the vault pki backend.

vault secrets tune -max-lease-ttl=8760h pki = We configure the ttl of the pki backend to be 8760 hours.

vault write pki/config/ca pem_bundle=@pem_bundle ttl=8760h => We are writing the CA certificate to vault and setting its ttl to 8760h

Now we have our PKI certificate uploaded to the vault. Let us now create a Vault PKI role. The role definition sets the conditions under which a certificate can be generated.

Once we have the role created, we should configure the endpoints from which the certificates can be issued and revoked.

Let us now create a vault policy for the corresponding vault PKI role.

Enabling Kubernetes Auth Method

The Kubernetes auth method can be used to authenticate with the vault using the Kubernetes service account token. This will help the vault to inject the vault token into the Kubernetes Pod. For this let us enable the vault Kubernetes backend by exec into the pod, since the token_reviewer_jwt has to be passed from the vault pod.

We now have a Kubernetes auth role created. This role is bound to the service account names issuer and namespace named default and the policy named betttercallpavan_pki which was created earlier.

Installing Cert-Manager Helm Chart

Let us know Install the cert-manager’s helm chart on our cluster.

$ helm repo add jetstack https://charts.jetstack.io
$ helm repo add jetstack https://charts.jetstack.io
$ helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.2.0 \
  --create-namespace \
  --set installCRDs=true

Image Credits: Google Images

Generating Certificate

Let us now create the issuer. Issuers, and ClusterIssuers, are Kubernetes resources that represent certificate authorities (CAs) that can generate signed certificates by honoring certificate signing requests. All cert-manager certificates require a referenced issuer that is in a ready condition to attempt to honor the request. An Issuer is a namespaced resource, and it is not possible to issue certificates from Issuer in a different namespace.

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cert-manager-vault
$ cd medium-manifests
$ kubectl apply -f vault-pki-issuer.yaml

And now create the cert

$ kubectl apply -f vault-issuer-cert.yaml

And now once the certificate request is sent, you should see the K8s secret to be created in the namespace where the Certificate Resource is created. And now the cert is ready to be used by any client/server.

$ kubectl get secret bettercallpavan-com-tls -n default

We will now deploy a sample nginx application, to verify our certificate.

$ kubectl apply -f app.yaml
$ ip=$(k get ing certmanagerapp-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
$ echo | openssl s_client -connect $ip:443 2>/dev/null | openssl x509 -noout -issuer
issuer=O = Acme Co, CN = www.bettercallpavan.com

Surprised?

Yes, these are all the steps that are needed to use Vault as a Certificate issuer in Cert Manager. I hope you have liked this article. Feel free to share your thoughts and your experiences working with Hashicorp Vault PKI in the comments section. In case you face any issues during the deployment, please raise an issue here or feel free to reach me on my E-mail ( pavan1999.kumar@gmail.com ).

Cleanup

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cert-manager-vault
$ cd medium-manifests
$ kubectl delete -f ./*
$ kubectl delete secret bettercallpavan-com-tls -n default

Conclusion

1. Cert-Manager can be used to get free SSL certificates valid for 90 days.
2. These certificates can be automatically attempted to be renewed within one month of expiration.

Here are some of my other articles that may interest you

Until next time…..

Recommended

Deep Dive into Cortex Metrics — Part I

Running Apache Spark on EKS with AWS Spot Instances

Deep Dive into Thanos-Part I

Creating Self Hosted GitHub runners in a Kubernetes Cluster