Free SSL Certificates With Let’s Encrypt for Grafana & Prometheus-Operator Helm Charts

Grafana is the most popular web-based tool for creating dashboard and alerts. Tools like Kubernetes and Helm have made it possible to deploy Grafana with only a few commands if have the existing infrastructure for launching Docker containers. However, the documentation and tutorials for setting up Grafana almost always skip over how to secure the web-based front end using SSL encryption.

Before Let’s Encrypt, it was tedious and expensive to add SSL encryption to a website. Let’s Encrypt’s APIs have made it simple (automated) and free to deploy SSL encryption, so there is no reason not to SSL encrypt your web-based applications, even if you are just in the testing phase.

In this post, I will show you how to setup Free SSL certificates with automated renewals using cert-manager and Let’s Encrypt for Grafana deployed on Kubernetes using Helm (version 3).

This example will use the NGINX Ingress Controller for the Kubernetes Ingress resource. I will not be going into very much detail in this post on how to configure the NGINX Ingress Controller, you can find those details here. This deploying SSL certificates with cert-manager doesn’t require any specific configuration for the NGINX Ingress Controller, so we can just use the default Helm Chart values.

First, create namespace for the deploying nginx-ingress Helm Chart:

kubectl create namespace nginx-ingress

Then install the nginx-ingress Helm Chart:

helm install nginx-ingress stable/nginx-ingress --namespace nginx-ingress

If everything was done correctly, when you get the services for the nginx-ingress namespace, you should see the EXTERNAL-IP address listed for the nginx-ingess-contoller service.

$ kubectl -n nginx-ingress get services
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
nginx-ingress-controller        LoadBalancer   10.233.51.219   XXX.XXX.XXX.XXX   80:30816/TCP,443:31905/TCP   49s
nginx-ingress-default-backend   ClusterIP      10.233.27.227   <none>           80/TCP                       49s

I highly recommend creating a hostname for Grafana before installing any additional Helm Charts. See your domain registrar’s documentation for how to do this. In this post I will use pretend that I own the example.com domain name and that I created the grafana.example.com hostname and mapped it to the EXTERNAL-IP address from the above step. A hostname for the SSL certificate will be necessary when configuring the prometheus-operator Helm Chart.

After Nginx Ingress is running, we need to install cert-manager in order to issue Let’s Encrypt SSL certificates. Click here for full cert-manager install instructions. The following instructions will cover installing cert-manager version 0.15.2 with the Let’s Encrypt ClusterIssuer.

First, create a Kubernetes namespace for deploying the cert-manager Helm Chart:

kubectl create namespace cert-manager

Add the cert-manager Helm repository:

helm repo add jetstack https://charts.jetstack.io
helm repo update

Install the cert-manager Helm Chart with the default values:

helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.2  --set installCRDs=true

Verify that cert-manager has started and the status of the pods is Running:

$ kubectl -n cert-manager get po
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-749df5b4f8-r5tqm              1/1     Running   0          49s
cert-manager-cainjector-67b7c65dff-t6hmj   1/1     Running   0          49s
cert-manager-webhook-7d5d8f856b-rccxp      1/1     Running   0          49s

Once cert-manager is running, we need to create the ClusterIssuer resource for Let’s Encrypt. Create a file named cluster-issuer-prod.yaml and add the following, remember to replace your-email@addresss.com with your E-mail address:

---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: your-email@address.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: letsencrypt-prod
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
      - http01:
          ingress:
            class: nginx

If you are testing you may want to use the staging server https://acme-staging-v02.api.letsencrypt.org/directory instead of the production server so you don’t run into any of the certificate request limitations when you are ready for production. Let’s Encrypt’s staging server doesn’t have the same API limitations as the production server, but it doesn’t issue valid SSL certificates.

Finally, apply the Let’s Encrypt ClusterIssuer:

kubectl apply -f cluster-issuer-prod.yaml

Now cert-manager is ready to start issuing Let’s Encrypt SSL certificates for any NGINX Ingress Controller backed ingress resources on your Kubernetes Cluster.

Now we will install the Grafana or Prometheus-Operator Helm Chart. This example will install the Prometheus-Operator Helm Chart, but the values.yml file for both the Grafana portions are the same.

Again the first step is to create a Kubernetes namespace for deploying the prometheus-operator Helm Chart:

kubectl create namespace prom

Now we need to configure the ingress values for Grafana in the Helm Chart’s values.yml. The complete configuration options are available on the prometheus-operator Helm Chart GitHub repository.

For the annotations, we want to specify that nginx is used for the ingress and that letsencrypt-prod is used for the cluster-issuer. Then we want to specify the host to use for the ingress and the tls host (SSL certificate common name), in most cases they will be the same. Below is an example of the grafana portion of my values.yml for the prometheus-operator Helm Chart:

## Using default values from https://github.com/helm/charts/blob/master/stable/grafana/values.yaml
##
grafana:
  enabled: true

  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
    hosts:
      - grafana.example.com
    tls:
      - hosts:
          - grafana.example.com
        secretName: grafana-tls

We can install the prometheus-operator Helm Chart once the values.yml file has been configured:

helm install prom stable/prometheus-operator -f values.yaml --namespace prom

Verify the prom-grafana pod is running (it make take a few minutes to get running):

$ kubectl -n prom get po
NAME                                                 READY   STATUS    RESTARTS   AGE
prom-grafana-798b7b89bf-rnbpt                        2/2     Running   0          10s
prom-kube-state-metrics-568dc84666-z5vm6             1/1     Running   0          10s
prom-prometheus-node-exporter-88h4k                  1/1     Running   0          10s
prom-prometheus-operator-operator-67d764bff6-j99jm   2/2     Running   0          56s
prometheus-prom-prometheus-operator-prometheus-0     3/3     Running   0          56s

Finally, you can view the status of the grafana-tls certificate. The ca.crt will be 0 bytes, but the tls.crt and tls.key should be greater than 0 bytes. If there is an error, the error message should show up here:

$ kubectl -n prom describe secret grafana-tls
Name:         grafana-tls
Namespace:    prom
Labels:       <none>
Annotations:  cert-manager.io/alt-names: grafana.example.com
              cert-manager.io/certificate-name: grafana-tls
              cert-manager.io/common-name: grafana.example.com
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-prod
              cert-manager.io/uri-sans:
Type:  kubernetes.io/tls
Data
====
ca.crt:   0 bytes
tls.crt:  3574 bytes
tls.key:  1671 bytes

Now you can go to https://grafana.example.com and it should be secured by a free valid SSL certficiate. That is it. As long as the cert-manager is running you don’t have to do anything, ever! cert-manager should automatically renew Let’s Encrypt SSL certificates without any user interaction every month.