Istio + cert-manager + Let’s Encrypt demystified

15 min readApr 25, 2019

Today the concept of service mesh is on the rise and when you try Istio, an implementation of this concept, you instantly understand why. This mesh concept shines especially when you have a microservice approach. But with all your services, sometimes you want to use a solution like Let’s Encrypt to automate certificate creation. And if you are on a Kubernetes cluster, you will end up finding a Cert-Manager project that can link your cluster to Let’s Encrypt when you need a certificate.

Now you know what to do, and when you use Istio, the documentation will help you achieve what you want. But sometime, what you understand from the web, might not explain much what really behind the curtain (this example). I hope this post to be helpful for you if you want to understand what is going on when you integrate Let’s Encrypt (via cert-manager) with Istio ingress gateway.

Here, I will speak about cert-manager then Istio for you to understand the theory. On the later part, I will give an example for you to follow. It will help you discover while building your own cluster.

Note: As Istio (even in 1.x version) and cert-manager have some moving piece, this post is about Istio 1.1.x (1.1.2 at the time of writing) and cert-manager 0.7.

Cert-Manager

Let’s start by cert-manager. Cert-manager will interact with Let’s Encrypt server and will create a ‘secret’ in Kubernetes containing the validated certificate. To do that, it uses the ACME (Automatic Certificate Management Environment) protocol which has recently been standardized as RFC 8555. This protocol defines how a Certificate Authority (CA) can automate the verification step for domain ownership. The verification in cert-manager with Let’s Encrypt issuer is either done via a DNS check or an HTTP check. Let’s quickly explain those:

DNS check

Cert-manager will start the registration of the domain (I will use `your.domain` here) on Let’s Encrypt server. It gets a token from the Let’s Encrypt response.
Cert-manager will then connect to your DNS server, and add a TXT entry on `_acme-challenge.your.domain` entry. This entry value will be computed from the token and your Let’s Encrypt account id.
Let’s Encrypt will lookup the DNS entry and upon successful check on the value, will issue the certificate
Cert-Manager will query Let’s Encrypt server to get the certificate. The certificate is stored in a `secret` Kubernetes object

HTTP check

Cert-manager will start the registration of the domain (we will use `your.domain` here) on Let’s Encrypt server. It gets a token from the Let’s Encrypt response.
Cert-manager will create a small server to serve a single page ( /.well-known/acme-challenge/{token} ) with a content based on the token and your Let’s Encrypt account id.
Cert-manager will create an `ingress` Kubernetes object on the `your.domain` host
Let’s Encrypt will query the `http://your.domain/.well-known/acme-challenge/{token}` URL — yes it is HTTP and not HTTPS as it is not yet valid(no certificate) — and upon successful check on the value, will issue the certificate
Cert-Manager will query Let’s Encrypt server to get the certificate. The certificate is stored in a `secret` Kubernetes object

Istio integration

To sum up:

Cert-Manager create a certificate for your domain. Istio will need to use it. (well, use the secret the certificate is in)
If you use the DNS challenge, you need to have access to your DNS server. Sometimes this challenge is not possible (your corporate network team is not really happy to let you play with domains or you use a domain not managed by you ( partnership, …) or you use an incompatible DNS server). In this case, you need to use the HTTP challenge.
The HTTP challenge is done via an HTTP call, which is simpler, but Cert-manager creates an ingress object on Kubernetes to route the query. Since we use Istio and as Istio use gateway / virtualservice, it is not as simple as it looks.

The DNS challenge is much easier, but as you imagine, I wrote this post because it is not fit with my use case, and I have to go with the HTTP challenge. So here, I will only speak about this last one.

Istio

Istio v0.8 introduced `gateway` and `virtualservice` object to manage fine-grained setup compare to simple `ingress` object. A simple way to explain this, is to think `gateway` as the external access of the load-balancer (listened port, domain, …, certificate if we want TLS …). The `virtualservice` on its end, describe the internal rules (proxying rule like path selection, service selection, policies execution(CORS, version management, …)).

Virtual service for the HTTP challenge

As I described earlier, cert-manager will create an `ingress` object to let Let’s Encrypt validate the certificate request. Now, What can we do to make this work with Istio?

Well, we can use a not-well-documented feature of Istio that make it convert `ingress` object to `virtualservice` one (controller source code is here). The thing is, Istio is hardcoded to look for a gateway with the name `istio-autogenerated-k8s-ingress` on the `istio-system` namespace (or the namespace Istio is installed in). If so, `ingress` will simply be converted to `virtualservice`.

This gateway can be created during the setup via the helm values here or if you want to have your own specific (options), you can create one. If you rely on the helm setup, the `gateway` will be created to listen to port 80 on every host (hosts: “*”). If you set the `enableHttps` values to true, it will add a listener on port 443 for every host, with a certificate hardcoded in a file ( directory `/etc/istio/ingress-certs/`, file `tls.crt` and `tls.key`) located in the istio-ingressgateway `deployment` (the gateway controller). It means it is not really useful.

Note that the virtual `virtualservice` created by converting `ingress` will not be created as an object in Kubernetes, It will not show if you list the `virtualservice` via `kubectl`.

With this gateway, the HTTP challenge done by the cert-manager will be successful, and the cert-manager will be able to generate the secret containing the certificate.

Gateway for https

With the previous steps, you know how to generate a secret with a domain certificate in it. Now how could you use it on a gateway? Luckily for us, In version 1.1 of Istio, we can enable the ‘Secret Discovery Service’ (SDS) service. This can be enabled with the help of a single helm value: `gateways.istio-ingressgateway.sds.enabled`.

With this, we can create a gateway that references a secret and not a file. The gateway need to simply reference the secret name in the `spec.servers[x].tls.credentialName` key. As the `spec.servers[x].tls.serverCertificate` and the `spec.servers[x].tls.privateKey are mandatory fields, you will need to set a meaningless value in it like “sds”.

Do not forget that the `secret` containing the certificate needs to be in the same namespace as the ingress gateway controller (usually, the istio-system namespace)

That’s it!

The catch

As of now, I hope you understand what is going on in your cluster. Unfortunately, there is still a small issue. If a user simply accesses via http to your services, well it will get a 404 error. It would be great if we could redirect http call to https. And it seems easy to set up as there is an option on the `gateway` object to do that: We can add a “httpsRedirect” option. Nice isn’t it?

Well… no… Unfortunately, setting this option will make the cert-manager stop working as Let’s Encrypt will try to access the well-known endpoint on http, be redirected to https and fail as the https is not set properly (certificate is missing).

To do this redirect, we have to create a service that redirect http to https and create a `virtualservice` binded to the http `gateway` with a single rule: everything is send to our redirect service. The catch is you cannot let the “host” to “*”. You have to set every host you use in the “host” list. Else when cert-manager will create an `ingress` to validate a certificate (renew for instance) you will have a temporary failure on the redirect.

Let’s implements this

This part is a simple step-by-step guide of what I explained above. The goal is from an empty Kubernetes cluster to be able to have some working HTTPS service with a certificate delivered by cert-manager.

We will deploy 3 services on 2 certificates:

The `helloworld` and the `httpbin` sample on their own domain but in a single certificate
The Istio `bookinfo` sample on its own domain, with its own certificate

We can enable in the Istio helm chart some dependencies or pre-configured objects. The goal here is to not use them to be able to show every single detail.

The required tools are:

A Kubernetes cluster :D — But one that can allocate an external IP on the load-balancers. Also, we will deploy some applications and CPU and memory limits are set on some component. So you will need a pool of servers that give you around 6CPU and 36Gb of memory.
Kubectl
Helm

Tools setup

Let’s start with basic stuff, connect to your Kubernetes cluster with kubectl and check that it is the correct one! disaster occurs so quickly ;)

We will use Helm, so you need to initialize it. Helm can be used as a simple template engine (client side) or as release management (a server in the cluster, the Helm CLI will interact with it). Here we will use the release management way.

Modern Kubernetes clusters have RBAC enabled. To make the Helm server part — named `tiller` — work, you need to create a service account that `tiller` will use to do its work. As we are not really ‘production’ grade, you can set an account with cluster-wide permission. `Tiller` will be deployed in the namespace kube-system, so you need to put the service account there:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
---
EOF

Now you can initialize the helm server part. This will create a service and a deployment named `tiller-deploy`. Helm will use this to manage releases:

$ helm init --service-account tiller --history-max 200

When the `tiller-deploy` pod is up on the kube-system namespace, that’s it, Helm is initialized. We can now setup Istio.

Istio setup

You can basically follow the official guide, but we will use the distributed charts and, on setup, we will set some specifics values.

Add the Istio helm charts repository and update your repository’s local cache. As of today, each Istio version has its own repository. On the URL, you can update the version if you want.

$ helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.1.2/charts/$ helm repo update

2. Install the Istio CRD from the istio-init chart.

$ helm install istio.io/istio-init --name istio-init --namespace istio-system

3. It is time to setup Istio itself. As stated above, you need to enable SDS. Here we will simply use the Istio’s default values plus the set a boolean to enable SDS

$ helm install istio.io/istio \
       --name istio \
       --namespace istio-system \
       --set gateways.istio-ingressgateway.sds.enabled=true

That’s it, you wait for the Istio pods to be ready and you’re good!

Cert-manager

Next step is to set up the cert-manager. It is well described here: https://docs.cert-manager.io/en/latest/getting-started/install.html#installing-with-helm Here I will copy & paste the command to run:

Install the CustomResourceDefinition resources separately

$ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml

Create the namespace for cert-manager

$ kubectl create namespace cert-manager

Label the cert-manager namespace to disable resource validation

$ kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true

Add the Jetstack Helm repository

$ helm repo add jetstack https://charts.jetstack.io

Update your local Helm chart repository cache

$ helm repo update

Install the cert-manager Helm chart

$ helm install \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.7.0 \
  jetstack/cert-manager

Domains names

As stated before, we will create 3 domains, but with 2 certificates. Here we will simply use `xip.io` domain. With the Let’s Encrypt rate limitation (50 certificates per week) it is virtually impossible to have a real certificate on the production. Here, I will use the staging Let’s Encrypt server, it means the certificates will not be valid. The goal here is to validate the process. If you want, you can adapt your domain name to use the production Let’s Encrypt server.

Let’s get your external IP from the Istio ingress gateway:

$ kubectl get svc -n istio-system istio-ingressgateway

The return is

NAME                   TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)             AGE
istio-ingressgateway   LoadBalancer   10.19.240.25   a.b.c.d   80:31380/TCP,...

The external IP is `a.b.c.d`. From that IP, you can define 3 domains name:

hello.a.b.c.d.xip.io for the helloworld sample
httpbin.a.b.c.d.xip.io for the httpbin sample
book.a.b.c.d.xip.io for the bookinfo sample

If you want to use your own domains, you can create them on your DNS server. Note that those domains need to be public! (Let’s Encrypt need to resolve them)

Prerequisite for certificate generation

Earlier, I describe the way cert-manager will create certificates and what is needed on the Istio side: an issuer for cert-manager, and a gateway that will transform `ingress` object.

The issuer part is described here. As Istio need to have certificates in its namespace (istio-system) you can create a simple issuer in the Istio namespace and not a cluster-wide one. For the configuration, we set the challenge type to http and set Let’s Encrypt server to staging.

$ cat <<EOF | kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: istio-system
spec:
  acme:
    email: your@email.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: example-issuer-account-key
    http01: {}
---
EOF

After a small amount of time, you should have an `issuer` up and running:

$ kubectl describe issuer -n istio-system...
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready

As we know this issuer will create `ingress` object, you need to create the Istio gateway that will do the transform. It is a simple gateway with a specific name (“istio-autogenerated-k8s-ingress”) listening everything on port 80:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-autogenerated-k8s-ingress
  namespace: istio-system
  labels:
    app: ingressgateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      protocol: HTTP2
      name: http
    hosts:
    - "*"
---
EOF

All is ready to create some certificates!

Generating certificates

Generating certificate is now as simple as creating an object `certificate` in the `istio-system` namespace.

Here we will create 2 certificates: 1 multi-domains, and 1 single domain to show you what is possible to do.

The `certificate` object needs to reference the issuer (“letsencrypt-staging” created before) and the ingress controller (Istio look for “istio” ingress class so let use that)

You will also need to define a `secretName` to store the pending/validated certificate.

For the multi-domains certificate, we will use the secret named `test-certificate`:

cat <<EOF | kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: test-certificate
  namespace: istio-system
spec:
  secretName: test-certificate
  issuerRef:
    name: letsencrypt-staging
  commonName: hello.a.b.c.d.xip.io
  dnsNames:
  - hello.a.b.c.d.xip.io
  - httpbin.a.b.c.d.xip.io
  acme:
    config:
    - http01:
        ingressClass: istio
      domains:
      - hello.a.b.c.d.xip.io
      - httpbin.a.b.c.d.xip.io
---
EOF

and for the book info sample, we will use the `book-certificate`:

cat <<EOF | kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: book-certificate
  namespace: istio-system
spec:
  secretName: book-certificate
  issuerRef:
    name: letsencrypt-staging
  commonName: book.a.b.c.d.xip.io
  dnsNames:
  - book.a.b.c.d.xip.io
  acme:
    config:
    - http01:
        ingressClass: istio
      domains:
      - book.a.b.c.d.xip.io
---
EOF

To check when the certificate will be ready, you can `describe` the certificates:

$ kubectl describe certificate -n istio-system
...    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
...

Done, certificates are generated!

Creating a gateway for https

Now that your certificates are there, you can create your Istio gateway. As a gateway can only take one certificate, you will need to create 2 gateways. We will create them in the Istio namespace as the configuration state it is the best practice to keep the gateway controller and the gateway in the same namespace.

Let’s create a `test-gateway` with the `test-certificate`:

cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: test-gateway
  namespace: istio-system
  labels:
    app: ingressgateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      protocol: HTTPS
      name: https-default
    tls:
      mode: SIMPLE
      serverCertificate: "sds"
      privateKey: "sds"
      credentialName: "test-certificate"
    hosts:
    - "*"
---
EOF

And the second gateway for the book info. Here you have to set the certificate and set a ‘hosts’ so that Istio will be able to use the specific gateway from the domain name:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: book-gateway
  namespace: istio-system
  labels:
    app: ingressgateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      protocol: HTTPS
      name: https-default
    tls:
      mode: SIMPLE
      serverCertificate: "sds"
      privateKey: "sds"
      credentialName: "book-certificate"
    hosts:
    - "book.a.b.c.d.xip.io"
---
EOF

You can use `openssl` client to check if everything is good:

$ openssl s_client -connect hello.a.b.c.s.xip.io:443
...
---
Certificate chain
 0 s:CN = hello.a.b.c.d.xip.io
   i:CN = Fake LE Intermediate X1
 1 s:CN = Fake LE Intermediate X1
   i:CN = Fake LE Root X1
---
...$ openssl s_client -connect book.a.b.c.s.xip.io:443
...
---
Certificate chain
 0 s:CN = book.a.b.c.d.xip.io
   i:CN = Fake LE Intermediate X1
 1 s:CN = Fake LE Intermediate X1
   i:CN = Fake LE Root X1
---
...

Now you can deploy the apps.

Deploying sample — overview

For each application, we will have the same process:

Creating a namespace for the application
Set the Istio label on the namespace to put your application in the mesh
Deploy the application (`deployment` and `service` object)
Deploy the `virtualservice` to bind your service to a gateway.

Deploying helloworld sample

As the overview state, you create some objects:

$ kubectl create ns hello
namespace/hello created$ kubectl label ns hello istio-injection=enabled
namespace/hello labeled$ kubectl apply -n hello -f https://raw.githubusercontent.com/istio/istio/1.1.2/samples/helloworld/helloworld.yaml
service/helloworld created
deployment.extensions/helloworld-v1 created
deployment.extensions/helloworld-v2 created$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: helloworld
  namespace: hello
spec:
  hosts:
  - "hello.a.b.c.d.xip.io"
  gateways:
  - istio-system/test-gateway
  http:
  - match:
    - uri:
        exact: /hello
    route:
    - destination:
        host: helloworld
        port:
          number: 5000
---
EOF

You can access the https://hello.a.b.c.d.xip.io/hello endpoint (validate the invalid certificate if you used staging Let’s Encrypt). One down, two to go!

Deploying httpbin sample

You can use the same process as helloworld:

$ kubectl create ns httpbin
namespace/httpbin created$ kubectl label ns httpbin istio-injection=enabled
namespace/httpbin labeled$ kubectl apply -n httpbin -f https://raw.githubusercontent.com/istio/istio/1.1.2/samples/httpbin/httpbin.yaml
service/httpbin created
deployment.extensions/httpbin created$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
  namespace: httpbin
spec:
  hosts:
  - "httpbin.a.b.c.d.xip.io"
  gateways:
  - istio-system/test-gateway
  http:
  - match:
    route:
    - destination:
        host: httpbin
        port:
          number: 8000
---
EOF

You can check it is working: https://httpbin.a.b.c.d.xip.io/status/418 it should show you a teapot.

Deploying the book info sample

The change from the previous deployment is the use of the other gateway to use the correct certificate.

$ kubectl create ns book
namespace/book created$ kubectl label ns book istio-injection=enabled
namespace/book labeled$ kubectl apply -n book -f https://raw.githubusercontent.com/istio/istio/1.1.2/samples/bookinfo/platform/kube/bookinfo.yaml
service/details created
deployment.extensions/details-v1 created
service/ratings created
deployment.extensions/ratings-v1 created
service/reviews created
deployment.extensions/reviews-v1 created
deployment.extensions/reviews-v2 created
deployment.extensions/reviews-v3 created
service/productpage created
deployment.extensions/productpage-v1 created$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
  namespace: book
spec:
  hosts:
  - "book.a.b.c.d.xip.io"
  gateways:
  - istio-system/book-gateway
  http:
  - match:
    - uri:
        exact: /productpage
    - uri:
        exact: /login
    - uri:
        exact: /logout
    - uri:
        prefix: /api/v1/products
    route:
    - destination:
        host: productpage
        port:
          number: 9080
---
EOF

That’s it. You can test https://book.a.b.c.d.xip.io/productpage

HTTPS redirect

Your services are working. Yeahhhhh!!!!! except for the first guy that will try to access those services. He will tell you that he gets a 404 status.

Yes, he did try to access with http and not https… Why bothering put ‘https://’ as it is almost automatic everywhere (The key word is ‘almost’ this example demonstrate why;) )

As stated before, we cannot set the automatic redirect to the http gateway. You need to create a basic redirect. Here, I choose to deploy a simple nginx server to do that.

$ kubectl create ns redirect
namespace/redirect created$ kubectl label ns redirect istio-injection=enabled
namespace/redirect labeled$ cat <<EOF | kubectl apply -n redirect -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    server {
      listen 80 default_server;
      server_name _;
      return 301 https://$host$request_uri;
    }
---
apiVersion: v1
kind: Service
metadata:
  name: redirect
  labels:
    app: redirect
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: redirect
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redirect
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redirect
  template:
    metadata:
      labels:
        app: redirect
    spec:
      containers:
      - name: redirect
        image: nginx:stable
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: /etc/nginx/conf.d
          name: config
      volumes:
      - name: config
        configMap:
          name: nginx-config
---
EOF

The last step is to bind the service to the HTTP gateway. I set all the hosts I use as I want to be sure that, at certificate renew time, I will not have some break on the redirect.

$ cat <<EOF | kubectl apply -n redirect -f -
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: redirect
spec:
  hosts:
  - "book.a.b.c.d.xip.io"
  - "test.a.b.c.d.xip.io"
  - "httpbin.a.b.c.d.xip.io"
  gateways:
  - istio-system/istio-autogenerated-k8s-ingress
  http:
  - route:
    - destination:
        host: redirect
        port:
          number: 80
---
EOF

Using an URL like http://book.a.b.c.d.xip.io/productpage now redirect to https.

Conclusion

I hope with this post and especially with the example, you now understand the small thing that makes all this work: The hardcoded magic gateway on Istio, The not-default option for SDS, the small tricks (well if you want to use them) for instance the one to redirect to https endpoint. I hope you learned one thing or two, and that makes your day better.

I will end this post by thanking you, reader, for reading this.