Strimzi Kafka with Traefik Ingress and a custom CA

Introduction

When deploying Apache Kafka on Kubernetes, Strimzi makes it easy to configure Kafka listeners with TLS encryption. But, with default settings, these listeners will only use certificates signed by the internal self-signed certification authority generated by Strimzi.

Fortunately, you can create and deploy your own SSL certificates using cert-manager. You can use any provider supported by it, even Let’s Encrypt. I’m using, however, our internal HashiCorp Vault as the root CA.

The problem

I’ve set up Strimzi Kafka multiple times but always (when external access was required) using Nginx Ingress. My current set up uses Traefik for multiple reasons and I didn’t feel the need to deploy a new Load Balancer for Kafka only.

I found that Traefik let me down a little bit here as the configuration is far more complex than if I were using Nginx.

My first problem was that Strimzi requires enabling SSL passthrough. In Nginx, this is very simple, you just annotate the Ingress with

nginx.ingress.kubernetes.io/ssl-passthrough: "true"

But in Traefik, as far as I could find from the documentation, you can only do it in the IngressRouteTCP options:

Solution

The solution turned out to be quite complex and a minor error in my config set me back a couple of hours. I hope this blog helps you if you’re trying to do the same.

Certificate

Firstly, ask cert-manager to create the SSL certificate.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kafka-tls
spec:
  commonName: broker-bootstrap.example.com
  dnsNames:
    - broker-bootstrap.example.com
    - broker-00.example.com
    - broker-01.example.com
    - broker-02.example.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: vault-issuer
  privateKey:
    size: 4096
  secretName: kafka-tls

You can use instead the certResolvers in Traefik if you have them configured. I opted for this external config instead with cert-manager which I found more flexible.

Cluster Config

You’ll need to add an external listener to the cluster config and advertise the host names. This MUST match your SSL certificate. Also, as we’re using an ingress in port 443, the advertisedPort must reflect this fact.

spec:
  kafka:
    listeners:
      - name: external
        port: 9094
        type: cluster-ip
        tls: true
        configuration:
          bootstrap:
            annotations:
              advertisedHost: broker-bootstrap.example.com
              advertisedPort: 443
          brokers:
            - broker: 0
              advertisedHost: broker-00.example.com
              advertisedPort: 443
            - broker: 1
              advertisedHost: broker-01.example.com
              advertisedPort: 443
            - broker: 2
              advertisedHost: broker-02.example.com
              advertisedPort: 443
          brokerCertChainAndKey:
            secretName: kafka-tls
            certificate: tls.crt
            key: tls.key

IngressRouteTCP

The ingress is not based on Host headers but on the SNI configuration in the certificate.

You’ll need to point each broker to its service. You can get the service names but running the command:

kubectl -n kafka-namespace get svc -owide

With this information, you can craft the ingress configuration:

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: kafka-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI(`broker-bootstrap.example.com`)
      services:
        - name: my-cluster-kafka-external-bootstrap
          port: 9094  
    - match: HostSNI(`broker-00.example.com`)
      services:
        - name: my-cluster-kafka-0
          port: 9094
    - match: HostSNI(`broker-01.example.com`)
      services:
        - name: my-cluster-kafka-1
          port: 9094
    - match: HostSNI(`broker-02.example.com`)
      services:
        - name: my-cluster-kafka-2
          port: 9094
  tls:
    secretName: kafka-tls
    passthrough: true

Testing

You’ll need to create a keystore and truststore . There are plenty of resources on how to do it out there, I used something very similar to this.

EDIT: I was asked to add more info on how to create the truststore and keystore. This is an excerpt from a script I use. Please note this is using the server SSL cert, you should have a client cert and not use the server one. It breaks all security principles.

#!/bin/bash

k get secret kafka-tls -ojsonpath='{.data.tls\.key}' | base64 --decode > cert.key
k get secret kafka-tls -ojsonpath='{.data.tls\.crt}' | base64 --decode > cert.crt
k get secret kafka-tls -ojsonpath='{.data.ca\.crt}' | base64 --decode > ca.crt

openssl pkcs12 -export -in "cert.crt" -inkey "cert.key" -name client.example.com -out "certs.p12" -passout pass:changeit
keytool -noprompt -importkeystore -deststorepass "changeit" -destkeystore "keystore.jks" -srckeystore "certs.p12" -srcstoretype PKCS12 -srcstorepass "changeit"
keytool -noprompt -import -alias caroot -keystore "keystore.jks" -keyalg RSA -storepass "changeit" -file "ca.crt"
keytool -noprompt -import -alias caroot -keystore "truststore.jks" -keyalg RSA -storepass "changeit" -file ca.crt

You will also need a client.properties like the example below:

security.protocol=SSL
ssl.truststore.location=/opt/ssl/truststore.jks
ssl.truststore.password=changeit
ssl.keystore.location=/opt/ssl/keystore.jks
ssl.keystore.password=changeit

Finally, you can test it with a client:

docker run --rm -ti --name kafka -v --entrypoint=/bin/bash \
  -v /opt/ssl:/opt/ssl quay.io/strimzi/kafka:0.32.0-kafka-3.2.0

# from the container:
/opt/kafka/bin/kafka-topics.sh --list \
  --command-config /opt/ssl/client.properties \
  --bootstrap-server broker-bootstrap.example.com:443

# or to test performance, you can use
/opt/kafka/bin/kafka-producer-perf-test.sh \
  --num-records 1000000 --throughput -1 \
  --record-size 2048 --topic test000 \
  --producer.config /opt/ssl/client.properties \
  --producer-props bootstrap.servers=broker-bootstrap.example.com:443 acks=all

Conclusion

I like Traefik. I’m slowly replacing Nginx with it. But I found this simple config quite burdensome. It would be great if Traefik supported more annotations in the default Ingress definition to configure things like the SSL passthrough.

I hope this helps. If you have any issues, get in touch!