On TLS cipher suites and staying sane and compliant with cert-manager and nginx-ingress

Prepare yourself mentally before reading on. Photo by nicollazzi xiong from Pexels

I am writing this story in order to save you lots of time — assuming you have trouble staying compliant with TLS cipher suites against various governmental and other third-party security frameworks.

Disclaimer: this article is meant for engineers that have little to no experience with networking and cryptography. If you are already face palming — sorry to hear. Maybe you should read some of my other articles to pass the time :)

One such example is the BSI TR-02102–2 from the German government, which at time of this writing accepts the TLS cipher suites listed in “3.3.1.1 (EC)DHE Cipher-Suiten” and “3.4.4 Cipher-Suiten”. And this is the one we will be using as an example in this post. For this purpose specifically you can use a checker like https://www.tls-check.de/.

If you want to check your existing TLS cipher suites you can use a CLI tool nmap, so let’s give it a shot for google.com and see how they fare (I have compressed the response with “..”):

u ~ $ nmap -sV — script ssl-enum-ciphers -p 443 google.com
Starting Nmap 7.92 ( https://nmap.org ) at 2021–11–16 14:16 CET
Nmap scan report for google.com (142.251.36.206)
Host is up (0.016s latency).
rDNS record for 142.251.36.206: muc12s12-in-f14.1e100.net
PORT STATE SERVICE VERSION
443/tcp open ssl/https gws
| ssl-enum-ciphers:
| TLSv1.0:
| ciphers:
| TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (ecdh_x25519) — A
| TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (ecdh_x25519) — A
..
| compressors:
| NULL
| cipher preference: server
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| TLSv1.1:
| ciphers:
| TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (ecdh_x25519) — A
| TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (ecdh_x25519) — A
..
| compressors:
| NULL
| cipher preference: server
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| TLSv1.2:
| ciphers:
| TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (ecdh_x25519) — A
| TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) — A
| compressors:
| NULL
| cipher preference: client
| warnings:
| 64-bit block cipher 3DES vulnerable to SWEET32 attack
| TLSv1.3:
| ciphers:
| TLS_AKE_WITH_AES_128_GCM_SHA256 (ecdh_x25519) — A
| TLS_AKE_WITH_CHACHA20_POLY1305_SHA256 (ecdh_x25519) — A
..

Analyzing this further against our BSI standard you will realize pretty quickly that there are numerous cipher suites that are unsupported simply because they are not on the list of acceptable cipher suites (for various reasons which are unimportant for the sake of this article).

Ok, now let’s single out the *CHACHA* cipher suite for TLS v1.2. What if we want to get rid of it AND it’s a default for cert-manager (it is).

Part 1: cert-manager

The first problem is to figure out who is handling these cipher suites in the first place. My initial assumption was: Well, cert-manager since we are using that to create our TLS certificates with Let’s encrypt as a CA in our k8s clusters.

This was the first rabbit hole. Specifically because of that assumption.

Source: https://www.pinterest.com/pin/688698968007878911/

Then I went on into the deep end in GitHub issues and cert-manager docs:

From the only mention of “cipher suites” on the cert-manager docs: https://cert-manager.io/docs/release-notes/release-notes-0.14/#feature

… to the GitHub issues about its implementation and where the defaults are coming from (spoiler alert they moved from the codebase to using the defaults of the Go package being used in that repo): https://github.com/jetstack/cert-manager/pull/2562

To then trying to manually set this in my cert-manager webhook deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: webhook
    app.kubernetes.io/component: webhook
    app.kubernetes.io/instance: cert-manager
    app.kubernetes.io/name: webhook
  name: cert-manager-webhook
  namespace: cert-manager
# ...
    spec:
      containers:
      - args:
        - --v=2
        - --secure-port=10250
        - --dynamic-serving-ca-secret-namespace=$(POD_NAMESPACE)
        - --dynamic-serving-ca-secret-name=cert-manager-webhook-ca
        - --dynamic-serving-dns-names=cert-manager-webhook,cert-manager-webhook.cert-manager,cert-manager-webhook.cert-manager.svc
        - --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256

Only to find out that none of this will ever work anyway because my TLS is terminated in my nginx-ingress and that is where I need to set this.

Definitely spent 2 days on this. Keep in mind I knew my way around k8s but I simply didn’t have to care about cipher suites before and I assumed the defaults will be good enough.

Part 2: nginx-ingress

Ok. Now it’s as easy as looking at the docs right? No. There are 2 gotchas.

Fun fact: same defaults for nginx-ingress as the cert-manager, so non-compliant by default.

Gotcha 1: the standard command for setting this only works for TLS v1.2

Which of course does not help if you have a non-compliant TLS v1.3 cipher suite but 1 step at a time.

The Docs: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#ssl-ciphers

nginx.ingress.kubernetes.io/ssl-ciphers: |
  “ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP”
nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"

This calls the ssl_ciphers nginx command in the background: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_ciphers

The second annotation ensures that your server defaults will be used instead of the client choosing in which order to use them — nice feature to be extra safe.

Some mini gotchas:

• unlike the cert-manager, here the delimiters are “:” instead of “,”
• You have to remove the “TLS_” in front of each cipher suite
• The names in e.g. the BSI standard do not match what we need to input 1:1 regardless of the “TLS_” removal — there are some subtle differences in the wording you need to be careful with. Otherwise nginx will throw validation errors.

Gotcha 2: the command for TLS v1.3 is different and doesn’t always work

Yes, you heard correctly. This only solves half the problem. The cipher suite we want to get rid of is also a default in TLS v1.3.

The annotation with a configuration snippet uses a command for nginx’ OpenSSL (1.0.2 or higher required, so check what your nginx-ingress is using under the hood): http://nginx.org/en/docs/http/ngx_http_ssl_module.html

annotations:
  nginx.ingress.kubernetes.io/configuration-snippet: |
    ssl_conf_command Ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384;

If the annotation above looks weird on LoC 4 it’s just because it can’t fit on 1 line.

Conclusion

Once you have wrapped your head around it, it all makes sense and it is kind of obvious. Hindsight is however always 20/20, which is why I decided to write this article — to save you some time in the future.

Lastly I sincerely hope that the situation with TLS v1.3 will improve soon because your mileage my very given the restrictions above.