Why I switched Kong For Traefik

Kong and Traefik are both reverse proxies which can be used to route incoming traffic to all of your services running inside your Kubernetes cluster. I used Kong for a very long time until a comment was placed on my previous Medium post which made me look into other ingress controllers out there.

Traefik

Traefik is written in Go. It is developed by Containous and has an integrated dashboard. It utilizes Kubernetes to store its state and makes use of Ingresses and Secrets to route all traffic to your services over https. Traefik is lightweight and very easy to use and set up. Performance-wise it should not be looked down upon in comparison with nginx. It is already used worldwide in production and heavily tested and benchmarked.

Traefik architecture (source)

containous/traefik

Kong

Kong is written in Lua on top of nginx. It is developed by the company Mashape and has a separate developed dashboard. It is widely used in production and requires a Postgres or Cassandra to store its state. Kong has been around for a while and is a more mature project, it has proven itself as a worthy reverse proxy for any kind of cluster.

Kong/kong

Keeping State

The main difference between the two is how they maintain state. State has to be kept somewhere in order to route traffic and store certificates. Kong does this by using Postgres or Cassandra. I explained in my previous post how I set up a Postgres database in my Kubernetes cluster and how this was used to keep the state of Kong. The main thing with databases is that you need migrations in order to migrate your state into a newer structure. I experienced some hassle with migrations and reading through changelogs to make sure that I knew what would break. With the introduction of Kong 1.0, I needed to split my API into services and routes. This requires a lot of work.

Traefik, on the other hand, uses the ‘Ingress’ resource definition of Kubernetes, it controls the incoming traffic and routes it to a specific service. It can be set up by using a DeamonSet or a Deployment, whichever you prefer. A service needs to be created which will create an AWS LoadBalancer to route all traffic to. You will then have to route all traffic in Route53 to this newly created LoadBalancer in order for Traefik to know what to do.

kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-loadbalancer
  ports:
    - protocol: TCP
      port: 443
      name: ssl
    - protocol: TCP
      port: 80
      name: web
type: LoadBalancer

The Ingresses you need to create are namespace bound, so traffic from those Ingresses has to go to a service within their namespaces. Each Ingress you make will automatically be detected by Traefik. You can add annotations to specify how this Ingress will behave. All configurations can be found on the Traefik documentation page.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: bouwe-ceunen-ingress
  namespace: bouwe-ceunen
  annotations:
    ingress.kubernetes.io/ssl-redirect: 'true'
spec:
  rules:
  - host: bouweceunen.com
    http:
      paths:
      - path: /
        backend:
          serviceName: bouwe-ceunen
          servicePort: http-port
  - host: www.bouweceunen.com
    http:
      paths:
      - path: /
        backend:
          serviceName: bouwe-ceunen
          servicePort: http-port
  tls:
  - secretName: bouwe-ceunen-cert

As you can see, all state is kept in Kubernetes, the way it should be. Why not utilise native Kubernetes resource definitions when they are available. There is no need for a Postgres or Cassandra, which is the case with Kong.

Let’s Encrypt Certificates

Kong stores certificates in Postgres, whereas Traefik stores these in a shared state store such as Zookeeper. It is capable of automatically requesting certificates with the right configuration, such as the one below. This ConfigMap will have to be mounted as a file in the Traefik DeamonSet or Deployment. Traefik will append the domains beneath [[acme.domains]]. I needed a shared state store so that all Traefik instances could access those newly created certificates. They cannot be stored locally in each pod, all other instances will have to find these certificates.

kind: ConfigMap
apiVersion: v1
metadata:
  name: traefik-configmap
  namespace: kube-system
data:
  traefik.toml: |
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.https]
      address = ":443"
      [entryPoints.https.tls]
    [zookeeper]
      prefix = "traefik"
      endpoint = "zookeeper-cluster.kafka:2181"
    [acme]
      email = "ops@bouweceunen.com"
      entryPoint = "https"
      onHostRule = true
      storage = "traefik/acme/account"
      caServer = "https://acme-v02.api.letsencrypt.org/directory"
      [acme.httpChallenge]
        entryPoint = "http"
    [[acme.domains]]

I wasn’t fond of the idea of setting up a Zookeeper or Consul cluster just for that. I also noticed that Traefik has some issues on GitHub related to; retrying to obtain certificates, the renewal of those and the fact that certificates are only fetched when an Ingress is created. Although they are heavily working on that, I chose to store the certificates in Kubernetes Secrets instead of letting Traefik do this and store it in a shared state cluster. I wrote a small Python script within a Docker container that scrapes Ingresses within a specified time interval and uploads those certificates obtained from Let’s Encrypt as Secrets in Kubernetes. This way no shared state store is needed and helps to keep everything to a minimum. To link these Secrets with Ingresses you need the tls annotation. This annotation in the Ingress denotes a name of a Secret that resides in that namespace. Those certificates can also be manually stored as Secrets by using the following command.

kubectl -n kube-system create secret tls kubernetes-bouweceunen-cert --key=tls.key --cert=tls.crt

BouweCeunen/certbot-kubernetes-secrets-aws

UI Dashboard

Another thing was the Kong dashboard, it is developed separately and it takes some time before it is compatible with the latest versions of Kong. Traefik comes with its own dashboard and it will always be compatible with the latest Traefik version. The UI of Traefik also feels snappier than the UI of Kong. All Ingresses are annotated as ‘FRONTENDS’ and all services as ‘BACKENDS’, it is very easy to see what exactly each Ingress does and to where it routes.

TL;DR

Traefik utilizes Kubernetes to store state and uses the Ingress resource definition to route traffic and uses Secrets to store certificates. The UI dashboard is developed side by side, so no waiting for a compatible dashboard as which is the case with Kong. Traefik is clean, lean and has less hassle with migrations as with Kong and its state store. Traefik Ingresses are also highly configurable and each configuration stays within the Ingress which makes it clear what that Ingress is doing.

The next thing to try is Istio which uses Envoy as a reverse proxy. Istio does so much more than only controlling ingress, so it needs more effort to set up and dig into what this service mesh is capable of. Envoy also has very promising benchmark results in comparison to Kong and Traefik.