Sending Kong logs to a GCP Pub/Sub topic

Let’s start by some definitions.

Kong is an open source API gateway that provides a number of features: authentication, traffic control, logging, etc. It can also be configured as an Ingress in a Kubernetes cluster.

In Kubernetes, messages come through the Kong Ingress and are dispatched to the appropriate services.

Kong Ingress accepts the requests and dispatches them to the right services

The functionality of Kong can be further extended by using plugins (https://docs.konghq.com/hub/). There is a plugin to validate the API keys, another to limit the request rate, etc.

In this post we will focus on Kong logging capabilities. Some of the logging related plugins allow to connect with known solutions like Loggly and Kafka. And if you are not using any of these solutions there are also plugins that send the logs using more general HTTP or TCP protocols.

On the other side, Google Cloud Platform Pub/Sub (just “Pub/Sub” from now on) is a service that provides distributed event streaming based on “topics”.

The challenge: send Kong logs to a Pub/Sub topic

We replaced a previous gateway with Kong. This previous gateway was sending the logs to a Pub/Sub topic, which would be consumed by another component to produce some analytics. How could we do this with Kong? How can we send Kong logged data to the Pub/Sub topic?

How to send Kong log data to the Pub/Sub topic?

As we saw before, we can tell Kong to do a number of things by installing some plugins. In our case, however, there is no plugin that sends events to a Pub/Sub topic. I explored a couple of approaches:

• Write a Kong plugin that sends the data directly to the Pub/Sub topic. Kong is based in nginx and the lua-nginx-module, that enables Lua scripting capabilities. From their documentation, “Kong is a Lua application designed to load and execute Lua modules”. Therefore we would write the plugin in Lua.
• Write a sidecar container in Go that receives the log information through a TCP port and sends the data to the Pub/Sub topic using the official Go client.

The problem with the Lua plugin approach is that there is no Lua Pub/Sub client. I researched a bit if it was possible to use some bindings with other languages or use HTTP requests directly but I ended up giving up this approach. I implemented the second one instead: the Go sidecar container.

The solution

The idea is to ship two containers in the same Kubernetes pod: one with Kong, including the “TCP Log” plugin, and another container, the sidecar, with a Go TCP server that sends to a Pub/Sub topic whatever is received through TCP. Then just wire those two together:

tcp-log plugin will send the log data to a sidecar container, which in turn will send it to the Pub/Sub topic

For that, we would use a manifest similar to the following one to deploy Kong along with the Go sidecar:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-kong
  namespace: kong
spec:
  template:
    spec:
      containers:
      # The Kong gateway
      - name: proxy
        image: [edited]/kong
      # The go sidecar, listening in port 10000
      - name: tcp-pubsub
        image: [edited]/tcp-pubsub
        args:
        # listen in port 10000
        - "10000"

The tcp-log Kong plugin would be configured with a KongPlugin CRD (Custom Resource Definition), which will point to the Go sidecar. As both Kong and the sidecar run in the same pod we point to localhost:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: plugin-tcpsidecar-log
plugin: tcp-log
config:
  host: localhost
  port: 10000

There is one potential issue with this solution. Kong will be receiving many requests and they will be sent through TCP to our sidecar. What happens when the sidecar behaves slowly? Will that affect the performance of the gateway?

I tested the load using JMeter and the Ultimate thread group, sustaining two concurrent requests for several minutes. I also introduced some artificial wait times in the sidecar simulating some slow network connection, for example. The following chart shows the response time for the successful responses:

At some point the system collapses

In the previous chart we can see that:

1. Initially requests come at a fast pace.
2. Kong sends the request data to the Go sidecar.
3. Go sidecar will accept the requests and will wait 3s before sending them to the Pub/Sub topic.
4. As long as the sidecar can get new connections Kong does not notice anything.
5. At some point, the sidecar start experiencing memory problems and cannot accept new connections, memory starts to accumulate in Kong.
6. At some point Kong runs out of memory and cannot handle more requests. Very little requests manage to get a successful response.

That’s not a desirable solution. If the logging does not work it’s fine to lose some log lines but it is not acceptable to get the gateway performance degraded.

The solution for this issue is to make the Go sidecar deal with the connections as fast as possible. We implemented a producer/consumer pattern with Go channels:

1. A thread is reading log lines from the TCP connection.
2. Whenever a log line is read it is just sent to the channel, which is a very fast operation.
3. In parallel, N threads consume data from the channel, doing the potentially expensive operation of sending to the Pub/Sub topic.
4. If the channel is full, the sending to the channel will immediately fail and the log line will be discarded.

Thus, in case of a problem with the Pub/Sub link the log lines will be discarded but it will not affect the performance of the gateway.

A new run of the JMeter test would provide a very different result:

Gateway does not get affected by a slow link to the Pub/Sub service

There are three requests that take a long time but with this solution the system does not collapse.