Custom Plugins on Kong Gateway

Implementing custom behavior to your Kong Gateway is easier than it sounds

Published in

project44 TechBlog

9 min readJan 3, 2023

In a previous article, we described how easy it was to configure Kong as a Gateway into a Kubernetes cluster of micro-services. Onboarding services was a breeze, and we lightly explored how to implement bundled plugins.

But Kong is more than a Gateway to be used out-of-the-box. One of the reasons we liked it so much at project44 was because it allows us to introduce custom plugins with our own business logic to act upon a request before it reaches any downstream services!

Kong’s documentation on how to do this in Lua is robust and extensive.

However, Python did not appear to have the same kind of detailed service at the time we were considering Kong as a solution. This in spite of the fact Python plugins are technically supported.

In this article, we’re going to build a custom plugin in Python and apply it to an instance of Kong running in a Kubernetes cluster. Reading the existing documentation as a prerequisite is not required, but as with any official docs, I highly recommend you touch-base for updates and clarifications as they’re needed.

Requirements

Python ≥v3.9
Docker ≥v20.10
Helm v3
A local Kubernetes: eg. minikube, kind, microk8s.
We’ll use minikube in this article, but any will do!

This article is written with the assumption you’ve followed the previous Kong Gateway article and already have a pre-configured instance of Kong running in a cluster we may start and stop.

The initial project set-up we’ll require is the one we created in the previous article about setting up Kong on Kubernetes.

Find the complete result for this project in this repository.

The Plugin

We’re building a custom plugin because we have custom business logic we need to employ to our gateway.

In this case, we’ll assume the following of all of our services behind the gateway:

Any request must contain a header with the key hello and a non-null value.
The plugin must inject a header in the inbound request with the key now, holding a value of the UTC datetime.

To get started, we need to configure Kong to run Python plugins.

Custom Kong Image

By default, the Kong image we deployed into our cluster includes no dependencies or handling for third-party plugins. We can resolve the lack of Python dependencies by using this as a base image and installing Python’s dependencies on top of it.

In the root of your project, create a Dockerfile; this will become the new image we deploy into our cluster. Find the Python dependencies addition flagged with a 💥…

# ./Dockerfile# Set the existing Kong image as the base
FROM kong:2.8
USER root# 💥 Install Python + dependencies
RUN apk update
RUN apk add python3 py3-pip python3-dev musl-dev libffi-dev gcc g++ file make# Initialize Kong
USER kong
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 8000 8443 8001 8444
STOPSIGNAL SIGQUIT
HEALTHCHECK --interval=10s --timeout=10s --retries=10 CMD kong health
CMD ["kong", "docker-start"]

The section titled # Initialize Kong is a standard, Kong-defined instruction one may find described in their plugin documentation.

If we left the container alone at this point, it is still deployable; it will have python3 installed inside the container, though! There are no instructions for handling new plugins, should any be available. The solution for that is something Kong calls a Plugin Server; a host for one or more plugins running in the same container as the Gateway.

The Python plugin server is defined in the Kong plugin dependency, the Python Plugin Development Kit (PDK). It stands as an executable for Kong to interact with our python plugins. For our purposes, all we need to do is create a file which can be called to initiate the plugin server.

Create a script to start the plugin server…

# ./pluginserver#!/usr/bin/env python3from kong_pdk import clicli.start_server()

The top line of this file is its name, and it does not need to be inside the file; the shebang (#!) noted below it is necessary in defining how this file should be run as an executable, and that should be kept at the top of the file.

The library we referenced for this plugin-server executable file is a dependency for running the plugin server; kong_pdk. It’s used as a dependency in plugin creation, so we’re also going to need to install it as a library dependency in our docker image.

Note the new additions flagged by the 💥…

# ./Dockerfile# Set the existing Kong image as the base
FROM kong:2.8
USER root# Install Python + dependencies
RUN apk update
RUN apk add python3 py3-pip python3-dev musl-dev libffi-dev gcc g++ file make# 💥 Install Python Plugin dependency
RUN PYTHONWARNINGS=ignore pip3 install kong-pdk==0.31# 💥 Copy in Plugin Server Exec
COPY --chown=kong --chmod=555 ./pluginserver.py /usr/local/bin/kong-python-pluginserver# Initialize Kong
USER kong
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 8000 8443 8001 8444
STOPSIGNAL SIGQUIT
HEALTHCHECK --interval=10s --timeout=10s --retries=10 CMD kong health
CMD ["kong", "docker-start"]

And finally, we’ll need to copy in our plugins, wherever we choose to create them. We’ll prepare to make several in the future, and create a whole directory for them all.

Create an empty directory in your project root for now, ./py-plugins, and add one more line to your Dockerfile …

# ./Dockerfile# Set the existing Kong image as the base
FROM kong:2.8
USER root# Install Python + dependencies
RUN apk update
RUN apk add python3 py3-pip python3-dev musl-dev libffi-dev gcc g++ file make#Install Python Plugin dependency
RUN PYTHONWARNINGS=ignore pip3 install kong-pdk==0.31# Copy in Plugin Server Exec
COPY --chown=kong --chmod=555 ./pluginserver.py /usr/local/bin/kong-python-pluginserver# 💥 Copy in Python Plugins
COPY --chown=kong --chmod=555 ./py-plugins /usr/local/kong/python-plugins# Initialize Kong
USER kong
ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 8000 8443 8001 8444
STOPSIGNAL SIGQUIT
HEALTHCHECK --interval=10s --timeout=10s --retries=10 CMD kong health
CMD ["kong", "docker-start"]

The file locations for all the items we copied in are arbitrarily named and they could have been anything. But we’ll need to refer back to them later! So keep note if you choose to organize things differently.
The use of --chown and --chmod in our COPY commands are required to ensure the kong user is able to read and execute the files we’ve copied in; if you’re using an older version of Docker, these commands may need to be in their own lines.

At this point, our Dockerfile is complete. We have layered on top of the base Kong image a number of new items!

Install Python.
Install the Kong plugin library.
Create the Python plugin-server executable.
Copy in all custom Plugins written in Python.

The last thing to do here is create the custom plugins with our desired business logic.

Writing the Plugin

With the Kong image prepared to run Python plugins, all we need to do is…

Write the business logic of our plugin, given the requirements we set out earlier.
Inform Kong it is available for use in our Gateway.

Custom Plugin Development

Provided our desire to check for the header hello and inject one with the key now, we may use the following script to populate the plugin file inside the plugin directory we’ve earmarked for copying into the docker image…

Documentation for the Kong PDK library can be found here.

# ./py-plugins/header_check.pyfrom datetime import datetime
import kong_pdk.pdk.kong as kongSchema = [
    {"HEADER_NAME": {"type": "string"}},
]
version = "0.1.0"
priority = 1000class Plugin:
    def __init__(self, config):
        self.config = config    def access(self, kong: kong.kong):        # Grab the name + value of the required header.
        header_name = self.config["HEADER_NAME"]
        header_value = kong.request.get_header(header_name)        # Raise if the header value cannot be found.        
        if header_value is None:
            return kong.response.error(
                400,
                f"Missing header: {header_name}"
            )        # Else, set the current time in the header `now`.
        else:
            kong.service.request.set_header(
                "now",
                str(datetime.utcnow())
            )

Though the format of the file is not the most pythonic, the structure is rigidly set by Kong.
The Schema at the top of the file is a key-value mapping of values which may be passed through from the Kubernetes Manifest this plugin will require shortly; at run-time, this schema is represented in Plugin().config as a dictionary.
In our case, this is how we choose to pass through the expected header key hello.

Plugin.access contains our business logic here, where access is one of many optional phase-handlers — access indicates we want to operate on the request before it reaches our downstream services. All options and definitions for phase-handlers are found here.

At this point, Kong has been provided the tools to execute Python plugins, and we’ve written the plugin we’d like it to apply, but we have yet to represent this plugin with a kubernetes manifest and configure Kong to execute this plugin on any inbound request.

Custom Plugin Manifest

Just as the bundled plugin required a manifest to hold a presence in the k8s cluster, our custom plugin needs one too. Create a manifest file in the ./helm directory…

#./helm/kong/templates/plugins/header-check.yamlapiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: header-check
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
config:
  HEADER_NAME: "hello"
plugin: header_check

metadata.name is an arbitrary, cluster-unique name for the plugin which must be kebab-case.
plugin is the same as the filename which houses the plugin code.
config is a key-value mapping for data to be passed through to the plugin Schema; note, we’re passing through hello as the HEADER_NAME.

Now our plugin may carry a presence in our cluster. But the Kong Gateway does not consider this plugin to have been fully enabled without being told about it in its own configuration.

Final Configuration Changes to Kong

Seek out the ./helm/kong/values.yaml file for three final changes:

Change the image to be deployed into our cluster, say we’ll call it custom-kong version 1.0.0.
Include the plugin by the name of the file in an environment variable called plugins.
Include instructions for Kong to execute the Python plugin server.

# ./helm/kong/values.yamlkong:  # 💥 Overwrite the kong image with the custom one.
  image:
    unifiedRepoTag: custom-kong:1.0.0
    pullPolicy: IfNotPresent  # 💥 Set the plugins and provide Kong the tools
  # needed to run the Python plugin server.
  env:
    database: "off"
    plugins: "bundled,header_check"
    pluginserver_names: "python"
    pluginserver_python_socket: >-
      /usr/local/kong/python_pluginserver.sock
    pluginserver_python_start_cmd: >-
      /usr/local/bin/kong-python-pluginserver \
        --plugins-directory /usr/local/kong/python-plugins \
        --no-lua-style
    pluginserver_python_query_cmd: >-
      /usr/local/bin/kong-python-pluginserver \
        --plugins-directory /usr/local/kong/python-plugins \
        --no-lua-style \
        --dump-all-plugins  # Prepare for k8s Ingress manifests
  ingressController:
    enabled: true
    ingressClass: kong  # There's an HTTP2 bug in Kong which creates
  # an excess of noise in the proxy logs.
  # The fix will be in Kong 2.8.2
  # https://github.com/Kong/kong/pull/8690
  admin:
    tls:
      parameters: []

Note, we’re also including a plugin called bundled; this is a reference to the entire suite of plugins which Kong comes with by default. This is the only value loaded into this environment variable by default when it is not explicitly provided, and it is required for continued usage of those bundled plugins which come with Kong, like the rate-limiter we already implemented.
With regard to the pluginserver environment variables, the locations specified are directories we’ve created in our Docker container; the pluginserver and plugins files are dependencies for Kong to use when handling requests, now.

Deploy!

With all of our pieces pulled together, we are able to build our Docker container with the name we used to configure Kong and deploy it to our cluster. From the root of your project directory, run the following…

$ docker build . -t custom-kong:1.0.0$ minikube image load custom-kong:1.0.0$ helm dep up ./helm/kong
$ helm install kong ./helm/kong

Minikube has a separate registry from your local registry; minikube image load makes the custom image we’re building accessible to your kubernetes cluster.

Once you’re deployed, open a tunnel through minikube and make a request.

$ minikube tunnel$ curl -X GET localhost/echo
> {"message": "Missing header: hello"}

It works! We’ve made a naked request without a hello header which yielded our custom error. What happens when we add the desired header value…

$ curl -X GET localhost/echo --header 'hello: world'

The response should be similar to the echo-response we saw in the previous article…

{
  "host": {
    "hostname": "localhost",
    "ip": "::ffff:172.17.0.3",
    "ips": []
  },
  "http": {
    "method": "GET",
    "baseUrl": "",
    "originalUrl": "/echo",
    "protocol": "http"
    ...
  ...
}

Note the request.headers body; seek out the now field:

"headers": {
      "host": "localhost",
      ...
      "hello": "world",
      "now": "2022-08-17 14:25:53.889690"
    }

We find there’s a new header added onto the inbound request alongside our hello header; now shows the time at which the request was processed by our plugin!

Testing

As it stands, Kong does not have an official means of unit-testing plugins. Internally, we’ve written our own means of representing a kong.kong object for unit-tests, and as recommended by the Kong team, a full end-to-end test is also required to ensure desired behavior of any custom plugins.

Conclusion

Together, we’ve outlined what it takes to implement a custom Kong plugin in Python.

Moving forward, we can trust the micro-service-gateway based infrastructure we built out and continue to add new plugins by sticking to the following process:

Write a new plugin file in Python.
Write the applicable plugin manifest.
Add the plugin to the plugins environment variable known to the Kong Gateway.

There’s plenty more to custom plugin development which we did not explore, including path/service-based plugins and employing more complicated operations in our plugin, but we hope this stands as a stable foundation for you to build upon in creating your own Kong plugins!