Improve your Kong Plugin Experience

Kong, the API Gateway used at Manomano

At Manomano we have moved to an API first strategy, (you can read the 2021: ManoMano IT Odyssey to know all about this).

To sustain this strategy, we have chosen Kong as our API Gateway (You can read more about this on API Gateway “a la Mano” ). Kong is a key asset for us. One big feature of Kong is the ease to use some plugins to extend a reverse proxy to a smart API Gateway fitting our needs. Quickly you as us will want to create your own plugins to have a better control, to implement some specific rules, or just because you love Lua.

I won’t detail here how to create your plugin, you can easily find several good tutorial, check the open sources plugin to inspire you, and the Kong documentation is quite clear.
The hard part is how to correctly develop, with a nice developer experience, how to do tdd and how to stop re-inventing the wheel.

What is Kong ?

From the Kong documentation itself:

To be more precise, Kong is a Lua application running in Nginx and made possible by the lua-nginx-module. Instead of compiling Nginx with this module, Kong is distributed along with OpenResty, which already includes lua-nginx-module. OpenResty is not a fork of Nginx, but a bundle of modules extending its capabilities

And for the plugins:

Hence, Kong is a Lua application designed to load and execute Lua modules (which we more commonly refer to as “plugins”) and provides an entire development environment for them, including an SDK, database abstractions, migrations, and more.

To write some Kong plugins, you will need to use Lua.

If the documentation from Kong is quite precise and provide lot of detail about writing a plugin. The part dedicated to the test is tiny.
This is mentioning busted and an helper to start Kong from busted, quite dry to understand exactly how to write a test.

No mention how to simply run your plugin, just having your plugin live can become a complicate mission.

But this had changed.

Pongo

During the autumn 2019, Pongo was announced.

/~\
  ______       C oo
  | ___ \      _( ^)
  | |_/ /__  _/__ ~\ __   ___
  |  __/ _ \| '_ \ / _` |/ _ \
  | | | (_) | | | | (_| | (_) |
  \_|  \___/|_| |_|\__, |\___/
                    __/ |
                   |___/

Pongo provides a simple way of testing Kong plugins

Pongo is based on docker and docker-compose. This is quite easy to have a running Kong with several plugins in few minutes.
For sure Pongo helps a lot to test a Kong plugin, but there is still a lack of documentation about how to write any tests.

Why testing Kong is not trivial ?

By design Kong is between a client and different endpoints. To test your plugins, you need to simulate those endpoints. If you want to load balance between several endpoints, route to a specific endpoint, etc. you will have quickly to set up several endpoint or to mock them.

Looking into the tests available into the Kong repository, we can find some example of endpoint mocks. We will list in the third article the different way to easily write test we have found.

While I was thinking to write those articles, an article about how to write a plugin was made, https://konghq.com/blog/custom-lua-plugin-kong-gateway. As you can see, a short paragraph about testing, nothing about schema entities check neither endpoint mockup.

In this series of articles we will see how to improve your day to day plugin development using the Kong helpers.
We will focus on two main set of helpers:

• Entity check and Fields validator
• Functional testing, setting up Kong and mocking up some upstream servers

Kapla tower — Véronique

Simple plugin

Time to open Vim (or you IDE) and start with a small example to understand how to have a nice developer experience by easily creating some tests. For this, we will create a simple plugin and write the tests. The code is available on github, few details about the code will be provided, the idea is to be focus on the test part.

We will create a plugin to block or allow traffic based on a http header and based on the source ip. As blocking traffic can have a huge business impact, a common way to validate the behavior is to have an option to mark the traffic instead of rejecting it, as a dry run mode. Our plugin will allow to set the name of the http header where to get the value to check, a list of allowed values, a list of deny values. For the ip, we will able to select where to get the ip from“connection”, “forwarded_ip” or “header”, a http header if we select this mode, and a list of allowed ips or ranges and a list of denied ips or ranges.
A mark mode with 4 options “all”, “allow”, “deny”, “none”.

• to flag both allowed and denied,
• just allowed,
• just denied, or
• no mark.

And a mark header to specify where to do the marking.

For example a configuration will be:

config = {
  ip_allow = {"10.0.0.1", "10.0.1.0/24"},
  ip_deny  = {"10.1.0.1", "10.1.1.0/24"},
  ip_source = "header",
  ip_header_source = "x-true-client-ip",
  mark_header = "x-limit",
  mark_action = "allow",
  header_name = "x-source",
  header_allow = {"trust"},
  header_deny = {"bad", "vilain"}
}

We will use Pongo, so you need:

• docker-compose (and hence docker)
• curl
• realpath

And that is.

Lets start a new plugin.

▶ mkdir kong-plugin-medium-test
▶ cd kong-plugin-medium-test
▶ git init
Initialized empty Git repository in /somewhere/over/the/rainbow 
▶ wget https://raw.githubusercontent.com/Kong/kong-plugin/master/.luacheckrc

First we need to validate our code against a linter. At Manomano we are using the classic setup of Lua linter. Just add .luacheckrc file to your repository, for example you can find this one on https://github.com/Kong/kong-plugin/blob/master/.luacheckrc

Time to create our plugin, create the directory:

mkdir -p kong/plugins/medium-test

The classic file for the plugin are:

handler.lua: the core of your plugin. It is an interface to implement, in which each function will be run at the desired moment in the lifecycle of a request / connection.

schema.lua: your plugin probably has to retain some configuration entered by the user. This module holds the schema of that configuration and defines rules on it, so that the user can only enter valid configuration values.

Schema module always starts with:

local typedefs = require "kong.db.schema.typedefs"
local plugin_name = ({...})[1]:match("^kong%.plugins%.([^%.]+)")

The first line allows to have a bunch of useful helper to type your fields. We will deep dive on those helpers on the next article.
The second line is to directly get the name of the plugin from the name of the directory.

We will return our schema configuration, time to add some fields:

return {
  name = plugin_name,
  fields = {
    { consumer = typedefs.no_consumer },
    { protocols = typedefs.protocols_http },
    { config = {
      type = "record",
      fields = {

We will start with basic stuff, we want to block or allow by header, we need:

• the header name
• array of values to block
• array of values to allow

{ header_name = typedefs.header_name },
        { header_allow = {
          type = "array",
          elements = { type = "string" },
          default = {}
        }},
        { header_deny = {
          type = "array",
          elements = { type = "string" },
          default = {}
        }},

Typedefs offers us a header_name type, we will look in detail after.

We want to have an option to mark or block the traffic, we need:

• the header name
• which action we want to mark, looking around on other plugin we can find that “one_of” allows to do a select for different values.

-- Header to flag query
        { mark_header = typedefs.header_name },
        -- deny and all mode will not block the query just flag it
        { mark_action = { type = "string", default = "none", one_of = {"none", "allow", "deny", "all"}, } },

schema file

Do a minimalist handler module:

We need a rockspec, nothing special here:

Victoria State — Australia — Aurélien

To install pongo just follow the documentation:

Clone the repository and install Pongo:

PATH=$PATH:~/.local/bin
git clone https://github.com/Kong/kong-pongo.git
mkdir -p ~/.local/bin
ln -s $(realpath kong-pongo/pongo.sh) ~/.local/bin/pongo

Here we are. We have prepared everything to start to test our plugin. We can now start a TDD approach.

We will continue, this Kong Plugin Helpers journey with 2 articles, one dedicated to the schema part (the plugin configuration) and one dedicated to the functional testing.

Schema article has been released: https://medium.com/manomano-tech/kong-plugin-configuration-how-to-define-schema-and-test-it-cf72f77e11f8

Functional testing article is available:
https://medium.com/manomano-tech/kong-plugin-easy-functional-testing-67949957527b