Mastering Advanced Ingress-Nginx Techniques: Unleash the Power of Lua Scripts

In the dynamic world of Kubernetes and micro services, Ingress controllers play a pivotal role in routing traffic to your applications. Among these, Ingress-Nginx stands as one of the most popular and versatile choices. While basic ingress configurations are essential, diving deeper into advanced features like Lua scripting can take your Ingress-Nginx mastery to the next level.

In this article, we’ll embark on a journey into the advanced usage of Ingress-Nginx, exploring how Lua scripts can empower you to implement custom requirement.

Let’s dive in and harness the power of Lua scripting with Ingress-Nginx!

Prerequisites

• A GKE cluster created (if you don’t have one, refer to my previous article).
• A registered domain and a TLS secret for this domain.

Installation

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install my-ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace

helm list result as below

% helm list -A
NAME             NAMESPACE     REVISION UPDATED                              STATUS   CHART               APP VERSION
my-ingress-nginx ingress-nginx 1        2023-10-07 14:57:18.129053 +0800 CST deployed ingress-nginx-4.7.2 1.8.2 

Assume you have your own domain, in my example, it’s project-65.com, along with a TLS secret for this domain. If you don’t have one, you can simply use cert-manager with Let’s Encrypt to issue a free one.

kubectl create ns test

kubectl create secret -n test tls test-tls-secret \
--key ./resource.project-65.com.key \
--cert ./resource.project-65.com.pem

Let’s configure a reverse proxy for an external site https://httpbin.org

Notes: If you already have cert-manager and have created a cluster-issuer, then you don’t need to manually create secrets as described in the previous steps. You can simply add annotations to your issuer for Let’s Encrypt to automatically issue certificates.

Similarly, if you have already set up external-dns (as mentioned in my previous article), you can just add annotations to create DNS records in the designated zone automatically, otherwise please add DNS record manually.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wade-demo-ingress
  namespace: test
  annotations:
    # cert-manager.io/cluster-issuer: letsencrypt-dns01-test
    # external-dns.alpha.kubernetes.io/zone: public
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

spec:
  ingressClassName: nginx
  rules:
    - host: resource.project-65.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-external-service
                port:
                  number: 443
  tls:
    - hosts:
      - resource.project-65.com
      secretName: test-tls-secret
---
apiVersion: v1
kind: Service
metadata:
  name: my-external-service
  namespace: test
spec:
  type: ExternalName
  externalName: httpbin.org

Apply this file via command like kubectl apply -f test-ingress.yaml

Don’t forget to add a DNS A record. After a few minutes, the domain will take effect. Then when you access URL https://resource.project-65.com, you will see same result as https://httpbin.org, this behavior is a feature of proxy_pass in Nginx.

If you encounter error “The plain HTTP request was sent to HTTPS port”

Make sure you added the annotation nginx.ingress.kubernetes.io/backend-protocol: “HTTPS”

Now, let’s add a Lua script to the ingress controller, and then follow the steps below.

Create Lua script via configMap, this script is used to prevent non-email format username in the request body.

apiVersion: v1
kind: ConfigMap
metadata:
  name: ngx-custom-script
  namespace: ingress-nginx
data:
  main.lua: |
    local ngx = ngx
    local _M = {}
    function _M.rewrite()
      local cjson = require("cjson")
      ngx.req.read_body()
      local request_body = ngx.req.get_body_data()
      ngx.log(ngx.INFO, "Request Body: " .. request_body) 
      if request_body then
        local username = cjson.decode(request_body).username
        local pattern = "^[%w.-]+@[%w.-]+%.[a-z]+$"
        local match, _ = string.find(string.lower(username), pattern)
        if username and (match == nil) then
          ngx.status = 403
          ngx.log(ngx.ERR, "Forbidden: Non-email username \'", username, "\' not allowed!")
          ngx.exit(ngx.HTTP_FORBIDDEN)
        end
      end
    end
    return _M

Edit the Ingress Nginx deployment, add a volume, and mount it to the path /etc/nginx/lua/plugins/modify_request/

kubectl edit deploy my-ingress-nginx-controller -n ingress-nginx

## add volumes and mount it
        volumeMounts:
        - mountPath: /etc/nginx/lua/plugins/modify_request/
          name: lua-scripts-volume
          readOnly: true
      volumes:
      - configMap:
          defaultMode: 420
          name: ngx-custom-script
        name: lua-scripts-volume

Pod will be recreated automatically due to this change.

kubectl get pod -n ingress-nginx

Edit ingress controller configMap, add plugins in data section.

kubectl edit cm my-ingress-nginx-controller -n ingress-nginx

Verify result, try to curl /post interface with an email format username in the request body, it works well.

% curl -X POST -H 'content-type: application/json' https://resource.project-65.com/post --data-raw '{"username":"wadexu@gmail.com"}'
{
  "args": {}, 
  "data": "{\"username\":\"wadexu@gmail.com\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "31", 
    "Content-Type": "application/json", 
    "Host": "resource.project-65.com", 
    "User-Agent": "curl/8.1.2", 
    "X-Amzn-Trace-Id": "Root=1-65228c20-3845bec87c4cafcf2f900d80", 
    "X-Forwarded-Host": "resource.project-65.com", 
    "X-Forwarded-Scheme": "https", 
    "X-Scheme": "https"
  }, 
  "json": {
    "username": "wadexu@gmail.com"
  }, 
  "origin": "10.254.71.6, 34.31.187.187", 
  "url": "https://resource.project-65.com/post"
}

Try to curl /post interface with an non-email format username in the request body

curl -X POST -H 'content-type: application/json' https://resource.project-65.com/post --data-raw '{"username":"wadexu"}'

Return 403 Forbidden, this means our script is running correctly.

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

If you curl https://httpbin.org/post directly with an non-email format username, the result will work as expected, and you won’t encounter a Forbidden error. This proves that our script is functioning correctly.

Now, let’s try some other examples. If you want to create a Lua script for a specific API, you can add the script in the location section within the server-snippet annotation.

    nginx.ingress.kubernetes.io/server-snippet: |
      location /put {
        if ($request_method = PUT) {
            rewrite_by_lua_block {
                local cjson = require("cjson")
                ngx.req.read_body()
                local request_body = ngx.req.get_body_data()
                if request_body then
                  local username = cjson.decode(request_body).username
                  local pattern = "^[%w.-]+@[%w.-]+%.[a-z]+$"
                  local match, _ = string.find(string.lower(username), pattern)
                  if username and (match == nil) then
                    ngx.status = 403
                    ngx.log(ngx.ERR, "Forbidden: Non-email username \'", username, "\' not allowed!")
                    ngx.exit(ngx.HTTP_FORBIDDEN)
                  end
                end
            }
        }
        proxy_pass https://httpbin.org;
      }

Originally, when we curl https://httpbin.org/put

curl -X PUT -H 'content-type: application/json' https://httpbin.org/put --data-raw '{"username":"wadexu"}'

The result is

{
  "args": {}, 
  "data": "{\"username\":\"wadexu\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "21", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/8.1.2", 
    "X-Amzn-Trace-Id": "Root=1-65228e16-6010f577596f37cd16e31b44"
  }, 
  "json": {
    "username": "wadexu"
  }, 
  "origin": "47.57.1.136", 
  "url": "https://httpbin.org/put"
}

Now we call our reverse proxy https://resource.project-65.com/put, you will receive a 403 Forbidden response. This behavior is governed by the Lua scripts in above code snippet.

 % curl -X PUT -H 'content-type: application/json' https://resource.project-65.com/put --data-raw '{"username":"wadexu"}'

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

If use email format username wadexu@test.com , you will receive the same response as from the original API at https://httpbin.org/post. No forbidden error.

curl -X POST -H 'content-type: application/json' https://resource.project-65.com/post --data-raw '{"username":"wadexu@test.com"}'
{
  "args": {}, 
  "data": "{\"username\":\"wadexu@test.com\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "30", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/8.1.2", 
    "X-Amzn-Trace-Id": "Root=1-65212916-2bec92fe5fc37f560c31169b"
  }, 
  "json": {
    "username": "wadexu@test.com"
  }, 
  "origin": "34.31.187.187", 
  "url": "https://httpbin.org/post"
}

Let’s explore another example, where we rename a cookie name.

We’ll prepare an API with customized headers to set a cookie named AuthToken with a value of my_value.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: base-api
  namespace: test
  annotations:
    # cert-manager.io/cluster-issuer: letsencrypt-dns01-test
    # external-dns.alpha.kubernetes.io/zone: public
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Frame-Options: DENY";
      more_set_headers "Set-Cookie: AuthToken=my_value; Domain=api.project-65.com; Path=/; Expires=Wed, 15 Jun 2023 10:00:00 GMT";
spec:
  ingressClassName: nginx-public
  rules:
    - host: api.project-65.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: nginx-service
                port:
                  number: 80
  tls:
    - hosts:
      - api.project-65.com
      secretName: api-test-tls-secret
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: test
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: test
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx-container
          image: nginx:latest
          ports:
            - containerPort: 80

curl to test

% curl -I https://api.project-65.com/api
HTTP/2 200 
date: Sat, 07 Oct 2023 10:01:06 GMT
content-type: text/html
content-length: 615
last-modified: Tue, 15 Aug 2023 17:03:04 GMT
etag: "64dbafc8-267"
accept-ranges: bytes
strict-transport-security: max-age=15724800; includeSubDomains
x-frame-options: DENY
set-cookie: AuthToken=my_value; Domain=api.project-65.com; Path=/; Expires=Wed, 15 Jun 2023 10:00:00 GMT

In the previous reverse proxy wade-demo-ingress Ingress configuration under the server-snippet section, let’s add a new location for /api to replace the cookie name using the string.gsub method in Lua.

      location /api {
        proxy_pass https://api.project-65.com;

        header_filter_by_lua '
          local cookies = ngx.header.set_cookie 
          if not cookies then return end
          if type(cookies) ~= "table" then cookies = {cookies} end
          for i, val in ipairs(cookies) do
              local newval, _ = string.gsub(val, "(AuthToken=)", "AuthToken_new=")
              cookies[i] = newval
          end 
          ngx.header.set_cookie = cookies
        ';
      }

Now, when you test it with curl, you will observe a response cookie named AuthToken_new

% curl -I https://resource.project-65.com/api     
HTTP/2 200 
date: Sat, 07 Oct 2023 10:06:12 GMT
content-type: text/html
content-length: 615
last-modified: Tue, 15 Aug 2023 17:03:04 GMT
etag: "64dbafc8-267"
accept-ranges: bytes
strict-transport-security: max-age=15724800; includeSubDomains
x-frame-options: DENY
set-cookie: AuthToken_new=my_value; Domain=resource.project-65.com; Path=/; Expires=Wed, 15 Jun 2023 10:00:00 GMT

The same result can be seen when using browser developer tools.

Tips:

1. There are proxy_cookie_domain and proxy_cookie_path to modify Domain name and path if you need that.

  proxy_cookie_domain resource.project-65.com .project-65.com;
  proxy_cookie_path /api /newapi;

2. If your upstream host enabled CORS, then in proxy you need some config like below.

    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "https://xxx.xx.com"

3. Below configurations can help with SSL handshake failures when your backend server is behind Cloudflare, more details can see official ingress-nginx docs.

    nginx.ingress.kubernetes.io/proxy-ssl-server-name: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-name: "xxx.xxx.com"

2023/09/13 09:59:01 [error] 2205#2205: *67672 SSL_do_handshake() failed (SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream, client: 223.106.142.148, server: xxx.xxx.com, request: "GET / HTTP/2.0", upstream: "https://104.18.22.205:443/", host: "xxx.xxx.com"

Reference:

ingress-nginx/rootfs/etc/nginx/lua/plugins/README.md at main · kubernetes/ingress-nginx

Annotations - Ingress-Nginx Controller