Ingress Controller with advanced config

Nginx controller with Advanced configuration

Recently There was one requirement where i have to setup ingress controller for one of my application.

Installation

The best way to setup Nginx is to use manifest, if you are doing it for first time. but if you want to automate please use helm chart. Please follow below links for installation according to requirement.
Install NGINX Ingress Controller in a Kubernetes cluster using manifests.
Install NGINX Ingress Controller in your Kubernetes cluster using Helm.

Note: Some times nginx pods running on nginx-ingress namespace will be showing crashloopbackOff error, please validate ingress version if it supported with current version of k8s. rectify that issue and pod will be in running stage.

running ingress controller pod

Ingress Routes

Now it’s time to create ingress routes. For now i am keeping it simple and below is the ingress route yaml file.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  generation: 1
  name: test-app-ingress
  namespace: app-temp-namespace
spec:
  ingressClassName: nginx
  rules:
  - host: test-app.com
    http:
      paths:
      - backend:
          service:
            name: test-app-svc
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

You can list down created route like below

kubectl get ing -n test-app-namespace                                                       
NAMESPACE            NAME                    CLASS   HOSTS                                           ADDRESS        PORTS   AGE
test-app-namespace   test-app-ingress        nginx   test-app.com                                                   80      1d

Now its time to check nginx configuration and you will be able to find default nginx configuration.

Find nginx pod inside and try to exec into pod and list down your configs with below command.

ls -ltr /etc/nginx/conf.d/

Now it’s time to see what these files contains. these files got created using default configuration of nginx and basically it conatains two nginx directives upstream and server. We will be refrring here mostly server directive.

server {
 
 
 listen 80;
 server_tokens on;
 server_name test-app.com;

 set $resource_type "ingress";
 set $resource_name "test-app-ingress";
 set $resource_namespace "app-temp-namespace";

 location / {
  set $service "test-app-svc"; 
  
  proxy_http_version 1.1;
  proxy_connect_timeout 60s;
  proxy_read_timeout 60s;
  proxy_send_timeout 60s;
  client_max_body_size 1m;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Port $server_port;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_buffering on;
 
  proxy_pass http://app-temp-namespace-test-app-ingress-test-app.com-test-app-svc-80;
  
  
 }
}

Till now we have setup nginx-controller and routes with default config. Now Our requirement is do fine tuning nginx controller according to requirement.

Let’s assume we need to update value of client_max_body_size to 4m
instead of dfault value 1m. How can we do that. What are the ways to achieve this.
Using Annotations
Using ConfigMap

Here Mostly we will focus on Advanced configuration using Annotations

Assumming we have to update value of client_max_body_size, so we need to update annotations while defining ingress routes.

annotations:
     nginx.org/client-max-body-size: "4m"

So our ingress yaml file will look like below

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  generation: 1
  name: test-app-ingress
  namespace: test-app-namespace
  annotations:
     nginx.org/client-max-body-size: "4m"       
spec:
  ingressClassName: nginx
  rules:
  - host: test-app.com
    http:
      paths:
      - backend:
          service:
            name: test-app-svc
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

After updating it, you should be able to see nginx configuration would have been updated like below:

server {
 
 
 listen 80;
 server_tokens on;
 server_name test-app.com;

 set $resource_type "ingress";
 set $resource_name "test-app-ingress";
 set $resource_namespace "app-temp-namespace";

 location / {
  set $service "test-app-svc"; 
  
  proxy_http_version 1.1;
  proxy_connect_timeout 60s;
  proxy_read_timeout 60s;
  proxy_send_timeout 60s;
  client_max_body_size 4m;  ### updated value from 1m to 4m
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Port $server_port;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_buffering on;
 
  proxy_pass http://app-temp-namespace-test-app-ingress-test-app.com-test-app-svc-80;
  
  
 }
}

Now assume we have to update some server level directives like client_header_buffer_size and large_client_header_buffers. In this case we have to specially add these as snippet because its not part of default configuration of Nginx. To achieve this we need to enable snippet feature in nginx.

Enabling snippet feature in Nginx

-enable-sinppets

Above parameters we need to pass in nginx-controller. Just providing you below partial yaml file. Ref

    spec:
      serviceAccountName: nginx-ingress
      containers:
      - image: nginx/nginx-ingress:1.12.4
        imagePullPolicy: IfNotPresent
        name: nginx-ingress
        args:
          - -nginx-configmaps=$(POD_NAMESPACE)/nginx-config
          - -default-server-tls-secret=$(POD_NAMESPACE)/default-server-secret
          - -enable-snippets ## for enabling snippet

Now our Nginx is enabled to understand annotations based on snippet.Update you ingress yaml file as below.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  generation: 1
  name: test-app-ingress
  namespace: test-app-namespace
  annotations:    
     nginx.org/server-snippets: |
              client_header_buffer_size 2048k;
              large_client_header_buffers 4 2048k;
spec:
  ingressClassName: nginx
  rules:
  - host: test-app.com
    http:
      paths:
      - backend:
          service:
            name: test-app-svc
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific

After applying above ingress route snippets content should be available on nginx.conf

server {
 
 
 listen 80;
 server_tokens on;
 server_name test-app.com;

 set $resource_type "ingress";
 set $resource_name "test-app-ingress";
 set $resource_namespace "app-temp-namespace";

 client_header_buffer_size 2048k;       ## Newl Added server snippet
 large_client_header_buffers 4 2048k;   ## Newl Added server snippet

 location / {
  set $service "test-app-svc"; 
  
  proxy_http_version 1.1;
  proxy_connect_timeout 60s;
  proxy_read_timeout 60s;
  proxy_send_timeout 60s;
  client_max_body_size 1m;  
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Port $server_port;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_buffering on;
 
  proxy_pass http://app-temp-namespace-test-app-ingress-test-app.com-test-app-svc-80;
  
  
 }
}

Like this you can different types of snippets according to requirement.
nginx.org/http-snippets
nginx.org/server-snippets
nginx.org/location-snippets

Will cover configMap based advanced configuration in some other post if required.

I hope now things are cleared. I have tried to explain how to configure Nginx with advanced configuration. Please share your feedback and let me know if you have any query.