Kubernetes nginx-ingress AWS with custom configs

Overview:

Kubernetes uses ingress controller for getting your application accessible outside of the cluster. This is some kind of the load balancer for routing external traffic to your service. It uses L4-L7 network layers. Ingress controller could be hardware or cloud balancer.

Kubernetes has two official different nginx controllers:

• ingress-nginx
• nginx-ingress(kubernetes-ingress)

The main issue is confusing with documentation as when you’re trying to search in google nginx-ingress, google shows both examples of the controller configuration manual. You should keep on eye what guide you exactly read and move forward.

We will talk more about nginx-ingress(by nginx) controller and compare it with ingress-nginx controller…

Architecture diagram

Diagram show in scope os AWS how does controller works and handles the client requests to the proper Kubernetes pod

Comparing controllers

Notes:

*1 — The configuration templates that are used by the Ingress controllers to generate NGINX configuration are different. As a result, for the same Ingress resource the generated NGINX configuration files are different from one Ingress controller to the other, which in turn means that in some cases the behavior of NGINX can be different as well.

*2 — Because the command-line arguments are different, it is not possible to use the same deployment manifest for deploying the Ingress controllers.

Deployment nginx-ingress and integration with AWS

Deployment types

• manifests
• helm
• NGINX Ingress Operator
• Multiple Ingress Controllers

Deployment types described here https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-operator/

Deployment through helm

Adding the Helm Repository

$ helm repo add nginx-stable https://helm.nginx.com/stable
$ helm repo update

Installing via Helm Repository

For installing the nginx chart with the release name nginx-ingress need to execute:

$ helm install nginx-ingress nginx-stable/nginx-ingress

You can check status of the deployment using kubectl console

$  kubectl -n ingress-nginx get all
NAME                                 READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-5b4c47cd8b-cxvpg   1/1     Running   0          51s
pod/nginx-ingress-5b4c47cd8b-djbwg   1/1     Running   0          51s
pod/nginx-ingress-5b4c47cd8b-jp6fm   1/1     Running   0          51s
NAME                                  TYPE           CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)                      AGE
service/nginx-ingress-nginx-ingress   LoadBalancer   10.100.100.100   aws-url.region.elb.amazonaws.com   80:30322/TCP,443:32087/TCP   15m
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-ingress   3/3     3            3           15m
NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-ingress-5b4c47cd8b   3         3         3       15m

Custom config:

One of the advantage of the nginx-ingress is ability to use custom configs. This feature is implemented by configmaps.

Here is example of the configmap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-config
  namespace: nginx-ingress
data:
  main-template: |
    worker_processes  {{.WorkerProcesses}};
    ...
        include /etc/nginx/conf.d/*.conf;
    }
  ingress-template: |
    {{range $upstream := .Upstreams}}
    upstream {{$upstream.Name}} {
      {{if $upstream.LBMethod }}{{$upstream.LBMethod}};{{end}}
    ...
    }{{end}}
  virtualserver-template: |
    {{ range $u := .Upstreams }}
    upstream {{ $u.Name }} {
      {{ if ne $u.UpstreamZoneSize "0" }}zone {{ $u.Name }} {{ $u.UpstreamZoneSize }};{{ end }}
    ...
    }
    {{ end }}

Detailed information is here https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/custom-templates https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/

Deployment nginx-ingress and integration with AWS

Kubernetes nginx-ingress controller manages services using annotations.

Annotations

You can use Kubernetes annotations to attach arbitrary non-identifying metadata to objects. Clients such as tools and libraries can retrieve this metadata.

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

Config example for configuring ingress controller of your services is below:

annotations:
  kubernetes.io/ingress.class: nginx
  kubernetes.io/ingress.allow-http: "false"
  external-dns.alpha.kubernetes.io/ttl: '60'
  external-dns.alpha.kubernetes.io/hostname: ${url}
  nginx.org/proxy-body-size: "10m"
  nginx.org/client-body-buffer-size: "256k"
  nginx.org/redirect-to-https: "true"
  nginx.ingress.kubernetes.io/location-snippets: |
  nginx.org/server-snippets: |
    custom config or additional properties is here;
  nginx.org/proxy-buffering: "false"
  nginx.org/ssl-redirect: "true"
  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
  service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
  service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
  service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '120'
  service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
  service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "${acm_cert_arn}"

A couple of important annotations:

• kubernetes.io/ingress.class: nginx — defines that nginx controller should be used for the routing traffic to your service
• service.beta.kubernetes.io/aws-load-balancer-ssl-cert:${cert_arn} — that is certificate arn from AWS Certificate manager that will be using
• nginx.ingress.kubernetes.io/location-snippets: and nginx.org/server-snippets: — shows where you can put in your custom config or additional properties
• service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https — configures AWS Load Balancer protocol(http/https) of the service/pod traffic redirects to.

Other annotations options are explained in the official nginx documentation of the controller.

Nginx controller supports custom annotations as well. You can see details here https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/custom-annotations/

Summary

According to the explanation of the current topic we can define a few pros and cons for the nginx-ingress controller that developing by nginx team. There are few of them:

Advantages:

• different type of deployments
• developing by nginx team
• ability to add custom config/properties of the nginx service
• native integration with AWS (ALB and EKS)
• apply new nginx config on fly

Disadvantages:

• complex configuration
• a lot of documentation
• need a lot of background knowledge
• difficult debugging

Investigation shows idea and examples of using nginx-ingress kubernetes controller that provided by nginx team. It helps a lot with managing services/ingress in AWS EKS cluster and expose them for the end users. Solution has native integration with AWS without additional configuration that helps deploy and easier maintain your project in scope of AWS.

URLs:

https://github.com/kubernetes/ingress-nginx
https://github.com/nginxinc/kubernetes-ingress
https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/nginx-ingress-controllers.md
https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/
https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/custom-templates
https://docs.nginx.com/nginx-ingress-controller/installation/
https://github.com/nginxinc/kubernetes-ingress/tree/master/deployments https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/