Using nginx-ingress controller to restrict access by IP (ip whitelisting) for a service deployed to a Kubernetes (AKS) cluster

While working on a project earlier this week we were given the following requirements :

  1. Create a Managed Kubernetes cluster (AKS) on Azure, in a existing Azure Subnet using an ARM template / az commands
  2. For a service deployed to the AKS cluster, restrict access to certain client source IPs (Ip Whitelisting)
  3. For a service deployed to the AKS cluster auto provision Letsencrypt TLS certificates. You can follow the post for this

This post details point 2 above. I will be creating a separate post for point 1 and 3 above, in the near future.

The rest of this post assumes that the AKS Kubernetes cluster is available, you have helm installed, and we have already executed the helm init command.

Create or update the nginx-ingress controller

The first thing we do now is install the inginx-ingress controller using helm. The github page for the nginx-ingress controller helm chart is at nginx-ingress. The install command to be used is :

Ingress controller install command

$ helm install stable/nginx-ingress  --set controller.service.externalTrafficPolicy=Local

The default value of controller.service.externalTrafficPolicy in the nginx ingress helm chart is ‘Cluster’, we need to change this value to ‘Local’. With the default value of ‘Cluster’ the ingress controller does not see the actual source ip from the client request but an internal IP. After setting this value to ‘Local’ the ingress controller gets the unmodified source ip of the client request.

Before we apply the ingress rule with source ip whitelisting for a service, let us create a sample web app deployment and service:

Create the the hello world web server deployment and service to test the whitelisting

Create Deployment

$ kubectl run web --image=tutum/hello-world --port=80 

Expose the deployment as a service

$ kubectl expose deployment web --port=80

Check if deployment service and pods are up

$ kubectl get deployment webNAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGEweb       1         1         1            1           17m$ kubectl get svc webNAME      CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGEweb   <none>        80/TCP    5m$ kubectl get pod -l=run=webNAME                   READY     STATUS    RESTARTS   AGEweb-5bff8ffd8c-twxwp   1/1       Running   0          17m

Apply the ingress IP Whitelisting rule for the service

The annotation ( )we need to apply to the kubernetes ingress resource using nginx-ingress is detailed at nginx-ingress.

We now create the ingress rule

Ingress YAML file

apiVersion: extensions/v1beta1
kind: Ingress
name: web-ingress
annotations: 49.36.X.X/32
# depending on the ingress controller version the annotation above may need to be modified to remove the prefix nginx. i.e. 49.36.X.X/32
- host:
- path: /
serviceName: web
servicePort: 80

Apply the ingress rule

$ kubectl apply -f web-ingress-whitelist.yaml

Thats it whitelisting is in place.

Testing and Debugging the whitelisting rules

Before steps below were executed dns configurations were modified so that the domain was pointing to the public ip of the nginx-ingress controller service.

Request from whitelisted ip

$ curl
$ curl -I
HTTP/1.1 200 OK
Server: nginx/1.13.5
Date: Sat, 19 May 2018 06:07:01 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/5.6.14

Request from Non Whitelisted ip

$ curl$ curl -I
HTTP/1.1 403 Forbidden
Server: nginx/1.13.5
Date: Sat, 19 May 2018 06:15:47 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive

Checking nginx-ingress controller logs

$ kubectl get pods
nginx-ingress-nginx-ingress-controller-586c47b885-rxm72 1/1 Running 0 1d
nginx-ingress-nginx-ingress-default-backend-65f4cd97fb-sbh7c 1/1 Running 0 1d
$ kubectl logs nginx-ingress-nginx-ingress-controller-586c47b885-rxm72 -f..
49.36.X.X - [49.36.X.X] - - [19/May/2018:06:11:21 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.52.1" 87 0.002 [default-web-80] 0 0.002 200

Cloud, Containers, K8s, DevOps | CKA | LFCS | Principal Software Engineer @ Microsoft