Run Ingress controller in KUBERNETES Cluster on AWS

Ekta Garg
Ekta Garg
Nov 16, 2018 · 7 min read
Load balancing solution for Kubernetes with Nginx

Hello Everyone. I am very excited to write about how to implement ingress controller in kubernetes because I implemented it successfully after my days of efforts and struggle. No worries, I’ll explain everything in detail

Kubernetes (commonly referred to as K8s) is a container-orchestration system. Kubernetes automates the deployment, scaling, maintenance, scheduling and operation of multiple containerized applications. It contains various tools for orchestration, secrets management, service discovery, scaling and load balancing.

Ingress Controller, an alternative to load balancers and allows inbound connections to the cluster. Using ingress, you can just run one load balancer and then can write your own rules so that it can be accessed from outside the cluster. You can use the ingress controller to reduce the cost of your load balancers and you can use only one load balancer that captures all the actual traffic and send it back to ingress controller.

The ingress controller can be configured to route the different traffic to all your apps based on HTTP rules. You can use ingress controller on HTTP and HTTPS enabled apps.

One great tool to enable this approach is External DNS. This tool automatically creates DNS records in your external DNS server(route53). For every hostname that you will use in ingress, it will create a new record in route53 to send the traffic to load balancer.

There are a lot of DNS providers like Google cloud DNS, AzureDNS, DigitalOcean etc but here in this blog, I’ll be using Route53.

HOW IT WORKS:

  1. So we’ll have two pods in kubernetes cluster: nginx controller in one pod and external DNS in another pod.
  2. There would be one ingress service and that service is of type load balancer. That service spins up AWS Loadbalancer.
  3. Next we are going to write ingress rules. like: helloworldv1.kubernetes.example.com=>pod1 helloworldv2.kubernetes.example.com=>pod2
  4. And these rules will be read by Nginx ingress controller. The external DNS will also read those rules and check whether there is a need to create DNS records.
  5. So External DNS will create those records in Route53. Then if a client comes in, its going to route53 to get the IP address. Route53 will provide the IP address and the IP address will be connect to load balancer.
  6. Then the load balancer will connect to ingress service. Ingress service will connect the traffic to Ingress controller. And ingress controller will see the rules to route this request.
Reference picture (Learn kubernetes: Edward Viaene)

Step by step implementation of HOW IT WORKS section:

  1. AWS IAM User policy update:

This will update IAM user policies so that creating, listing record set in route53 can work. Add below written code to iam-policy.sh file and run this:

run ./iam-policy.sh

#!/bin/bashDEFAULT_REGION="ap-south-1"
AWS_REGION="${AWS_REGION:-${DEFAULT_REGION}}"
NODE_ROLE="nodes.kubernetes.example.com"export AWS_REGIONaws iam put-role-policy --role-name ${NODE_ROLE} --policy-name external-dns-policy --policy-document '{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets"
],
"Resource": [
"arn:aws:route53:::hostedzone/*"
]
},
{
"Effect": "Allow",
"Action": [
"route53:ListHostedZones",
"route53:ListResourceRecordSets"
],
"Resource": [
"*"
]
}
]
}

2. Run your deployments and services

kubectl apply -f helloworld.yml

(There are two different approaches. kubectl create is what we call Imperative Management. On this approach, you tell the Kubernetes API what you want to create, replace or delete, not how you want your K8s cluster to look like. kubectl apply is part of the Declarative Management approach, where changes that you have applied to a live object (i.e. through scale) are maintained even if you apply other changes to the object.)

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: helloworld-deployment
spec:
replicas: 1
template:
metadata:
labels:
app: helloworld-v1
spec:
containers:
- name: k8s-demo
image: example.com/helloworld
ports:
- name: nodejs-port
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: helloworld-v1
spec:
type: NodePort
ports:
- port: 80
nodePort: 30305
targetPort: 80
protocol: TCP
name: http
selector:
app: helloworld-v1

Likewise, you can create as many deployments and services as your application modules require.

3. Create nginx-ingress-controller

As I explained in HOW IT WORKS section that we will have to create two pods so this will create first pod Nginx Controller.

kubectl apply -f nginx-ingress-controller.yml

# updated this file with the latest ingress-controller from https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: nginx-ingress-controller
spec:
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
spec:
serviceAccountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.17.1
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/echoheaders-default
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
labels:
app: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
---apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: default
---

4. Create a load balancer for ingress on the cluster

As explained above, here we are going to create an ingress service of type load balancer. Run this command :

kubectl apply -f ingress-lb.yml

kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
labels:
app: ingress-nginx
annotations:
# Enable PROXY protocol
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
# Increase the ELB idle timeout to avoid issues with WebSockets or Server-Sent Events.
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
# external-dns
external-dns.alpha.kubernetes.io/hostname: ingress.kubernetes.example.info
spec:
type: LoadBalancer
selector:
app: ingress-nginx
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 443
targetPort: https
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
labels:
app: ingress-nginx
data:
use-proxy-protocol: "true"

5. Create external DNS

Here we are now going to create the other pod for External DNS which is going to create a record set in Route53.

kubectl apply -f external-dns.yml

apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: external-dns
rules:
- apiGroups: [""]
resources: ["services"]
verbs: ["get","watch","list"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns
subjects:
- kind: ServiceAccount
name: external-dns
namespace: default
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: external-dns
spec:
strategy:
type: Recreate
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns
containers:
- name: external-dns
image: registry.opensource.zalan.do/teapot/external-dns:latest
args:
- --source=service
- --source=ingress
- --domain-filter=kubernetes.example.com # will make ExternalDNS see only the hosted zones matching provided domain, omit to process all available hosted zones
- --provider=aws
- --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization
- --aws-zone-type=public # only look at public hosted zones (valid values are public, private or no value for both)
- --registry=txt
- --txt-owner-id=kubernetes.example.com

6. Create ingress rules

Now we are going to write ingress rules. You will write as many rules as your total number of hosts. And here my host helloworldv1.kubernetes.example.com goes to helloworld-v1 service and helloworldv2.kubernetes.example.com goes to helloworld-v2 service. Apply this command:

kubectl apply -f ingress-rules.yml

# An Ingress with 2 hosts and 3 endpoints
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: helloworld-rules
spec:
rules:
- host: helloworldv1.kubernetes.example.com
http:
paths:
- path: /
backend:
serviceName: helloworld-v1
servicePort: 80
- host: helloworldv2.kubernetes.example.com
http:
paths:
- path: /
backend:
serviceName: helloworld-v2
servicePort: 80

You can now see running pods using “kubectl get pods”. You should be able to see echoheaders, nginx ingress controller, external dns, hosts (helloworldv1, helloworldv2).

We are almost done! Now go to the browser and enter the hostname helloworldv1.kubernetes.example.com and you should be able to see your application running on .

This is because the External DNS created the DNS record and it pointed to load balancer and load balancer pointed to ingress and ingress has all the correct rules.

So this is External DNS set up which you can use to have your DNS names automatically added.


Keep reading and appreciating for writing good articles. Cheers!

Ekta Garg

Written by

Ekta Garg

A feminist by nature, a developer by profession, and a blogger by passion. Follow me to follow my journey (only technical as of now)

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade