Experimenting with Ingress Controllers on Oracle Container Engine (OKE) — Part 1
In a previous post, I briefly mentioned creating a Kubernetes Ingress and IngressController but did not explain much about it. In this post, I’ll briefly explain what they are and then run an experiment with 4 popular Ingress Controllers on OKE.
Ingress and Ingress controllers
You can read more about them in this excellent post or HAProxy’s but I’ll give you a summary here. A Kubernetes Ingress is essentially a set of rules that handle external requests to access services in the Kubernetes cluster.
Think of the Ingress Controller as a router but with a bit of intelligence. Whenever a request is received, it routes the request according to rules such as a hostname or a URL path or a combination of both. The Ingress Controller listens to the Kubernetes API for Ingress resources and routes these requests to the pods according to these rules. Essentially, an Ingress Controller is a system that is able to do reverse proxying.
These rules are specified in an Ingress (see this post by Jay Gorrell for a more detailed explanation.
The controller obtain the endpoints from the service specified in the Ingress in the form of POD_IP: PORT_NUMBER.
kubectl get endpoints acme-website
NAME ENDPOINTS AGE
acme-website 10.244.0.9:80,10.244.1.16:80,10.244.2.3:80 25hLet’s look at an example.
The diagram above illustrates 4 ingress rules:
- Website: which uses only the hostname rule and handles http://www.acme.com/
- Blog: which uses the hostname and the path rules and handles http://www.acme.com/blog
- Support: which also uses only the hostname rule, albeit a different hostname from the website and uses http://support.acme.com
- Default: This is to handle requests that do not have a matching rule
There are many Ingress Controllers and in the rest of this post, we will take some of them for a spin on OKE. This is not recommending one over the other. Like every software, you need to do your own evaluation. Sometimes, you might even end up having to use more than one.
The Ingress Controllers that we will be looking at are the following:
Deploying ACME website and blog
As we will be reusing the ACME website and blog to test each controller, let’s deploy them first.
Clone this repo and apply the ACME manifests:
$ git clone https://github.com/hyder/okesamples/
$ cd okesamples
$ kubectl apply -f ingresscontrollers/acme/Verify that they have been correctly deployed. Get the list of pods:
kubectl get pods -o namepod/acme-blog-5b6cf84489-qrjsq
pod/acme-website-69596fd678-fzzvs
Use port-forwarding to access them to test they have been deployed properly:
kubectl port-forward acme-website-69596fd678-fzzvs 80:80You should now be able to access the website in your browser at http://localhost/
kubectl port-forward acme-blog-5b6cf84489-qrjsq 80:80You should now be able to access the blog in your browser at http://localhost/
Using nginx-ingress
nginx-ingress is an Ingress Controller based on the popular NGINX web server. It is open source, maintained by the kubernetes community, rich in features and also has a helm package. More importantly, it is well documented.
Installing it is quite straightforward and we will use the helm package that available.
helm install --name acmecontroller stable/nginx-ingress \
--set controller.name=acme \
--set defaultBackend.enabled=true \
--set defaultBackend.name=acmedefaultbackend \
--set rbac.create=trueOptions such as the defaultBackend.enabled and rbac.create are already enabled by default. The reason I am showing them here is to also contrast the effort of doing this manually or let helm do the hard work for you.
Verify that the LoadBalancer (by default, the controller service type is LoadBalancer which you can override if you do not want to) has been created:
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
acmecontroller-nginx-ingress-acme LoadBalancer 10.96.63.247 129.146.155.105 80:31166/TCP,443:31952/TCP 2m41s app=nginx-ingress,component=acme,release=acmecontroller
acmecontroller-nginx-ingress-acmebackend ClusterIP 10.96.177.110 80/TCP 2m41s app=nginx-ingress,component=acmebackend,release=acmecontrolle
r
kubernetes ClusterIP 10.96.0.1 443/TCP 3d2hEdit the ingresscontrollers/nginx/acme-website-ingress.yaml:
spec:
rules:
# replace www.acme.com with your FQDN
- host: www.acme.comEdit the ingresscontrollers/nginx/acme-blog-ingress.yaml:
spec:
rules:
# replace www.acme.com with your FQDN
- host: www.acme.com/blogCreate the Ingress for the website:
kubectl apply -f ingresscontrollers/nginx/acme-website-ingress.yamlCreate the Ingress for the blog:
kubectl apply -f ingresscontrollers/nginx/acme-blog-ingress.yamlingress.extensions/rewrite created
Add the DNS record for your domain as explained in the previous post and you can now access the website at http://www.acme.com/ and the blog at http://www.acme.com/blog (replace acme.com with your domain) and you should now be able to access both the website and the blog.
There are many more configuration options for the nginx controller. You can view them here.
Let’s clean up all the nginx-related deployment for now but keep all the acme deployment and services:
kubectl delete -f ingresscontrollers/nginx/
helm delete --purge acmecontrollerUsing HAProxy
haproxy-ingress is another Ingress Controller based on the popular HAProxy, the well-known HTTP/TCP Load Balancer. It is also open source, rich in features and also has a helm package.
As the haproxy-ingress chart is still in the incubator repository, we need to add the incubator repo first:
helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/We can now use the helm chart to install it.
helm install --name haproxycontroller incubator/haproxy-ingress \
--set controller.name=haproxycontroller \
--set defaultBackend.enabled=true \
--set defaultBackend.name=haproxydefaultbackend \
--set rbac.create=trueEdit the ingresscontrollers/haproxy/acme-website-ingress.yaml:
spec:
rules:
# replace www.acme.com with your FQDN
- host: www.acme.comEdit the ingresscontrollers/haproxy/acme-blog-ingress-2.yaml. This time, we want to use blog.acme.com to access the blog:
spec:
rules:
# replace www.acme.com with your FQDN
- host: blog.acme.comCreate the Ingress for the website:
kubectl apply -f ingresscontrollers/haproxy/acme-website-ingress.yamlCreate the Ingress for the blog:
kubectl apply -f ingresscontrollers/haproxy/acme-blog-ingress-2.yamlingress.extensions/acme-blog-ingress created
If you wish to deploy the blog under www.acme.com/blog, you can deploy using the acme-blog-ingress.yaml instead.
kubectl apply -f ingresscontrollers/haproxy/acme-blog-ingress.yamlSince a new OCI Load Balancer will be created when we deployed the HAProxy controller, we need to add first update the DNS ‘A’ Record for the website. Get the IP Address of the Load Balancer as described previously:
kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
haproxycontroller-haproxy-ingress-haproxycontroller LoadBalancer 10.96.68.165 129.146.148.61 80:32398/TCP,443:30754/TCP 38s app=haproxy-ingress,component=haproxycontroller,release=haproxycontroller
haproxycontroller-haproxy-ingress-haproxydefaultbackend ClusterIP 10.96.160.78 <none> 8080/TCP 38s app=haproxy-ingress,component=haproxydefaultbackend,release=haproxycontrollerSince we are also accessing the blog with a new host (blog.acme.com instead of www.acme.com/blog), we also need to add a new ‘A’ Record in OCI DNS for the blog:
You can use the instructions explained in the previous post to do that . Remember to click on “Publish Changes” button for the DNS changes to be effective. You can now access the website at http://www.acme.com/ and the blog at http://blog.acme.com/ (replace acme.com with your domain).
Let’s clean up all the haproxy-related deployment for now but keep all the acme deployment and services:
kubectl delete -f ingresscontrollers/haproxy/
helm delete --purge haproxycontrollerUsing Traefik
Traefik is a relatively newer (than HAProxy/Nginx) HTTP reverse proxy and load balancer that can also be used as an Ingress Controller. It is also open source, has a web UI, great documentation as well a helm package in the stable repo. Create a namespace for traefik:
kubectl create ns traefikInstall the Traefik controller using traefik’s helm package:
kubectl create ns traefikhelm install stable/traefik --namespace traefik \
--name traefikcontroller --set rbac.enabled=true \
--set kubernetes.ingressClass=traefik
We will discuss Ingress Class in Part 2.
Create the ingresses as before except this time use those in the traefik directory:
kubectl create -f ingresscontrollers/traefik/Remember to change the hostnames in the yaml files and update the DNS records for the new LoadBalancer and you can then test.
Let’s clean up before testing our last Ingress Controller:
kubectl delete -f ingresscontrollers/traefik/
helm delete — purge traefikcontrollerContour
The last controller we will test is Contour, an ingress controller based on Envoy. Contour does not have an official helm chart (although there is an unofficial one and a PR that has been submitted for the stable repo) yet so we will do the deployment manually.
Clone the contour git repo:
git clone https://github.com/heptio/contour.git
cd contour/examples/deployment-grpc-v2
kubectl apply -f .Contour can be deployed using either Deployment or DaemonSet. Other than describing what happens with each deployment option, Contour’s documentation doesn’t explain why or when you would use one over the other. Instead, I found this in Traefik’s documentation which also offers similar options. In this example, we will use the deployment. If you use the default yamls, contour will be deployed in the heptio-contour namespace. In order to get the IP address of the Load Balancer, you therefore have to specify the namespace when retrieving it:
kubectl get -n heptio-contour service contour -o wideNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
contour LoadBalancer 10.96.168.70 129.146.214.77 80:30082/TCP,443:31043/TCP 36s app=contour
Update the DNS ‘A’ Records as before (remember to change the hostnames) and then create the Ingresses.
kubectl create -f ingresscontrollers/contour/acme-website-ingress.yaml
kubectl create -f ingresscontrollers/contour/acme-blog-ingress.yamlThe website’s ingress will work fine. However, accessing the blog with http://www.acme.com/blog will not. To achieve this with Contour, you need to use Contour’s Ingress API (IngressRoute). My first IngressRoute looked like this:
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: acme-blog-ingress
labels:
app: acme-blog
spec:
virtualhost:
fqdn: www.acme.com
routes:
- match: /blog
services:
- name: acme-blog
port: 80This did not work either. A bit of googling and I landed on a similar blog post which recommended to add prefixRewrite. Turned out that this was also documented but a bit further down the IngressRoute page. Let’s add this:
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
name: acme-blog-ingress
labels:
app: acme-blog
spec:
virtualhost:
fqdn: www.acme.com
routes:
- match: /blog
prefixRewrite: "/"
services:
- name: acme-blog
port: 80Apply the IngressRoute again and this time you will be able to access the blog.
Let’s clean up once again for Part 2 where we will look at using multiple controllers concurrently.
kubectl delete -f ingresscontrollers/contour/
kubectl delete ns heptio-contourConclusion
The area of Ingress Controllers is experiencing a similar level of innovation as the Kubernetes networking. You can see those on Kubernetes Ingress Controller documentation.
The purpose of this post was to test a few of these Ingress Controllers and see how they would work with OKE so you don’t feel your choices are limited. In Part 2, we’ll look at how they can be customized as well as how they can work together if such a use case arises. I hope you find this useful.
Update:
My colleague Karthik Gaekwad suggested I add this link for those who need to chose: How to choose your Ingress Controller. It’s a concise and also very comprehensive list in terms of features and can help you narrow down your choice quickly.
