Ingress Control in Google Kubernetes Engine: Part One

Published in

The Blog of RETINA-AI Health, Inc.

8 min readApr 8, 2020

Due to the ongoing COVID19 pandemic, much of the world has implemented social distancing measures, and video conferencing has become a necessary means of continuing business during these trying times. For example, Telehealth and remote AI diagnostic systems have suddenly taken center stage in healthcare. However, of late there has been a spate of security breaches in video conferences around the world. And though most have been pedestrian scrapes of video conference URLs from social media, others supposedly have been more sophisticated; including breaks into the backend of video conferencing service clusters. One way for vendors to mitigate against such security vulnerabilities is to decrease the attack surface of their microservice clusters. In particular, by ensuring that one’s cluster is only accessible through a single gateway, one significantly decreases the chances of a malicious agent gaining access into the cluster. This is in contrast to clusters with multiple potential entry points which can be exploited by bad actors. One way to achieve a single entry point and tighten security is via an ingress gateway device.

Ingress is a mechanism through which traffic external to a cluster is admitted into the cluster in a managed way. Ingress can provide a number of functions including TLS termination, and URL-based routing. There are two components to ingress: the ingress object and the ingress controller. The ingress object is specified as a manifest YAML, while the ingress controller is an application typically installed and running in a pod on the cluster. The ingress specifies the rules of our traffic entry management, while the ingress controller is a control loop that communicates with the components of our cluster and implements the rules in the ingress object. There are several types of ingress controllers including NGINX, NGINX+, HAProxy, Traefik, and the Google Kubernetes Engine (GKE) ingress controller. When working in GKE, the Google Cloud Load Balancer (GCLB) is installed automatically at cluster creation. In what follows, I describe setting up ingress on a two microservices cluster in GKE, in accordance with the documentation: GKE Ingress for HTTP(S) Load Balancing.

Sample Web Applications

We will be using the following two flask web applications in this tutorial. The “yoruba-hello” app prints an unabridged “hello world” in your browser in the yoruba language, while the “english-hello” app prints an unabridged “hello world” in your browser in english. Their codes are as shown below:

I built the above apps into a docker image and pushed the images into google container registry (gcr.io). I’ve demonstrated how to do this in my previous tutorials here and here. In this tutorial, we will simply be calling the apps by their gcr.io addresses from inside our deployment YAMLs.

Setting up Ingress in GKE

To setup an ingress in GKE one needs to prepare the following YAMLs:

Deployment YAMLs
Service YAMLs of type NodePort
Ingress Object YAML

Illustration of Ingress into a Kubernetes Cluster

Deployments

The deployment YAML of our english hello app is as follows:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: english-deployment
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: english
    spec:
      containers:
        - name: english-container
          image: gcr.io/ml-proj-25721/english-hello
          ports:
            - containerPort: 8080
              protocol: TCP

And the deployment YAML of our yoruba hello app is as follows:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: yoruba-deployment
  namespace: default
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: yoruba
    spec:
      containers:
        - name: yoruba-container
          image: gcr.io/ml-projn-25721/yoruba-hello
          ports:
            - containerPort: 8080
              protocol: TCP

In the above manifests, the Kind tells us it is a deployment, the name identifies the manifest, replicas tells how many pods we want to have, and the label app is the key field via which we will map this deployment with its associated service. Under the container fields, the image field contains the registry address of the docker container containing our app. Ports and network protocols are also specified as shown.

A quick check shows us we do not have any deployments on this cluster:

We create deployment for both apps using the command: kubectl create -f

yielding the following:

Of note, both our deployments are now up and running, however, they are not accessible. To make them accessible we need to associate a service with them. In particular, given that our end goal is the creation of an ingress, we need to associate a NodePort service with each deployment. Of note, the IP type of nodeport services is Cluster IP and hence only reachable from inside the cluster. This is essential to keeping the services secure and only reachable through our controlled ingress device.

Services

Now, let’s create two NodePort Services. The english hello app service YAML is as follows:

apiVersion: v1
kind: Service
metadata:
  name: english-service
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: english
  type: NodePort

And the yoruba hello app service YAML is as follows:

apiVersion: v1
kind: Service
metadata:
  name: yoruba-service
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: yoruba
  type: NodePort

In the above YAMLs, the kind is Service, the name field specifies the service name with which we will bind each service to the ingress object below. The port field specifies the service’s port, while the targetPort specifies the port on which the container containing our app is running. The app field binds the service to the deployment. i.e. app value must match exactly in our deployment and corresponding service YAMLs. The type specifies service type, NodePort in this case.

Next, observe in this demonstration that there are no services other than the master on my cluster:

Now, let’s create two the NodePort services, using the commands,

kubectl create -f yoruba-service.yaml and kubectl create -f english-service.yaml respectively:

Examining our services, we now see that both the english and yoruba services have been created. And under ports, we see the nodeport numbers they have each been assigned:

Take careful note of the next step. You have to explicitly set firewall rules to open the nodeports to internet traffic. When one creates a LoadBalancer service in GKE, the firewall rules are automatically configured by default to allow traffic in. However for NodePort services, the ports are closed by default and need to be explicitly opened. Not doing so in the right order means the ingress will not work. Problems often arise when one does not open up the ports before creating the ingress. To open the ports 30697 and 30582, we create a firewall rule as follows:

$ gcloud compute firewall-rules create our-rules --allow tcp:30697,tcp:30582

Next we are ready to create the ingress.

Ingress

The ingress object is as shown below:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: trilingress
  annotations:
    kubernetes.io/ingress.class: "gce"spec:
  rules:
  - http:
      paths:
      - path: /english
        backend:
          serviceName: english-service
          servicePort: 8080
      - path: /yoruba
        backend:
          serviceName: yoruba-service
          servicePort: 8080

In the above ingress manifest, we see that the kind is ingress, the name is “trilingress,” and we have annotated the ingress controller class as “gce” (Google Compute Engine). This annotation is important for scenarios in which there are more than one ingress controllers on the cluster, e.g NGINX, KONG, etc, each looping in search for ingress rules to implement. to avoid conflicts, we therefore need to specify which of the controllers our manifest is intended for. In the specs we see specification of the URL paths and associated services. These service names correspond exactly to the “names” in the service YAMLs. And servicePort corresponds exactly to the port field in the service YAML.

Next we check to see what if any ingresses already exist on this cluster. There are none:

Now, we create an ingress

It takes as while to complete as we see above. Also in console, if we navigate to Kubernetes Engine>>Services & Ingress>>Ingress , we see the following in process:

And the kubectl describe ingress command shows more details in process:

After a bit of a wait (typically 1–2 minutes), the ingress formation is complete and we see the green OK mark in console:

And all services are in a healthy state:

Now, navigating to the respective URLs yields the following:

English Greeting Service. Accessed from my Phone.

Yoruba Greeting Service. Accessed from my Phone.

SUCCESS!!

Conclusion

In this tutorial we have seen how to use the ingress to implement URL-based routing to the various microservices in our cluster. This helps with security by decreasing our attack surface, but is only one part of web security. In part two of this tutorial, we will see how use ingress to implement TLS termination to ensure that the connection between the client and our service is secured. We will also see how to use ingress to implement human-readable custom domain name, instead of an IP address number.

BIO: Dr. Stephen G. Odaibo is CEO & Founder of RETINA-AI Health, Inc, and is on the Faculty of the MD Anderson Cancer Center. He is a Physician, Retina Specialist, Mathematician, Computer Scientist, and Full Stack AI Engineer. In 2017 he received UAB College of Arts & Sciences’ highest honor, the Distinguished Alumni Achievement Award. And in 2005 he won the Barrie Hurwitz Award for Excellence in Neurology at Duke Univ School of Medicine where he topped the class in Neurology and in Pediatrics. He is author of the books “Quantum Mechanics & The MRI Machine” and “The Form of Finite Groups: A Course on Finite Group Theory.” Dr. Odaibo Chaired the “Artificial Intelligence & Tech in Medicine Symposium” at the 2019 National Medical Association Meeting. Through RETINA-AI, he and his team are building AI solutions to address the world’s most pressing healthcare problems. He resides in Houston Texas with his family

REFERENCES
S.G. Odaibo, Machine Learning in Production: Serving Up Multiple ML Models at Scale the TensorFlow Serving + Kubernetes + Google Cloud + Microservices WayS. G. Odaibo, Docker Containers, Python Virtual Environments, & Virtual MachinesGoogle, GKE Ingress Documentation