Mastering Ingress Context Path Based Routing in AWS EKS using Terraform
Where Paths are matched better than a Matchmaking Show !!!
Introduction:
Ingress context path-based routing is a concept commonly used in the context of Kubernetes and other container orchestration platforms. In Kubernetes, an Ingress is an API object that manages external access to services within a cluster. It acts as a traffic manager for incoming requests, directing them to the appropriate services based on rules.
In simple terms, context path-based routing involves a few key components. First, there’s the Kubernetes Ingress Object where we list out the paths for our services in the EKS Cluster. Think of this as a set of instructions for directing web traffic. Then, there’s the Ingress Controller, which acts like a manager — specifically, AWS Load Balancer Controller in this case. This manager handles the Ingress Objects we’ve set up and creates an AWS Application Load Balancer (ALB) with all the specified paths. The ALB, created by the Load Balancer Controller, sends incoming web traffic to the right services in the EKS Cluster, depending on the paths mentioned in the URL.
Note : This article assumes that readers have a basic understanding of AWS, Kubernetes, and Terraform and it does not provide any information on setting up an EKS Cluster in AWS.
Prerequisites :
- AWS Cloud Account
- EKS Cluster setup in AWS
- Ensure that Terraform, AWS CLI, and Kubectl are installed on the local system.
Installation of the AWS Load Balancer Controller on EKS with Terraform :
We’ll install the AWS Load Balancer Controller (LBC) in AWS EKS using Helm. From the Kubernetes Ingress perspective, it needs an Ingress Controller in the EKS Cluster to generate an AWS Application Load Balancer (ALB). While there are various Ingress Controllers like Nginx Ingress Controller and HAProxy Ingress Controllers, to specifically create an AWS ALB (Application Load Balancer), we require the installation of the AWS Load Balancer Controller in our cluster.
The AWS Load Balancer Controller (LBC) has the capability to create both Application Load Balancers (ALB) and Network Load Balancers (NLB) in AWS.
So, when we deploy a Kubernetes resource of type Ingress in an AWS EKS Cluster, the Load Balancer Controller recognizes it and creates an ALB in the AWS Cloud. Similarly, whenever we deploy a Kubernetes resource of type Service, the Load Balancer Controller generates an NLB in the AWS Cloud. Therefore, these Kubernetes objects, such as Ingress and Service, lead to the creation of corresponding AWS resources, ALB and NLB respectively, in the AWS Cloud. This functionality is achieved by installing the AWS Load Balancer Controller in the EKS Cluster.
Before you install the Load Balancer Controller in the EKS Cluster, there are certain prerequisites that need to be met.
First, in the AWS Cloud, we need to create an IAM Policy and attach it to an IAM Role. This IAM Policy should include the necessary permissions for the Load Balancer Controller to create both Application Load Balancers (ALB) and Network Load Balancers (NLB) in the AWS Cloud.
Additionally, we should create an AWS Load Balancer Controller (LBC) Service Account in the kube-system namespace of the EKS Cluster. This Service Account needs to be annotated with the IAM Role we created earlier.
When we deploy the Load Balancer Controller in the EKS Cluster, it will create three components, including the Load Balancer Controller Deployment, Load Balancer Controller webhook Cluster IP Service, and a Secret named AWS-Load-Balancer-TLS.
The Load Balancer Controller Deployment communicates with the LBC Service Account, providing the necessary permissions for the Load Balancer Controller to create ALB or NLB in AWS.
The Load Balancer Controller (LBC) keeps an eye on Kubernetes Ingress-related deployments in the Kubernetes API Server. So, when a Kubernetes administrator deploys an Ingress Manifest, the Load Balancer Controller detects this Ingress and creates an Application Load Balancer (ALB) in the AWS Cloud. In AWS terminology, it’s referred to as an ALB, while in Kubernetes terms, it’s known as an Ingress Resource.
The Load Balancer Controller (LBC) is a resource within the EKS Cluster. It gains the ability to create an ALB in the AWS Cloud by leveraging a Service Account that is annotated to an IAM Role. This is made possible through a concept known as IRSA (IAM Roles for Service Accounts).
We can automate the process of creating an IAM Policy and IAM Role, as well as the installation of the Load Balancer Controller (LBC) in the EKS Cluster, using Helm and Terraform.
In the AWS EKS Cluster, another prerequisite for installing the AWS Load Balancer Controller (LBC) is the presence of the IAM-OIDC-Connect-Provider. To implement IRSA (IAM Roles for Service Accounts), we need the IAM-OIDC-Connect-Provider resources to be available in the EKS Cluster.
To achieve this, we add the EKS Cluster as an OpenID Connect Provider in the AWS IAM Service. This step is crucial for implementing the IRSA concept.
The provided Terraform code automates the creation of an AWS IAM OpenID Connect Provider associated with an Amazon EKS cluster.
We have the option to create an IAM Policy in the AWS Cloud. There’s a corresponding GitHub repository for this policy. By utilizing the Terraform HTTP Data Provider, we can download and incorporate this policy into our Terraform configuration.
Following that, we need to generate an IAM Policy and IAM Role configured with “Assume Role With Web Identity.” Subsequently, we link the IAM Policy to the IAM Role. The provided Terraform code streamlines the creation of these IAM resources.
Once all the prerequisites are set up in the EKS Cluster, we can proceed to install the AWS Load Balancer Controller (LBC) using the Helm resource.
To achieve this, it’s essential to define the Helm Provider in our Terraform manifests. In the Helm release configuration, we need to specify the Helm chart repository URL and the name of the Helm chart to fetch the AWS Load Balancer Controller chart.
The installation should take place in the kube-system namespace. Additionally, we must include the image.repository parameter for the AWS Load Balancer Controller Docker image in the Helm release. The value for “image.repository” should comprise the AWS account ID and the ECR repository URL, with changes based on the region. Create a Service Account and annotate it with the IAM Role ARN.
Following that, we need to create an Ingress Class object.
In Kubernetes, an IngressClass serves as a means to define and distinguish between various Ingress controllers within a cluster. This enables users to specify which controller should manage the Ingress resources. It’s worth noting that multiple Ingress controllers can exist together in a cluster, and each Ingress resource can be linked to a specific Ingress Controller. It’s important to note that we cannot attach multiple Ingress Controllers to a single Ingress Class.
In our case, there’s only one Ingress Controller in our EKS Cluster, which is AWS Load Balancer Controller (LBC). We associate this LBC with an Ingress Class. Subsequently, we specify the name of this Ingress Class in the Kubernetes Ingress Manifest. This allows AWS Load Balancer Controller to recognize and handle the Kubernetes Ingress.
Verify the deployment of the Load Balancer Controller, Webhook ClusterIP Service, and Service Account in the EKS Cluster, by using kubectl commands :
The deployment of the Load Balancer Controller will create two Pods. By describing the Controller, you can obtain all the details, including the current Service Account being used and information about Liveness Probes.
In the Events Section, you’ll observe that the Replica Set is scaled up to 2. Check the pods created for the Load Balancer Controller, and review the logs for these pods. If there are no error logs, it indicates that the controller is functioning properly.
To check the Webhook ClusterIP Service, you can examine the services present in the Kube-system Namespace. Describe the service to see all the details of the Webhook Service. Check the “Selector Label,” ensuring that the value for the “Selector” is consistent in both the Load Balancer Controller (LBC) Deployment and the Webhook ClusterIP Service. You will also find the IP addresses of the AWS Load Balancer Controller Pods listed as values for “Endpoints.”
Inspect the Service Account created in the EKS Cluster using the following commands. The output is formatted in YAML, offering a comprehensive view of the service account. This includes metadata, associated secrets, and other configuration details.
By conducting these checks, we can confirm that the AWS Load Balancer Controller has been successfully installed, is operational, and has been configured correctly in your Amazon EKS cluster.
Kubernetes Ingress Object :
When creating an AWS Application Load Balancer (ALB) using the Ingress Manifest, you can define all the ALB settings within the Ingress Manifest using a parameter called Ingress Annotations. By referring to the AWS Load Balancer Controller documentation, you will discover a list of annotations (settings) specifically designed for configuring the AWS ALB.
In the Ingress Manifest, there’s another parameter called Ingress Class Name. Here, we specify the name of the Ingress Class that was created earlier. This specific Ingress Class is linked to the Load Balancer Controller (LBC). Consequently, when deploying this Ingress in the EKS Cluster it gets associated with the Load Balancer Controller. If there are multiple Ingress Controllers in the EKS Cluster, each Ingress will be linked to a specific controller using the Ingress Class Name.
Note: If we designate the Ingress Class in EKS Cluster as the default Ingress Class, there is no need to specify “Ingress Class Name” argument in the Ingress manifest. All Ingress manifests deployed will automatically be associated with the default Ingress Class.
The Ingress Spec is another crucial parameter in the Ingress manifest. In this parameter, we define the routing rules and provide information about the default backend.
Deploying Kubernetes Ingress manifests to Test the creation of ALB through Load Balancer Controller and Implement Context Path Based Routing in EKS :
When setting up an application in the Kubernetes cluster, you create a Deployment Manifest to install the app, a Service Manifest to make the app accessible to users, and an Ingress Manifest to direct incoming traffic from users to the app.
To test Context Path Based Routing, deploy multiple apps in the EKS cluster. Create separate “NodePort Services” for each app to make them accessible, and set up an Application Load Balancer (ALB) using an Ingress manifest to direct incoming traffic to the apps.
To implement this scenario, deploy multiple Nginx images in the EKS cluster. Create NodePort Services for each of these images. Subsequently, set up an Ingress manifest. This Ingress manifest will establish an Application Load Balancer (ALB) for the Nginx images, enabling access to these images through different paths.
The Ingress manifest for Nginx images includes annotations for AWS Application Load Balancer (ALB) settings. We’ve specified the Ingress Class Name to associate it with the Load Balancer Controller (LBC). Additionally, we’ve defined paths through which users can access Nginx images. These paths are taken by the ALB, and rules will be created in the ALB’s Target Groups. This is how we can create an ALB (Application Load Balancer) in AWS.
Check if the Application Load Balancer (ALB) is created in AWS, and verify if the Nginx images can be accessed through the ALB endpoint :
Utilize kubectl commands to view the pods for the Nginx application, examine the NodePort services generated in the cluster, and inspect the Ingress configuration within the cluster.
When you list the Ingress service, you can find the endpoint of the Application Load Balancer (ALB) under the “Address” attribute. This indicates that an ALB is successfully created in the AWS cloud.
If you paste this address into the browser, you can access the Nginx web page. Additionally, if you append the path “/latest” after the endpoint, you can access another Nginx image that has been deployed.
In the AWS Console, navigate to the Load Balancer section and locate the load balancer created by the Ingress. Within the ALB, go to the Listener settings, where you can find the rules (paths) specified in the Ingress manifest. These rules are configured to route traffic to their respective target groups.
Conclusion :
Implementing Ingress context path-based routing on AWS EKS using Terraform brings a powerful way to manage and organize your applications. By directing traffic based on specific paths enhances the overall accessibility and functionality of your applications in AWS EKS Cluster !!!