DevSecOps Three-Tier Project using AWS EKS, ArgoCD, Prometheus, Grafana, and Jenkins

Overview:
Embark on an immersive journey through the intricacies of modern DevOps practices, as we meticulously guide you in constructing a robust Three-Tier architecture on AWS. This project encompasses a spectrum of industry-leading tools, including AWS EKS, ArgoCD, and Jenkins, bringing together development, security, and operations seamlessly.

Key Features Covered:

1. IAM User Empowerment:
   — Establish a fortified foundation by creating a purpose-built IAM user on AWS, ensuring precise permissions for seamless deployment and management.

2.Infrastructure as Code (IaC):
— Utilize Terraform and AWS CLI to orchestrate the deployment of a Jenkins server (EC2 instance), showcasing the art of infrastructure creation.

3. Jenkins Configuration Excellence:
— Configure the Jenkins server with precision, integrating vital tools such as Jenkins, Docker, Sonarqube, Terraform, Kubectl, AWS CLI, and Trivy to create a sophisticated DevOps environment.

4. EKS Cluster Deployment Prowess:
— Ascend to Kubernetes mastery using eksctl commands, creating an Amazon EKS cluster, a managed Kubernetes service on AWS.

5. Optimized Load Balancer Configuration:
— Implement AWS Application Load Balancer (ALB) configurations to streamline traffic within the EKS cluster, ensuring optimal user experiences.

6. Secure Docker Image Repositories:
— Establish secure Docker image repositories using Amazon Elastic Container Registry (ECR) for both frontend and backend components.

7. ArgoCD for GitOps Excellence:
— Implement and configure ArgoCD, embracing the principles of continuous delivery and GitOps for streamlined application deployment.

8. Sonarqube Integration for Code Quality Assurance:
— Seamlessly integrate Sonarqube into the DevSecOps pipeline, ensuring rigorous code quality analysis and adherence to industry standards.

9. Jenkins Pipelines Mastery:
— Craft sophisticated Jenkins pipelines for the seamless deployment of backend and frontend code to the EKS cluster, showcasing proficiency in CI/CD methodologies.

10. Comprehensive Monitoring Setup:
— Deploy Helm, Prometheus, and Grafana to establish an extensive monitoring infrastructure for the EKS cluster, enabling insightful performance tracking.

11. ArgoCD Application Deployment Expertise:
— Utilize ArgoCD to orchestrate the deployment of the Three-Tier application, encompassing database, backend, frontend, and ingress components with precision.

12. DNS Configuration for Seamless Accessibility:
— Configure DNS settings adeptly to provide seamless access to the application through custom subdomains, ensuring a polished user experience.

13. Data Persistence Strategies:
— Implement robust strategies for persistent volume and claims, guaranteeing the continuity of database pods and safeguarding critical data.

14. Conclusion and Performance Monitoring:
— Conclude the project with a comprehensive summary of achievements and establish ongoing performance monitoring of the EKS cluster through Grafana.

Dive deep into the world of DevOps excellence, and emerge with the skills to architect, deploy, and secure scalable applications on AWS.

Step 1: Creating an IAM User and Generating AWS Access Key

Managing access and permissions is paramount to maintaining a secure and efficient cloud infrastructure. One fundamental step in this process is creating an IAM (Identity and Access Management) user and generating an AWS Access Key. Follow these meticulous steps to ensure a seamless setup for testing purposes, though exercise caution when assigning such privileges in a production environment.

Navigate to the AWS IAM Service and locate the “Users” section.

1. Click on “Create User” to initiate the user creation process.

Assign a meaningful name to your user and proceed by clicking “Next.”

Opt for the “Attach policies directly” option and search for “Administrator Access.” Select it to grant the user comprehensive administrative permissions. Exercise caution, as providing such access is not recommended for organizational projects.

Click “Next” to move forward in the setup process.

Confirm your selections and create the user by clicking on “Create User.”

1. Once the user is created, select it and navigate to the “Security Credentials” section. Generate an access key by clicking on “Create Access Key.”

Choose “Command Line Interface (CLI)” as the key type, confirm your selection, and proceed to the next step.

Provide a meaningful description for the access key and finalize the process by clicking “Create Access Key.”

Your credentials are now visible, and you have the option to download a CSV file for future reference.

Step 2: Installing Terraform & AWS CLI for Jenkins Server Deployment on AWS

In order to seamlessly deploy a Jenkins Server on the AWS Cloud, it is imperative to have Terraform and AWS CLI installed and configured on your local machine. Follow these step-by-step instructions to ensure a smooth setup.

Terraform Installation Script:

Use the following commands to install Terraform:

bashCopy codewget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update
sudo apt install terraform -y

AWS CLI Installation Script:

Execute the commands below to install AWS CLI:

bashCopy codecurl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt install unzip -y
unzip awscliv2.zip
sudo ./aws/install

Configuring Terraform:

1. Edit the /etc/environment file using the following command:

sudo vim /etc/environment

For Windows, the process to set environment variables is slightly different as compared to Linux. Here’s how you can do it:

Open Environment Variables Window:

Right-click on the Start button and select “System.”

In the System window, click on “Advanced system settings” on the left.

In the System Properties window, click on the “Environment Variables” button.

Edit or Add Variables:

In the Environment Variables window, you will see two sections: User variables and System variables.

To add or edit a variable for all users, choose “System variables.” To set it only for your user account, choose “User variables.”

Click on the “New” button to add a new variable or select an existing one and click on the “Edit” button.

Add Terraform Environment Variables:

For Terraform, you typically need to set the PATH variable to include the directory where Terraform is installed.

Click on “New” or select “Path” and click on “Edit.”

Add the path to the Terraform installation directory (e.g., C:\Program Files\Terraform) to the end of the existing values.

Save Changes:

Click “OK” in each window to save the changes.

Verify the Installation:

Open a new Command Prompt or PowerShell window and type terraform --version to ensure that Terraform is recognized.

Configuring AWS CLI:

Run the following command to configure AWS CLI:

aws configure

Input your AWS Access Key ID, Secret Access Key, preferred region, and output format when prompted.

Step 3: Deploying Jenkins Server (EC2) with Terraform

Clone the Git repository: laxman-kandhukuri/Three-Tier-DevSecOps (github.com)

Navigate to the Jenkins-Server-TF directory.

If you want use remote terraform server with S3, Modify the backend.tf file, changing the bucket name and DynamoDB table (both created manually on AWS).

Replace the PEM file name with your specific PEM file name used on AWS.

Initialize the Terraform backend:

terraform init

Validate the syntax:

terraform validate

Check the blueprint of AWS services to be created:

terraform plan -var-file=variables.tfvars

Deploy the infrastructure on AWS:

terraform apply -var-file=variables.tfvars --auto-approve

Connect to your Jenkins-Server by clicking on “Connect.”

Copy the SSH command and paste it on your local machine.

Step 4: Configuring Jenkins

Log into your Jenkins server.

Ensure the installation of services like Jenkins, Docker, Sonarqube, Terraform, Kubectl, AWS CLI, and Trivy:

jenkins --version
docker --version
docker ps
terraform --version
kubectl version
aws --version
trivy --version
eksctl --version

Configure Jenkins by copying the public IP of your Jenkins Server and accessing it on your browser with port 8080.

Click on “Install suggested plugins.”

Allow the installation of plugins and continue as admin.

Fill details then save and finish, then start using Jenkins.

1. The Jenkins Dashboard should display as expected.

Step 5: Deploying EKS Cluster using eksctl Commands

Return to your Jenkins Server terminal to configure AWS.

Navigate to “Manage Jenkins.”

Click on “Plugins.”

Select the available plugins and install the following plugins:

• AWS Credentials
• Pipeline: AWS Steps

After installing both plugins, restart your Jenkins service by selecting the “Restart Jenkins” option.

Log in to your Jenkins Server again.

Setting up AWS Credentials in Jenkins:

Navigate to “Manage Jenkins” and click on “Credentials.”

Click on “Global Credentials.”

Select “AWS Credentials” as the kind and add the ID, AWS Access Key, and Secret Access Key. Click on “Create.”

1. The credentials should resemble the example shown in the snippet.

Now, add GitHub credentials:

• In the same “Manage Jenkins” > “Credentials” section, click on “global.”
• select Username and password type.
• Add the username and personal access token of your GitHub account.

Both credentials will look like this.

By meticulously configuring AWS and GitHub credentials in Jenkins, you ensure secure and seamless integration with these platforms, allowing for a smooth deployment process for your EKS Cluster.

Step 6: Configuring Load Balancer on EKS for Ingress Controller

Create an eks cluster using the below commands.

eksctl create cluster --name Three-Tier-K8s-EKS-Cluster --region ap-south-1 --node-type t2.medium --nodes-min 2 --nodes-max 2
aws eks update-kubeconfig --region ap-south-1 --name Three-Tier-K8s-EKS-Cluster

You can change name and region as per you.

Validate the node readiness with

kubectl get nodes

Download the policy for LoadBalancer prerequisites

curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.5.4/docs/install/iam_policy.json

Create the IAM policy:

aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy --policy-document file://iam_policy.json

Create OIDC Provider:

eksctl utils associate-iam-oidc-provider --region=ap-south-1 --cluster=Three-Tier-K8s-EKS-Cluster --approve

Create Service Account:

eksctl create iamserviceaccount --cluster=Three-Tier-K8s-EKS-Cluster --namespace=kube-system --name=aws-load-balancer-controller --role-name AmazonEKSLoadBalancerControllerRole --attach-policy-arn=arn:aws:iam::<your_account_id>:policy/AWSLoadBalancerControllerIAMPolicy --approve --region=ap-south-1

Deploy the AWS Load Balancer Controller:

sudo snap install helm --classic
helm repo add eks https://aws.github.io/eks-charts
helm repo update eks
helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=my-cluster --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller

After 2 minutes, check the status of the deployed pods:

kubectl get deployment -n kube-system aws-load-balancer-controller

These steps guide you through the process of setting up and configuring a Load Balancer on your EKS cluster

Step 7: Creating Amazon ECR Private Repositories for Frontend and Backend Tiers

Click on “Create repository” in the Amazon ECR (Elastic Container Registry) console.

1. Select the “Private” option to configure the repository settings and click on “Save.”

Repeat the process for the backend repository and click on “Save.”

Now, your ECR Private Repositories are set up.

1. Configure ECR locally to enable image uploads:

Copy the first command provided for login. Now, run the copied command on your Jenkins Server.

Step 8: Installing and Configuring ArgoCD

Create a dedicated namespace for our three-tier application on EKS:

kubectl create namespace three-tier

As our ECR repositories are private, create a Kubernetes secret to handle ECR authentication:

kubectl create secret generic ecr-registry-secret \
  --from-file=.dockerconfigjson=${HOME}/.docker/config.json \
  --type=kubernetes.io/dockerconfigjson --namespace three-tier
kubectl get secrets -n three-tier

Install ArgoCD by creating a separate namespace and applying the ArgoCD configuration for installation:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.7/manifests/install.yaml

Validate the installation by ensuring all pods are running:

kubectl get pods -n argocd

Expose the ArgoCD server as a Load Balancer:

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

Confirm the Load Balancer creation in the AWS Console.

Access ArgoCD by copying the Load Balancer DNS and entering it in your browser.

Note: You might encounter a warning; proceed by clicking “Advanced” and then the link appearing under “Hide advanced.”

Obtain the password for the ArgoCD server:

Install jq if not already installed:

sudo apt install jq -y

Run the following commands to export ArgoCD server and password:

export ARGOCD_SERVER=$(kubectl get svc argocd-server -n argocd -o json | jq -r '.status.loadBalancer.ingress[0].hostname')
export ARGO_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
echo $ARGO_PWD

Use the provided username and the obtained password to sign in to ArgoCD.

Step 9: Configuring Sonarqube for DevSecOps Pipeline

Copy your Jenkins Server public IP and access it on your preferred browser using port 9000. Log in with the default credentials: username — admin, password - admin.

1. Update the password for the admin account.

Click on “Administration,” then “Security,” and select “Users.”

Click on “Update tokens,” then “Generate.”

Copy the generated token, keep it secure, and click on “Done.”

Configure webhooks for quality checks:

Click on “Administration,” then “Configuration,” and select “Webhooks.”

Click on “Create.”

Provide a project name, and in the URL, enter your Jenkins Server public IP with port 8080 and add sonarqube-webhook in the suffix.

Click on “Create.”

Example URL: http://<jenkins-server-public-ip>:8080/sonarqube-webhook/

Verify the creation of the webhook.

Create a Sonarqube Project for frontend code:

Click on “Manually.”

Provide a display name for your project and click on “Setup.”

Click on “Locally.”

Select “Use existing token” and click on “Continue.”

Choose “Other” and set OS as Linux.

Create a Sonarqube Project for backend code:

Click on “Create Project.”

Provide a project name and click on “Set up.”

Click on “Locally.”

Select “Use existing token” and click on “Continue.”

Choose “Other” and set OS as Linux.

Store Sonarqube credentials:

Navigate to jenkins Dashboard -> Manage Jenkins -> Credentials.

Select the kind as “Secret text,” paste your token in “Secret,” and keep other settings as is.

Click on “Create.”

Adding Jenkins Credentials for ECR Repo URI

Add AWS Account ID as Jenkins Credentials:

Navigate to Jenkins Dashboard -> Manage Jenkins -> Credentials.

Select the kind as “Secret text,” paste your AWS Account ID in “Secret,” and keep other settings unchanged.

Click on “Create.”

Add ECR Image Name for Frontend as Jenkins Credentials:

Navigate to Dashboard -> Manage Jenkins -> Credentials.

Select the kind as “Secret text,” paste your frontend repo name in “Secret,” and keep other settings unchanged.

Click on “Create.”

Add ECR Image Name for Backend as Jenkins Credentials:

Navigate to Dashboard -> Manage Jenkins -> Credentials.

Select the kind as “Secret text,” paste your backend repo name in “Secret,” and keep other settings unchanged.

Click on “Create.”

Step 10: Installing and Configuring Required Jenkins Plugins

Install the following plugins:

Navigate to Dashboard -> Manage Jenkins -> Plugins -> Available Plugins.Install the following plugins:

Docker
Docker Commons
Docker Pipeline
Docker API
docker-build-step
Eclipse Temurin installer
NodeJS
OWASP Dependency-Check
SonarQube Scanner

Configure Installed Plugins:

Go to Dashboard -> Manage Jenkins -> Tools.

Configure JDK:

Search for JDK and provide the configuration as shown below.

Configure SonarQube Scanner:

Search for the SonarQube Scanner plugin and configure it as shown below.

Configure NodeJS:

Search for NodeJS and configure it as shown below.

Configure OWASP Dependency Check:

Search for Dependency-Check and configure it as shown below.

1. Configure Docker:
2. Search for Docker and configure it as shown below.

click on “Apply & Save.”

Set the Path for SonarQube in Jenkins:

Go to Dashboard -> Manage Jenkins -> System.

Search for SonarQube installations.

Provide the name as it is, then in the Server URL, copy the SonarQube public IP (same as Jenkins) with port 9000. Select the Sonar token added recently and click on “Apply & Save.”

Creating Jenkins Pipeline for Backend Code Deployment

Note: Before pipeline build change some details in source code and make sure that you update ECR on Kubernetes/Frontend/deployment.yaml and /Kubernetes/Backend/deployment.yaml in frontend and backend floder.

Go to the Jenkins Dashboard.

Click on “New Item.”

Provide the name of your Pipeline and click on “OK.”

Under the “Pipeline” section, choose “Pipeline script from SCM” in the Definition dropdown.

Choose your preferred SCM system (e.g., Git) and provide the repository URL.

Specify the branch to build or leave it blank to build the default branch.

In the “Script Path” field, provide the path to your Jenkinsfile for the backend. For example: Jenkinsfile-Backend

Click on “Apply & Save.”

Note: Adjust the pipeline script and Jenkinsfile according to the specific requirements of your project.

Trigger the build by clicking on the build Now.

Ensure the pipeline runs successfully after addressing any common mistakes.

Creating Jenkins Pipeline for Frontend Code Deployment

Go to the Jenkins Dashboard.

Click on “New Item.”

Provide the name of your Pipeline and select the “Pipeline” type.

Under the “Pipeline” section, choose “Pipeline script from SCM” in the Definition dropdown.

Choose your preferred SCM system (e.g., Git) and provide the repository URL.

Specify the branch to build or leave it blank to build the default branch.

In the “Script Path” field, provide the path to your Jenkinsfile for the frontend. For example: Jenkinsfile-Frontend

Click on “Apply & Save.”

1. Trigger the build by clicking on the build Now.

Ensure the pipeline runs successfully after addressing any common mistakes.

Note: Adjust the pipeline script and Jenkinsfile according to the specific requirements of your project.

Setting Up Monitoring for EKS Cluster

We will achieve the monitoring using Helm
Add the prometheus repo by using the below command.

helm repo add stable https://charts.helm.sh/stable

Install Prometheus from the Prometheus Community repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install stable prometheus-community/kub-prometheus-stack

Verify the service:

kubectl get svc

Access Prometheus and Grafana consoles from outside the cluster:

Change the Service type from ClusterType to LoadBalancer for Prometheus and Grafana services:

Edit the stable-kube-prometheus-sta-prometheus service

kubectl edit svc stable-kube-prometheus-sta-prometheus

Modify the service types from ClusterType to LoadBalancer.

Edit the stable-grafana service

kubectl edit svc stable-grafana

Modify the service types from ClusterType to LoadBalancer.

Verify the LoadBalancer DNS names:

kubectl get svc

You can also validate from your console.

Access Prometheus Dashboard:

Open <Prometheus-LB-DNS>:9090 in your browser.

Click on “Status” and select “Targets” to view available targets.

Access Grafana Dashboard:

Copy the ALB DNS of Grafana and paste it into your browser.

Use the username ‘admin’ and password ‘prom-operator’ to log in.

Configure Prometheus as a data source in Grafana:

• Click on “Data Source.”
• Select “Prometheus.”

• In the Connection, paste your <Prometheus-LB-DNS>:9090.

• If the URL is correct, you will see a green notification.
• Click on “Save & Test.”

Create a dashboard to visualize Kubernetes Cluster Logs:

• Click on “Dashboard.”

• Create a new dashboard.
• Import a Kubernetes Dashboard using ID 6417.
• Select the data source created earlier and click on “Import.”

You can view your Kubernetes Cluster Data.

Step 11: Deploying Three-Tier Application Using ArgoCD

Configure Private Repository in ArgoCD:

Click on “Settings” and select “Repositories.”

• Click on “CONNECT REPO USING HTTPS.”

Provide the repository name where your Manifests files are present.

Enter the username and GitHub Personal Access token, then click on “CONNECT.”

If the Connection Status is Successful, the repository is connected successfully.

Create Applications in ArgoCD:

Click on “CREATE APPLICATION.”

Database Application:

Select the repository configured earlier.

In the Path, specify the location of your Manifest files and provide other details.

Click on “CREATE.”

Backend Application:

Repeat the same process for the backend application.

Frontend Application:

Repeat the same process for the frontend application.

Ingress Application:

Repeat the same process for the ingress application.

Deployed Applications Verification:

• Once applications start deploying, check their status.
• For the Ingress application, it creates an Application Load Balancer (ALB).

Configure Domain Provider:

• Copy the ALB-DNS and go to your Domain Provider (e.g., porkbun).
• In DNS settings, add a CNAME type with the hostname (e.g., backend) pointing to your ALB.
• Save the changes.

Note: In this example, a subdomain ‘backend._youdomain_name is created.

Check Application Deployments:

Verify all four application deployments in the ArgoCD interface.

Experience the Magic:

After 2 to 3 minutes, access your subdomain in the browser to witness the deployed Three-Tier Application.

Conclusion:

In the culmination of this extensive DevSecOps Kubernetes project, we accomplished the following key milestones:

Established AWS Environment:

• Created an IAM user and utilized Terraform for AWS setup, ensuring secure and efficient cloud resource provisioning.

Jenkins Setup and Integration:

• Deployed Jenkins on AWS configured essential tools, and seamlessly integrated it with SonarQube for continuous code quality analysis.

EKS Cluster Configuration:

• Set up an EKS cluster, configured a Load Balancer, and established private Elastic Container Registry (ECR) repositories, ensuring secure and efficient container image storage.

Monitoring Implementation:

• Implemented monitoring solutions using Helm, Prometheus, and Grafana, providing visibility into the health and performance of the Kubernetes cluster.

GitOps with ArgoCD:

• Installed and configured ArgoCD for implementing GitOps practices, enabling declarative and version-controlled application deployments.

CI/CD with Jenkins Pipelines:

• Created Jenkins pipelines for continuous integration and continuous deployment (CI/CD), orchestrating the deployment of a Three-Tier application in a robust and automated manner.

Data Persistence Strategies:

• Ensured data persistence within the Kubernetes cluster through the effective utilization of persistent volumes and claims, guaranteeing the integrity and availability of critical application data.

By successfully navigating through these intricacies of DevSecOps and Kubernetes, we have established a resilient and automated development pipeline, adhering to best practices and ensuring a secure, scalable, and efficient cloud-native application environment. This project serves as a foundation for implementing DevSecOps principles in real-world scenarios, fostering collaboration, security, and continuous improvement in the software development lifecycle.

Stay connected on LinkedIn: LinkedIn Profile

Stay up-to-date with GitHub: GitHub Profile

Happy Learning!!