Ci-Cd With Ansible — DevSecOps Petshop project

You can find GitHub Repository from here :

GitHub - Djay-ui/petstore

Welcome, tech enthusiasts!

Today, we embark on a detailed exploration of deploying a Java-based Petshop application, a ubiquitous scenario among organizations. This guide will focus on leveraging Jenkins as our CI/CD tool while employing Docker containers and Kubernetes clusters for efficient application hosting. Additionally, we’ll integrate Ansible to streamline and automate the deployment process, aiming to provide a comprehensive understanding of this deployment landscape.

Deployment Strategies:

Our approach to deploying the Petshop Java application involves a multifaceted strategy, harnessing the strengths of various technologies. Initially, we’ll use Jenkins as our CI/CD tool, orchestrating the entire deployment pipeline. Subsequently, we’ll delve into two deployment methods: one utilizing Docker containers for their agility and consistency, and the other harnessing Kubernetes clusters to showcase scalability and orchestration capabilities. Additionally, Ansible will play a pivotal role in automating and managing the deployment process, ensuring efficiency and consistency across environments.

STEPS:

Step 1 -- Create an Ubuntu(22.04) T2 Large Instance

Step 2 -- Install Jenkins, Docker and Trivy

Step 3 -- Install Plugins like JDK, Sonarqube Scanner, Maven, OWASP Dependency Check

Step 4 -- Configure Sonar Server in Manage Jenkins

Step 5 -- Install OWASP Dependency Check Plugins

Step 6 -- Docker plugin and credential Setup

Step 7 -- Adding Ansible Repository in Ubuntu and install Ansible

Step 8 -- Kuberenetes Setup

Step 9 -- Master-slave Setup for Ansible and Kubernetes

Step 1 : Create an Ubuntu(22.04) T2 Large Instance :

Launch an AWS T2 Large Instance. Use the image as Ubuntu. You can create a new key pair or use an existing one. Enable HTTP and HTTPS settings in the Security Group and open all ports (not best case to open all ports but just for learning purposes it’s okay).

Step 2 : Install Jenkins, Docker and Trivy :

Connect to your console, and enter these commands to Install Jenkins

vi jenkins.sh

#!/bin/bash
sudo apt update -y
#sudo apt upgrade -y
wget -O - https://packages.adoptium.net/artifactory/api/gpg/key/public | tee /etc/apt/keyrings/adoptium.asc
echo "deb [signed-by=/etc/apt/keyrings/adoptium.asc] https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list
sudo apt update -y
sudo apt install temurin-17-jdk -y
/usr/bin/java --version
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
                  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
                  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
                              /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update -y
sudo apt-get install jenkins -y
sudo systemctl start jenkins
sudo systemctl status jenkins

sudo chmod 777 jenkins.sh
./jenkins.sh    # this will installl jenkins

Once Jenkins is installed, you will need to go to your AWS EC2 Security Group and open Inbound Port 8080 and 8090, 9000 for sonar, since Jenkins works on Port 8080.

• Change Jenkins port from 8080 To 8090 :

sudo systemctl stop jenkins
sudo systemctl status jenkins
cd /etc/default
sudo vi jenkins   #chnage port HTTP_PORT=8090 and save and exit
cd /lib/systemd/system
sudo vi jenkins.service  #change Environments="Jenkins_port=8090" save and exit
sudo systemctl daemon-reload
sudo systemctl restart jenkins
sudo systemctl status jenkins

Now, grab your Public IP Address

<EC2 Public IP Address:8090>
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Unlock Jenkins using an administrative password and install the suggested plugins.

Create a user click on save and continue, Jenkins Getting Started Screen :

2.1 : Install Docker :

sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker $USER   #my case is ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock

After the docker installation, we create a sonarqube container (Remember added 9000 ports in the security group).

docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Now our sonarqube is up and running :

username : admin
password : admin

Update New password, This is Sonar Dashboard :

2.3 : Install Trivy :

vi trivy.sh

sudo apt-get install wget apt-transport-https gnupg lsb-release -y

wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null

echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list

sudo apt-get update

sudo apt-get install trivy -y

Next, we will log in to Jenkins and start to configure our Pipeline in Jenkins.

Step 3 : Install Plugins like JDK, Sonarqube Scanner, Maven, OWASP Dependency Check :

3.1 : Install Plugin :

Goto Manage Jenkins →Plugins → Available Plugins →

Install below plugins

1 → Eclipse Temurin Installer (Install without restart)

2 → SonarQube Scanner (Install without restart)

3.2 : Configure Java and Maven in Global Tool Configuration :

Goto Manage Jenkins → Tools → Install JDK(17) and Maven3(3.6.0) → Click on Apply and Save

Name :jdk17 version : jdk 17.0.8.1+1

name : maven3 veriosn :3.6.0

3.3 : Create a Job :

Label it as PETSHOP, click on Pipeline and OK.

Enter this in Pipeline Script,

pipeline{
    agent any
    tools {
        jdk 'jdk17'
        maven 'maven3'
    }
    stages{
        stage ('clean Workspace'){
            steps{
                cleanWs()
            }
        }
        stage ('checkout scm') {
            steps {
                git 'https://github.com/Djay-ui/petstore.git'
            }
        }
        stage ('maven compile') {
            steps {
                sh 'mvn clean compile'
            }
        }
        stage ('maven Test') {
            steps {
                sh 'mvn test'
            }
        }
   }
}

The stage view would look like this,

Pipeline petstore

Step 4: Configure Sonar Server in Manage Jenkins :

Grab the Public IP Address of your EC2 Instance, Sonarqube works on Port 9000, so <Public IP>:9000. Goto your Sonarqube Server. Click on Administration → Security → Users → Click on Tokens and Update Token → Give it a name → and click on Generate Token

Create a token with a name and generate

copy Token

Goto Jenkins Dashboard → Manage Jenkins → Credentials → Add Secret Text. It should look like this

You will this page once you click on create

Now, go to Dashboard → Manage Jenkins → System and Add like the below image.

Click on Apply and Save

The Configure System option is used in Jenkins to configure different server

Global Tool Configuration is used to configure different tools that we install using Plugins

We will install a sonar scanner in the tools.

In the Sonarqube Dashboard add a quality gate also

Administration → Configuration →Webhooks

Click on Create

Add details , name : jenkins , URl : <public ip of jenkins:8090>

Let’s go to our Pipeline and add Sonarqube Stage in our Pipeline Script.

#under tools section add this environment

environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }



# in stages add this

stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('sonar-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Petshop \
                    -Dsonar.java.binaries=. \
                    -Dsonar.projectKey=Petshop '''
                }
            }
        }
        stage("quality gate"){
            steps {
                script {
                  waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token' 
                }
           }
        }

Click on Build now, you will see the stage view like this

To see the report, you can go to Sonarqube Server and go to Projects.

You can see the report has been generated and the status shows as passed. You can see that there are 6.7k lines. To see a detailed report, you can go to issues.

Step 5 : Install OWASP Dependency Check Plugins :

GotoDashboard → Manage Jenkins → Plugins → OWASP Dependency-Check. Click on it and install it without restart.

First, we configured the Plugin and next, we had to configure the Tool

Goto Dashboard → Manage Jenkins → Tools →

Name : DP-Check version : 6.5.1 (Click on Apply and Save here)

Now go configure → Pipeline and add this stage to your pipeline and build.

stage ('Build war file'){
            steps{
                sh 'mvn clean install -DskipTests=true'
            }
        }
        stage("OWASP Dependency Check"){
            steps{
                dependencyCheck additionalArguments: '--scan ./ --format XML ', odcInstallation: 'DP-Check'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }

The stage view would look like this,

You will see that in status, a graph will also be generated and Vulnerabilities.

Step 6 : Docker plugin and credential Setup :

We need to install the Docker tool in our system, Goto Dashboard → Manage Plugins → Available plugins → Search for Docker and install these plugins

Docker

Docker Commons

Docker Pipeline

Docker API

docker-build-step

and click on install without restart

Now, goto Dashboard → Manage Jenkins → Tools →

Name : docker version :lattest

Also add DockerHub Username and Password under Global Credentials

create a personal Access token from the docker hub which is used for ansible-playbook

copy it and save for later.

Let’s install Ansible on the Jenkins server

Step 7 : Adding Ansible Repository in Ubuntu :

Now we are going to run the below commands on the Ansible server

sudo apt-get update
sudo apt install software-properties-common

Add the ansible repository via PPA

sudo add-apt-repository --yes --update ppa:ansible/ansible

sudo apt install python3

Install Ansible on Ubuntu 22.04 LTS :

sudo apt install ansible -y

sudo apt install ansible-core -y
ansible --version

Create an Inventory file in Ansible :

To add inventory you can create a new directory or add in the default Ansible hosts file,

cd /etc/ansible
sudo nano hosts

Now go to the host file inside the Ansible server and paste the public IP of the Jenkins :

[local]#any name you want
Ip of Jenkins

save and exit from the file.

Let’s install The Ansible plugin to integrate with Jenkins.

Now add Credentials to invoke Ansible with Jenkins.

In the private key section, Select Enter directly and add your Pem file for the key.

and finally, click on Create.

Give this command in your Jenkins machine to find the path of your ansible which is used in the tool section of Jenkins.

which ansible

Copy that path and add it to the tools section of Jenkins at ansible installations.

Now write an Ansible playbook to create a docker image, tag it and push it to the docker hub, and finally, we will deploy it on a container using Ansible.

This is an docker.yaml file , in git repo. Use your dockerhub username in this

- name: docker build and push
  hosts: docker  # Replace with the hostname or IP address of your target server
  become: yes  # Run tasks with sudo privileges

  tasks:
    - name: Update apt package cache
      apt:
        update_cache: yes   

    - name: Build Docker Image
      command: docker build -t petstore .
      args:
        chdir: /var/lib/jenkins/workspace/petstore

    - name: tag image
      command: docker tag petstore:latest dhananjaytupe748/petstore:latest 

    - name: Log in to Docker Hub
      community.docker.docker_login:
        registry_url: https://index.docker.io/v1/
        username: sevenajay
        password: <docker pat>

    - name: Push image
      command: docker push dhananjaytupe748/petstore:latest

    - name: Run container
      command: docker run -d --name pet1 -p 8081:8080 dhananjaytupe748/petstore:latest

Add this stage to the pipeline to build and push it to the docker hub, and run the container.

   stage('Install Docker') {
            steps {
                dir('Ansible'){
                  script {
                         ansiblePlaybook credentialsId: 'ssh', disableHostKeyChecking: true, installation: 'ansible', inventory: '/etc/ansible/', playbook: 'docker-playbook.yaml'
                        }     
                   }    
              }
        }

Output of pipeline

Now copy the IP address of Jenkins and paste it into the browser

<jenkins-ip:8081>/jpetstore

<jenkins-ip:8081>/jpetstore

Step 8 : Kuberenetes Setup :

Connect your machines to Putty or Mobaxtreme

Take-Two Ubuntu 20.04 instances one for k8s master and the other one for worker.

Install Kubectl on Jenkins machine also.

#Kubectl on Jenkins to be installed
sudo apt update
sudo apt install curl
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client

On Two Ubuntu 20.04 instances master and worker.

— — — On Master_Node — — —

sudo su
hostname master
bash
clear

— — — On Worker_Node — — —

sudo su
hostname worker
bash
clear

— — — On Both Master and Worker Node — — —

sudo apt-get update 

sudo apt-get install -y docker.io
sudo usermod –aG docker Ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock

sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

sudo tee /etc/apt/sources.list.d/kubernetes.list <<EOF
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update

sudo apt-get install -y kubelet kubeadm kubectl

sudo snap install kube-apiserver

— — — On Master_Node — — —

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
# in case your in root exit from it and run below commands
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

— — — On Worker_Node — — —

sudo kubeadm join <master-node-ip>:<master-node-port> --token <token> --discovery-token-ca-cert-hash <hash>

— — — On Master_Node — — —

cd 
ls -a 
cd .kube/
cat config

copy it and save it in local machine on a desktop as secret-file.txt

Install Kubernetes Plugin, Once it’s installed successfully ,

go to manage Jenkins → manage credentials → Click on Jenkins global → add credentials

choose here that secret-file.txt

Step 9: Master-slave Setup for Ansible and Kubernetes :

To communicate with the Kubernetes clients we have to generate an SSH key on the ansible master node and exchange it with K8s Master System.

ssh-keygen

Change the directory to .ssh and copy the public key (id_rsa.pub)

cd .ssh
cat id_rsa.pub  #copy this public key

Once you copy the public key from the Ansible master, move to the Kubernetes machine change the directory to .ssh and paste the copied public key under authorized_keys.

cd .ssh #on k8s master 
sudo vi authorized_keys

Note: Now, insert or paste the copied public key into the new line. make sure don’t delete any existing keys from the authorized_keys file then save and exit.

By adding a public key from the master to the k8s machine we have now configured keyless access. To verify you can try to access the k8s master and use the command as mentioned in the below format.

ssh ubuntu@<public-ip-k8s-master>

Verifying the above SSH connection from the master to the Kubernetes we have configured our prerequisites.

Now go to the host file inside the Ansible server and paste the public IP of the k8s master.

[k8s]#any name you want
public ip of k8s-master

Test Ansible Master Slave Connection :

Use the below command to check Ansible master-slave connections.

ansible -m ping k8s
ansible -m ping all#use this one

If all configuration is correct then you would get below output.

let’s create a simple ansible playbook for Kubernetes deployment.

---
- name: Deploy Kubernetes Application
  hosts: k8s  # Replace with your target Kubernetes master host or group
  gather_facts: yes  # Gather facts about the target host

  tasks:
    - name: Copy deployment.yaml to Kubernetes master
      copy:
        src: /var/lib/jenkins/workspace/petstore/deployment.yaml  # Assuming Jenkins workspace variable
        dest: /home/ubuntu/
      become: yes  # Use sudo for copying if required
      become_user: root  # Use a privileged user for copying if required

    - name: Apply Deployment
      command: kubectl apply -f /home/ubuntu/deployment.yaml

Now add the below stage to your pipeline.

stage('k8s using ansible'){
            steps{
                dir('Ansible') {
                    script{
                        ansiblePlaybook credentialsId: 'ssh', disableHostKeyChecking: true, installation: 'ansible', inventory: '/etc/ansible/', playbook: 'kube.yaml'
                    }
                } 
            }
        }

output

In the Kubernetes cluster(master) give this command

kubectl get all
kubectl get svc

<slave-ip:serviceport(30699)>/jpetstore

<slave-ip:serviceport(30699)>/jpetstore

Terminate instances :

Terminate all Running Intances to Avoid billing

Complete Pipeline :

pipeline{
    agent any
    tools {
        jdk 'jdk17'
        maven 'maven3'
    }
    environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }
    stages{
        stage ('clean Workspace'){
            steps{
                cleanWs()
            }
        }
        stage ('checkout scm') {
            steps {
                git 'https://github.com/Aj7Ay/jpetstore-6.git'
            }
        }
        stage ('maven compile') {
            steps {
                sh 'mvn clean compile'
            }
        }
        stage ('maven Test') {
            steps {
                sh 'mvn test'
            }
        }
        stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('sonar-server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Petstore \
                    -Dsonar.java.binaries=. \
                    -Dsonar.projectKey=Petstore '''
                }
            }
        }
        stage("quality gate"){
            steps {
                script {
                  waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token' 
                }
           }
        }
        stage ('Build war file'){
            steps{
                sh 'mvn clean install -DskipTests=true'
            }
        }
        stage("OWASP Dependency Check"){
            steps{
                dependencyCheck additionalArguments: '--scan ./ --format XML ', odcInstallation: 'DP-Check'
                dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
            }
        }
        stage('Ansible docker Docker') {
            steps {
                dir('Ansible'){
                  script {
                        ansiblePlaybook credentialsId: 'ssh', disableHostKeyChecking: true, installation: 'ansible', inventory: '/etc/ansible/', playbook: 'docker.yaml'
                    }     
                }    
            }
        }
        stage('k8s using ansible'){
            steps{
                dir('Ansible') {
                    script{
                        ansiblePlaybook credentialsId: 'ssh', disableHostKeyChecking: true, installation: 'ansible', inventory: '/etc/ansible/', playbook: 'kube.yaml'
                    }
                } 
            }
        }
   }
}

Thanks for Reading .