DevOps Blog 3: Building a Robust CI/CD Pipeline -From AWS EC2 to Spring Boot with Maven, Docker, and Jenkins

This blog is focused on setting up CI & CD pipeline for the Maven based Springboot API project.

Summary

Project: Building a Robust CI/CD Pipeline — From AWS EC2 to Spring Boot with Maven, Docker, and Jenkins. Docker In Docker (dind).

Outcome: You will be exploring how to startup AWS EC2 instance, Creating custom docker image using Jenkins base image and packaging maven & Docker client along with it, Creating various stages in Jenkinsfile (stages include checkout, compile, test, build, application docker image creation and starting a docker container with a jenkins docker).

High-Level Architecture

Prerequisites

1. AWS Account: Active AWS account for EC2 instance creation.
2. GitHub Account: Required for repository management and access tokens.
3. Basic Understanding: Familiarity with Jenkins, Docker, Maven, and Spring Boot.
4. Local Environment: Maven and Git installed on your local machine.
5. Editor: A code editor (like Visual Studio Code) for project management.

Steps

1. Access AWS Console:

- Navigate to https://aws.amazon.com and log in to your AWS account.

- From the AWS Management Console home, click on “Services” and select “EC2”.

2. Set Up EC2 Instance:

- Click the "Launch Instance" button.
- Select the Free Tier Amazon Linux 2 AMI.
- Ensure it's t2.micro for the free tier eligibility.
- On the "Configure Security Group" step, create a new security group with the following rules:
- Type: SSH, Port: 22, Source: Anywhere.
- Type: Custom TCP, Port: 8080 (application), Port:5050 (Jenkins), Source: Anywhere.
- Review and launch the instance.

3. Install Required Packages:

  SSH into your EC2 instance.
  sudo yum update -y  #Update the package list
  sudo yum install docker git -y #Install Docker and Git
  service docker start  #Start the docker service
  service docker status #Check Status of the docker service

4. Clone the code repository

• Clone the repository that I have created for sample SpringBoot application to print “Hello World” message using /hello endpoint (GET request)
• Create your own repository in your GitHub account.

https://github.com/Mukezz/CICDDemoBlog3.git

Using above repo create your own repo under your Github account
Command to clone the repo, 
git clone https://github.com/Mukezz/CICDDemoBlog3.git

5. Set Up Docker for Jenkins:

- Switch to the root user: `sudo su`

- Create a Dockerfile with the following content:

#Retrieve the docker container id
PuTTy ssh into the ec2 instance with username ec2-user using your-ec2-ip-address 

sudo su                 #sudo as root user
vi Dockerfile           # Creating Dockerfile

Dockerfile

Below Dockerfile creates a custom image “Docker-in-Docker” which contains Jenkins as base image and installs the docker & maven on top of it to create a custom image “jenkins-maven-docker”. As the goal of this project is to deploy the API code directly as a container, we need a docker client accessible within Jenkins container in order to build the application docker and start the application container within the Jenkins container.

# Use the official Jenkins image as a base image
FROM jenkins/jenkins:lts

# Install necessary packages
USER root
RUN apt-get update && \
    apt-get install -y apt-transport-https \
                       ca-certificates \
                       curl \
                       gnupg-agent \
                       software-properties-common && \
    rm -rf /var/lib/apt/lists/*

# Install Docker CLI
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - && \
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" && \
    apt-get update && \
    apt-get install -y docker-ce-cli && \
    rm -rf /var/lib/apt/lists/*

# Install Maven
RUN apt-get install -y maven

# Set the environment variable for Maven
ENV MAVEN_HOME /usr/share/maven
ENV PATH $MAVEN_HOME/bin:$PATH

Base Image: Uses the official Jenkins LTS (Long-Term Support) version as the starting point.

Package Installation: Switches to the root user and updates package repositories. It then installs a set of packages needed for https connections, managing repositories, and curl operations. After installation, it cleans up by removing the package lists.

Docker CLI Installation: It adds Docker’s official GPG key and sets up Docker’s stable repository for Debian. Following that, it installs the Docker command-line interface (CLI) and cleans up package lists again.

Maven Installation: Installs Maven, a popular tool for building and managing Java projects.

Maven Environment Variables: Sets environment variables to ensure that Maven is correctly configured and can be easily accessed from the command line within the container.

6. Build the Docker image & Start the Container

docker build -t jenkins-maven-docker .

• Run Jenkins using the custom Docker container — docker run command is to start the custom image by exposing the port 5050 (soruce port of the container is 8080). Attaching the volume /var/run/docker.sock from the host to access the docker service which is running on the host instance.

docker run -d -p 5050:8080 -p 50000:50000 -v /var/run/docker.sock:/var/run/docker.sock jenkins-maven-docker

In summary, above command runs a new detached container from the jenkins-maven-docker image, binding ports 5050 and 50000 on the host to ports 8080 and 50000 in the container, respectively. Additionally, it allows the container to interact with the Docker daemon of the host machine by mounting the Docker socket.

7. Complete Jenkins Setup:

Now that we have the Jenkins up and running

• Retrieve the initial admin password from the EC2 instance: `cat /var/jenkins_home/secrets/initialAdminPassword`

#How to retrieve the initial admin password password

1. Login into EC2 instance using PuTTy session
2. sudo docker container ls #get the container id from the list of running container
3. sudo docker exec -it <CONTAINER_ID> /bin/bash #Login into the container "jenkins-maven-docker"
4. cat /var/jenkins_home/secrets/initialAdminPassword
5. exit

• Access Jenkins via `http://<your-ec2-ip>:5050/`.
• Follow the setup wizard: set up an admin user and install the recommended plugins.

8. Incorporate Jenkins Pipeline

Create a separate GitHub repository for Jenkinsfile. Create a `Jenkinsfile` at the root of your project. This file will contain the stages and steps for your Jenkins pipeline:

https://github.com/Mukezz/CICDDemoBlog3/blob/main/Jenkinsfile

pipeline {
    agent any

    tools {
        maven 'Maven_3_8_7'  // Replace 'Maven_3_8_7' with the name of your Maven installation in Jenkins
    }

    stages {
        stage('Checkout') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '*/main']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'github-credentials', url: 'https://github.com/Mukezz/CICDDemoBlog3.git']]]) //Replace git url with yours
            }
        }

        stage('Compile') {
            steps {
                script {
                    sh "mvn clean compile"
                }
            }
        }

        stage('Test') {
            steps {
                script {
                    sh "mvn test"
                }
            }
        }

        stage('Building Jar') {
            steps {
                script {
                    sh 'mvn clean install'
                }
            }
        }

        stage('Running Dockerfile to create image') {
            steps {
                script {
                    sh "docker build -t devops/mukeshblogpost3 ."
                }
            }
        }

        stage('Starting the application docker container') {
            steps {
                script {
                    sh "docker run -d -p 8080:8080 devops/mukeshblogpost3"
                }
            }
        }              
    }
}

This Jenkinsfile instructs Jenkins to:

Stages: The main tasks the pipeline will execute, broken down into several stages:

Checkout: Fetches the source code from a GitHub repository named ‘CICDDemoBlog3’ under the YOUR account, using the ‘main’ branch.

Compile: Runs the Maven compile command to compile the Java source code.

Test: Executes the Maven test command to run unit tests.

Building Jar: Uses Maven to clean any previous builds and then installs (builds) the project, creating a JAR file.

Running Dockerfile to create image: Constructs a Docker image named ‘devops/mukeshblogpost3’ from the current directory’s Dockerfile.

Starting the application docker container: Launches a Docker container from the ‘devops/mukeshblogpost3’ image, exposing port 8080.

After creating the `Jenkinsfile`, commit and push it to your GitHub repository.

9. Dockerfile for creating application image

• Using below Dockerfile create the image, maven build has compiled the jar in the target directory. The same Dockerfile is available in the code repo — https://github.com/Mukezz/CICDDemoBlog3/blob/main/Dockerfile

FROM eclipse-temurin:11
VOLUME /tmp
COPY ./target/cidemo*jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]

FROM eclipse-temurin:11
This line specifies the base image for the container. The image being used is eclipse-temurin:11, which is an OpenJDK distribution (previously known as AdoptOpenJDK). Specifically, this uses version 11 of the JDK. This means that the container will have Java 11 installed and set up by default.

VOLUME /tmp
Volumes are used to persist data and can also be used for shared data among containers. Here, a volume is created on the path /tmp. This can be useful for temporary storage, cache data, or logs which the application inside the container might generate. By using a volume, the data inside /tmp won't be lost when the container stops or is deleted.

COPY ./target/cidemo*jar app.jar
This line copies a JAR file from the host's ./target/ directory into the container. The source path specifies cidemo*jar which will match any JAR file in the ./target/ directory that starts with the name "cidemo". This is useful if, for example, the exact name of the JAR file might change due to versioning, but you know it will always start with "cidemo". The JAR file is renamed to app.jar inside the container.

ENTRYPOINT ["java","-jar","/app.jar"]
When the container starts, it will execute the command specified in the ENTRYPOINT. In this case, it will run the Java application packed inside the app.jar file. The java -jar /app.jar command tells the Java runtime to start the application within the specified JAR file.

10. Setting up Maven Installation in Jenkins

11. Configure a Jenkins Pipeline Job:

- In Jenkins, select “New Item” and then choose “Pipeline”.

- Name your pipeline and then scroll to the “Pipeline” section.

- Under “Definition”, select “Pipeline script from SCM”. For SCM, choose “Git”, and then provide the URL to your GitHub repository.

• Ensure the path to your `Jenkinsfile` is correctly set.

12. Trigger Jenkins Job

Go to your newly created Jenkins job and click “Build Now” to trigger the pipeline. Monitor the execution and, upon successful completion, you’ll see the stages of the pipeline execution.

13. Cleaning up EC2 Instance

Please make sure to clean up the AWS EC2 instance to avoid any charges on your account. Login to AWS console, goto EC2 and select your instance to terminate.

Please leave your comments for any questions or clarification

Follow me on LinkedIn at https://www.linkedin.com/in/mukesh-vivekanandan/