Implementing CI/CD Pipeline with Jenkins and AWS EC2 — Part 2: Configuring Our App’s CI/CD Pipeline

8 min readOct 14, 2023

In Part 1 of this guide, we discussed setting up Jenkins on an AWS EC2 instance. With the foundational components in place, it’s time to start integrating our application and making the magic of CI/CD happen.

Prerequisites:

Completion of Part 1: Installing Jenkins on AWS EC2.
A basic Express.js application (If you don’t have one, refer to our sample project on GitHub).

Steps:

1. Accessing Jenkins Dashboard and Setting up a Pipeline for our Express.js App:

Access Jenkins Dashboard: If you followed Part 1, you’d have opened port 8080 in your EC2 security group. Open your browser and navigate to http://your-ec2-public-ip:8080. This will lead you to the Jenkins dashboard. If prompted, login with the credentials you set up previously.

2. Create a New Jenkins Job:

Start by clicking on New Item from the Jenkins dashboard.
Name the job (e.g., ExpressApp-CI) and select Pipeline.
Click OK to create the job.

General Section Configuration:

In the General section, add a simple description(optional) and do the following:

Check “GitHub project”: Link Jenkins directly to your GitHub project.
Project URL: Input your GitHub repository URL, e.g., https://github.com/your-username/your-repo-name/.

This step ensures Jenkins knows where to fetch the source code when executing the pipeline.

Configure Build Triggers:

Jenkins offers multiple ways to determine when and how a build should be triggered. Here are some popular choices:

Build After Other Projects Are Built: A beneficial choice if your project depends on other projects.
Build Periodically: Set a schedule, like nightly or weekly builds.
GitHub Hook Trigger for GITScm Polling: Automate builds when changes are pushed to the GitHub repository.
Poll SCM: Periodically check for changes in the repository and build if any are detected.
Quiet period: This is the time (in seconds) Jenkins waits before building after detecting a change. Useful to group multiple commits together and prevent unnecessary consecutive builds.
Trigger Builds Remotely (e.g., from scripts): Allows for external scripts or tools to kick off the build process.

For our purpose, we’re selecting GitHub Hook Trigger for GITScm Polling. This option allows Jenkins to automatically build every time updates are pushed to our GitHub repository. It ensures our deployment process is immediate and in sync with the latest changes, epitomising the essence of continuous integration.

Pipeline:

Definition & SCM: In our case, we’ll choose Pipeline script from SCM(Source Management Control) and Git as SCM
Repository URL: Provide the URL of your Git repository.
Credentials: If the repository is private, you will need to add the necessary credentials to allow Jenkins access.
Branches to Build: By default, Jenkins is set to build from a branch named master. However, in our example, we have renamed our primary branch to main. Ensure you specify the correct branch name to ensure Jenkins is pulling from the desired branch.
Script Path: This specifies the name of the file for Jenkins configuration, so we’ll keep it Jenkinsfile

Next, click on Save to finalise the Pipeline

Manually Triggering the Pipeline:

Before we dive into creating our Jenkinsfile into our project and fully automating our pipeline, it's advisable to manually trigger the pipeline. This will allow us to verify that Jenkins can indeed fetch the code from the GitHub repository and that our basic pipeline settings are correct.

Note: You should have the Jenkinsfile in the root directory of your project for this to work so let’s add an empty file called Jenkinsfile

Visual Confirmation of the Manual Build:

Below is a screenshot illustrating the successful manual build of our pipeline. As can be seen, Jenkins fetched our code from the GitHub repository, and our initial pipeline configurations appear to be in order.

Setting up the Jenkinsfile for Continuous Integration:

The heart of our pipeline lies in the Jenkinsfile. It defines the stages and steps that Jenkins should execute whenever our defined criteria (in this case, a push to the main branch) are met. Here's how to set it up:

Creating the Jenkinsfile: In the root directory of your project (if you didn’t do before), create a new file named Jenkinsfile (no file extension). This will serve as the script for our pipeline.
Defining Stages and Steps: Within the Jenkinsfile, define the various stages and steps of your pipeline. For instance, stages for “Build”, “Test”, and “Deploy” could be defined. Under each stage, you specify the necessary steps, commands, or scripts that should be executed.

pipeline {
    agent any

    stages {
        stage('Install Packages') {
            steps {
                script {
                    sh 'yarn install'
                }
            }
        }

        stage('Run the App') {
            steps {
                script {
                    sh 'yarn start &'
                    sleep 5
                }
            }
        }

        stage('Visit /health route') {
            steps {
                script {
                    sh 'curl http://localhost:3000/health'
                }
            }
        }

        stage('Cleanup') {
            steps {
                script {
                    sh 'pkill -f "node"'
                }
            }
        }
    }
}

Breaking Down the Jenkins Pipeline Script:

Pipeline Agent:

agent any

This instructs Jenkins to allocate a workspace and run the entire pipeline on any available agent. In simpler terms, it asks Jenkins to pick any available executor (or environment) to run our pipeline.

2. Install Packages:

stage('Install Packages') {
    steps {
        script {
            sh 'yarn install'
        }
    }
}

This stage installs all the dependencies for our Express.js application using Yarn. It ensures that before we start our application, all the necessary packages are in place.

3. Run the App:

stage('Run the App') {
    steps {
        script {
            sh 'yarn start &'
            sleep 5
        }
    }
}

Here, we’re starting our Express.js app. The & allows the app to run in the background, freeing up the terminal. The sleep 5 ensures Jenkins waits for a few seconds, giving our app ample time to initialise and start listening on its port.

4. Test the App:

stage('Test the app') {
    steps {
        script {
            sh 'curl http://localhost:3000/health'
        }
    }
}

In this final stage, we’re testing our app. The curl command fetches the response from the /health route of our application. If everything is set up correctly, and the app is running as expected, this step should complete without any errors.

By following this pipeline, we are not only ensuring that our app is correctly set up and running, but we’re also verifying that our most crucial route (/health) is functional.

Setting Up Node.js and Yarn on Jenkins:

Before we execute our pipeline, it’s essential to make sure that Jenkins has access to both Node.js and Yarn. If you have faced an error like yarn: command not found.This is due to the absence of the necessary environment and tools on the Jenkins server. To solve this, we have to do the following:

Installing the NodeJS Plugin:

Navigate to Manage Jenkins → Manage Plugins.
In the Available tab, search for NodeJS and install the NodeJS plugin.
After installation, go to Manage Jenkins → Global Tool Configuration.
Under NodeJS Installations, click on Add NodeJS. Name it (e.g., "nodejs") and choose the Node.js version you want. This will ensure that Jenkins can access Node.js.

2. Configuring Yarn:
While Jenkins now knows about Node.js, Yarn might still be missing. Here’s how you can set up Yarn on your Jenkins server:

SSH into your EC2 instance.
Run the following commands to install Yarn:

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update && sudo apt install yarn

3. Setting Up Node.js and Yarn in Jenkins in One Go:
Access Node.js Configuration: From Jenkins dashboard, go to “Manage Jenkins” > “Tools” and find “NodeJS installations”. Click “Add NodeJS”.

Name & Version: Assign it the name “nodejs”.
Auto-Install & Global Packages: Ensure “Install automatically” is checked. Under “Global npm packages to install”, add yarn so it's globally available.
Set Refresh Interval: Under “Global npm packages refresh hours”, set to 72 hours, allowing Jenkins to check for yarn updates every three days.
Finalise: Click “Save” or “Apply”.

4. Configuring Jenkinsfile:
In your Jenkinsfile, add a tools block to specify the Node.js version (as named in the global tool configuration). This tells Jenkins which Node.js installation to use.

tools {
    nodejs "nodejs"
}

Executing the CI/CD Pipeline:

Now that we’ve configured Jenkins with the necessary tools and set up our Jenkinsfile, it’s time to execute the pipeline and see it in action.

Testing the CI/CD Pipeline:

Initiating the Build:

Go to Jenkins dashboard and select your job (`ExpressApp-CI`).
Click “Build Now” to manually start the pipeline using the Jenkinsfile setup.

2. Monitoring the Build:

Check the pipeline’s progress by clicking on the current build number on the left sidebar.
The console output provides real-time feedback on each stage: “Install Packages”, “Run the App”, and “Test the App”.

3. Review Build Outcome:

A blue ball icon next to your build number indicates success, while red suggests failure. For any issues, the console output is your primary troubleshooting tool.

In Conclusion:
Throughout Part 1 and 2 of this guide, we’ve not only established Jenkins on AWS EC2 but also integrated our Express.js application into a budding CI/CD pipeline. With the configurations in place, we’re primed to make the true power of CI/CD felt in our development workflow.

Up Next:
Gear up for “Implementing CI/CD Pipeline with Jenkins and AWS EC2 — Part 3: Advanced Deployment and Optimizing Workflows. In this upcoming installment, we’ll dive deeper into refining our pipeline, introduce more sophisticated configurations, and pave the way for a seamless, automated deployment. Your journey towards mastering CI/CD is about to get even more exciting. Stay with us!