Implementing CI/CD Pipeline with Jenkins and AWS EC2 — Part 2: Configuring Our App’s CI/CD Pipeline
In Part 1 of this guide, we discussed setting up Jenkins on an AWS EC2 instance. With the foundational components in place, it’s time to start integrating our application and making the magic of CI/CD happen.
Prerequisites:
- Completion of Part 1: Installing Jenkins on AWS EC2.
- A basic Express.js application (If you don’t have one, refer to our sample project on GitHub).
Steps:
1. Accessing Jenkins Dashboard and Setting up a Pipeline for our Express.js App:
- Access Jenkins Dashboard: If you followed Part 1, you’d have opened port 8080 in your EC2 security group. Open your browser and navigate to
http://your-ec2-public-ip:8080
. This will lead you to the Jenkins dashboard. If prompted, login with the credentials you set up previously.
2. Create a New Jenkins Job:
- Start by clicking on
New Item
from the Jenkins dashboard. - Name the job (e.g.,
ExpressApp-CI
) and selectPipeline
. - Click
OK
to create the job.
General Section Configuration:
In the General section, add a simple description(optional) and do the following:
- Check “GitHub project”: Link Jenkins directly to your GitHub project.
- Project URL: Input your GitHub repository URL, e.g.,
https://github.com/your-username/your-repo-name/
.
This step ensures Jenkins knows where to fetch the source code when executing the pipeline.
Configure Build Triggers:
Jenkins offers multiple ways to determine when and how a build should be triggered. Here are some popular choices:
- Build After Other Projects Are Built: A beneficial choice if your project depends on other projects.
- Build Periodically: Set a schedule, like nightly or weekly builds.
- GitHub Hook Trigger for GITScm Polling: Automate builds when changes are pushed to the GitHub repository.
- Poll SCM: Periodically check for changes in the repository and build if any are detected.
- Quiet period: This is the time (in seconds) Jenkins waits before building after detecting a change. Useful to group multiple commits together and prevent unnecessary consecutive builds.
- Trigger Builds Remotely (e.g., from scripts): Allows for external scripts or tools to kick off the build process.
For our purpose, we’re selecting GitHub Hook Trigger for GITScm Polling. This option allows Jenkins to automatically build every time updates are pushed to our GitHub repository. It ensures our deployment process is immediate and in sync with the latest changes, epitomising the essence of continuous integration.
Pipeline:
- Definition & SCM: In our case, we’ll choose
Pipeline script from SCM
(Source Management Control) andGit
as SCM - Repository URL: Provide the URL of your Git repository.
- Credentials: If the repository is private, you will need to add the necessary credentials to allow Jenkins access.
- Branches to Build: By default, Jenkins is set to build from a branch named
master
. However, in our example, we have renamed our primary branch tomain
. Ensure you specify the correct branch name to ensure Jenkins is pulling from the desired branch. - Script Path: This specifies the name of the file for Jenkins configuration, so we’ll keep it
Jenkinsfile
Next, click on Save
to finalise the Pipeline
Manually Triggering the Pipeline:
Before we dive into creating our Jenkinsfile
into our project and fully automating our pipeline, it's advisable to manually trigger the pipeline. This will allow us to verify that Jenkins can indeed fetch the code from the GitHub repository and that our basic pipeline settings are correct.
Note: You should have the
Jenkinsfile
in the root directory of your project for this to work so let’s add an empty file calledJenkinsfile
Visual Confirmation of the Manual Build:
Below is a screenshot illustrating the successful manual build of our pipeline. As can be seen, Jenkins fetched our code from the GitHub repository, and our initial pipeline configurations appear to be in order.
Setting up the Jenkinsfile for Continuous Integration:
The heart of our pipeline lies in the Jenkinsfile
. It defines the stages and steps that Jenkins should execute whenever our defined criteria (in this case, a push to the main
branch) are met. Here's how to set it up:
- Creating the Jenkinsfile: In the root directory of your project (if you didn’t do before), create a new file named
Jenkinsfile
(no file extension). This will serve as the script for our pipeline. - Defining Stages and Steps: Within the
Jenkinsfile
, define the various stages and steps of your pipeline. For instance, stages for “Build”, “Test”, and “Deploy” could be defined. Under each stage, you specify the necessary steps, commands, or scripts that should be executed.
pipeline {
agent any
stages {
stage('Install Packages') {
steps {
script {
sh 'yarn install'
}
}
}
stage('Run the App') {
steps {
script {
sh 'yarn start &'
sleep 5
}
}
}
stage('Visit /health route') {
steps {
script {
sh 'curl http://localhost:3000/health'
}
}
}
stage('Cleanup') {
steps {
script {
sh 'pkill -f "node"'
}
}
}
}
}
Breaking Down the Jenkins Pipeline Script:
- Pipeline Agent:
agent any
This instructs Jenkins to allocate a workspace and run the entire pipeline on any available agent. In simpler terms, it asks Jenkins to pick any available executor (or environment) to run our pipeline.
2. Install Packages:
stage('Install Packages') {
steps {
script {
sh 'yarn install'
}
}
}
This stage installs all the dependencies for our Express.js application using Yarn. It ensures that before we start our application, all the necessary packages are in place.
3. Run the App:
stage('Run the App') {
steps {
script {
sh 'yarn start &'
sleep 5
}
}
}
Here, we’re starting our Express.js app. The &
allows the app to run in the background, freeing up the terminal. The sleep 5
ensures Jenkins waits for a few seconds, giving our app ample time to initialise and start listening on its port.
4. Test the App:
stage('Test the app') {
steps {
script {
sh 'curl http://localhost:3000/health'
}
}
}
In this final stage, we’re testing our app. The curl
command fetches the response from the /health
route of our application. If everything is set up correctly, and the app is running as expected, this step should complete without any errors.
By following this pipeline, we are not only ensuring that our app is correctly set up and running, but we’re also verifying that our most crucial route (/health
) is functional.
Setting Up Node.js and Yarn on Jenkins:
Before we execute our pipeline, it’s essential to make sure that Jenkins has access to both Node.js
and Yarn
. If you have faced an error like yarn: command not found.
This is due to the absence of the necessary environment and tools on the Jenkins server. To solve this, we have to do the following:
- Installing the NodeJS Plugin:
- Navigate to
Manage Jenkins
→Manage Plugins
. - In the
Available
tab, search forNodeJS
and install the NodeJS plugin. - After installation, go to
Manage Jenkins
→Global Tool Configuration
. - Under
NodeJS Installations
, click onAdd NodeJS
. Name it (e.g., "nodejs") and choose the Node.js version you want. This will ensure that Jenkins can access Node.js.
2. Configuring Yarn:
While Jenkins now knows about Node.js
, Yarn might still be missing. Here’s how you can set up Yarn on your Jenkins server:
- SSH into your EC2 instance.
- Run the following commands to install Yarn:
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update && sudo apt install yarn
3. Setting Up Node.js and Yarn in Jenkins in One Go:
Access Node.js Configuration: From Jenkins dashboard, go to “Manage Jenkins” > “Tools” and find “NodeJS installations”. Click “Add NodeJS”.
- Name & Version: Assign it the name “nodejs”.
- Auto-Install & Global Packages: Ensure “Install automatically” is checked. Under “Global npm packages to install”, add
yarn
so it's globally available. - Set Refresh Interval: Under “Global npm packages refresh hours”, set to
72
hours, allowing Jenkins to check foryarn
updates every three days. - Finalise: Click “Save” or “Apply”.
4. Configuring Jenkinsfile:
In your Jenkinsfile
, add a tools
block to specify the Node.js version (as named in the global tool configuration). This tells Jenkins which Node.js installation to use.
tools {
nodejs "nodejs"
}
Executing the CI/CD Pipeline:
Now that we’ve configured Jenkins with the necessary tools and set up our Jenkinsfile
, it’s time to execute the pipeline and see it in action.
Testing the CI/CD Pipeline:
- Initiating the Build:
- Go to Jenkins dashboard and select your job (`ExpressApp-CI`).
- Click “Build Now” to manually start the pipeline using the
Jenkinsfile
setup.
2. Monitoring the Build:
- Check the pipeline’s progress by clicking on the current build number on the left sidebar.
- The console output provides real-time feedback on each stage: “Install Packages”, “Run the App”, and “Test the App”.
3. Review Build Outcome:
- A blue ball icon next to your build number indicates success, while red suggests failure. For any issues, the console output is your primary troubleshooting tool.
In Conclusion:
Throughout Part 1 and 2 of this guide, we’ve not only established Jenkins on AWS EC2 but also integrated our Express.js
application into a budding CI/CD pipeline. With the configurations in place, we’re primed to make the true power of CI/CD felt in our development workflow.
Up Next:
Gear up for “Implementing CI/CD Pipeline with Jenkins and AWS EC2 — Part 3: Advanced Deployment and Optimizing Workflows. In this upcoming installment, we’ll dive deeper into refining our pipeline, introduce more sophisticated configurations, and pave the way for a seamless, automated deployment. Your journey towards mastering CI/CD is about to get even more exciting. Stay with us!