Monitoring Made Easy: Enhancing CI/CD with Splunk and Jenkins Integration

Take any CI/CD pipeline to the next level by integrating Splunk with its advanced monitoring capabilities

Part 1: Setting up a CI/CD Pipeline Process with Jenkins and Docker in AWS

Have you ever imagined a CI/CD pipeline that not only delivers your applications faster but also provides valuable real-time insights into every stage of the process? In this article, you’ll take any CI/CD pipeline to the next level by integrating Splunk with its advanced monitoring capabilities in order to unlock unprecedented visibility, control, and actionable insights to empower developers and DevOps teams to boost their pipelines.

Note: This project is a continuation of this article, where a CI/CD pipeline is set up to deploy a Docker image into Docker Hub. However, you can still follow the walkthrough to get to know how to integrate Jenkins with Splunk into AWS to enable monitoring of your builds.

Switching to a new EC2 instance and installing Splunk

• Change the t2.micro instance to a t2.medium or t2.large (preferably) since when starting Splunk Enterprise it uses a whole vCPU and almost 3 GB in memory, thus the previous instance that was used in the Jenkins pipeline is not able to support the two technologies running at the same time.

Splunk’s monitor usage using the htop command

• When creating your new EC2 t2.large instance add the ‘Github webhooks’ security group and your own to it, along with the previous specifications the t2.micro had.
• Stop the t2.micro and the t2.large instances and detach their respective volumes. In case each of the volumes are in different AZs, you would need to create a snapshot from the t2.micro EBS.

t2.micro EBS snapshot

• Create a new volume that has the same AZ as the t2.large instance and add the snapshot that you created previously.

Creating a volume from a snapshot

• Attach your new volume as dev/sda1 in order to be the root volume of your new EC2 instance as shown highlighted below.

Attaching the new volume to the t2.large instance

• Enable port 8000 and 8088 for Splunk in your inbound rules of your previously created security group.

EC2 security group

• Start up again your new EC2 instance and verify if the /var/lib/jenkins/ directory exists. If so, it’s already installed along with all the configurations we made on the previous Jenkins tutorial.
• Allow port 8000 and 8088 in your Linux instance to allow incoming traffic.

# Allow incoming traffic for Splunk Enterprise
sudo ufw allow 8000

# Allow incoming traffic for the Splunk HTTP Event Collector
sudo ufw allow 8088

• Go to the Splunk website and download Splunk for Linux as shown below.

Downloading Splunk Enterprise for Linux

Downloading via the CLI

• Execute the following commands in your EC2 instance.

# Update package repositories
sudo apt update

# Upgrade installed packages
sudo apt upgrade

# Download Splunk Enterprise
wget -O splunk-9.0.4.1-419ad9369127-linux-2.6-amd64.deb "https://download.splunk.com/products/splunk/releases/9.0.4.1/linux/splunk-9.0.4.1-419ad9369127-linux-2.6-amd64.deb"

# Install Splunk Enterprise
sudo dpkg -i splunk-9.0.4.1-419ad9369127-linux-2.6-amd64.deb

# Verify the Splunk package status
dpkg --status splunk

# Create the SPLUNK_HOME environment variable to the Splunk Enterprise installation directory
export SPLUNK_HOME=/opt/splunk/

# Navigate to the bin directory
cd $SPLUNK_HOME/bin

# Start Splunk and accept its license
sudo ./splunk start --accept-license

# Enable Splunk to start at boot
sudo ./splunk enable boot-start

After executing the above commands, it will ask you to create your admin username and its password.

Creating the admin user in Splunk

Logging in to Splunk with the created credentials

Installing plugins and apps into Splunk and Jenkins

Install the Splunk App for Jenkins

• Go to ‘+Find more apps’ in the Splunk Home page in the left side bar.
• Look for ‘jenkins’ and install its Splunk app as below.
• Enter your Splunk.com credentials (not the ones you created for the admin user).

Installing the Splunk App for Jenkins

• After installing it, you’ll see the Splunk App for Jenkins in the Home page.

Home page — Splunk App for Jenkins installed

• Go to the App bar and in Settings select ‘Data inputs’.
• Click on HTTP Event Collector and then to Global Settings to enable a token as shown in the second picture below.

Splunk Data Inputs

Token Global Configuration

• Create a new token beside where it says ‘Global Settings’ and configure as below.

Configuring a new token

Input Settings

Review of Data Inputs for HEC

• Go again to ‘Settings’ and then to the ‘HTTP Event Collector’ to see your token value. Save this one as it will be used later in Jenkins to communicate data.

Install the Splunk plugin into Jenkins

• Go to Manage Jenkins -> Plugin manager and install the Splunk plugin without restart.

Installing Splunk Plugin

• Go to Manage Jenkins -> Configure System -> Splunk for Jenkins Configuration.
• HTTP Input host: Enter the EC2 IP address
• HTTP Input Port: It will take the default for the Splunk HEC (8088).
• HTTP Input Token: Enter your Splunk Token created above.
• Test the connection and make sure that it says ‘Splunk connection verified’ to continue.
• Apply and save.

Splunk for Jenkins configuration

Testing the Splunk integration into Jenkins

• Build the ‘My flask app’ job again.

Building Jenkins pipeline

• Now go to Splunk App For Jenkins in the Splunk Home Page and you will see your job’s log there.

Splunk App for Jenkins

• If you switch to Admin user in the Splunk App for Jenkins, you’ll see mor details like a build status history, the jobs’ audit trail, which nodes are executing certain builds, the Jenkins’ performance, and even you can set up alerts in cases where your build might take longer than expected to notify you, or whenever there’s a failure.

Splunk App for Jenkins — Admin view

Audit Trail logs

Jenkins Health

Jenkins nodes

• When performing a search, you can save your results as reports, create charts and set up dashboards, as you would normally do without the Splunk App for Jenkins.

Jenkins Search

Project Summary and Insights

As we bring our CI/CD journey to a close, we’ve witnessed the transformation of a standard pipeline into a fully monitored powerhouse. By integrating Splunk to Jenkins, there are several things you can monitor to gain insights into your Jenkins pipelines and improve your CI/CD processes. For example, you can monitor the status of builds and create visualizations to track them and identify issues or trends within them; you can also view the resource utilization of your Jenkins servers, such as its memory, CPU, and disk usage; monitor the pipeline stage performance to identify any bottlenecks for faster builds and deployment times; errors and logs analysis in order to identify patterns, exceptions, errors in order to troubleshoot issues quickly; among others.

If you want to learn more about Splunk and how to use it, check my previous article on this tool.

Further Reading:

Microservices Monitoring: Cutting Engineering Costs and Saving Time

Kubernetes Monitoring with OpenTelemetry

OpenTelemetry Tracing: Everything you need to know

Observability in software development - doing more with our data