Open in app

Sign in

Write

Sign in

Building an Automated CI/CD Pipeline using GitHub, Jenkins, NGINX, and Python

Kcsanjeeb
9 min readFeb 12, 2024

--

Welcome to our blog where we delve into the collaborative efforts between developers and server administrators or DevOps engineers to optimize the Continuous Integration and Continuous Delivery (CI/CD) process. In this introductory post, we’ll explore the roles and responsibilities of both parties in our project, highlighting the seamless integration of Flask, GitHub, NGINX, and Jenkins for efficient development and deployment pipelines. Let’s dive in!

  1. Developer:
  • Responsibilities:
  • Write and maintain the Flask application code.
  • Push code changes to the GitHub repository.
  • Collaborate with team members for code reviews and updates.
  • Involvement in CI/CD process:
  • Providing a sample flaskproject.ini file with placeholder/default values.
  1. Server Administrator / DevOps Engineer:
  • Responsibilities:
  • Manage server infrastructure and configurations.
  • Configure and maintain Nginx for serving Flask applications.
  • Configure uWSGI for deploying Flask applications.
  • Create Systemd unit files for managing uWSGI services.
  • Involvement in CI/CD process:
  • Configuring the flaskproject.ini file for uWSGI based on production environment requirements.
  • Setting up Jenkins jobs for automated deployment.

Machine 1: Developer Machine 🧑🏻‍💻

To configure the network on the developer machine, we utilized the nmcli command-line tool. First, we added a new Ethernet connection named "default". Then, we modified this connection to set a manual IPv4 address (192.168.208.101/24), gateway (192.168.208.2), and DNS server (192.168.208.2). After configuring the connection, we brought it up using nmcli conn up "default". Finally, we restarted the NetworkManager service to apply the changes. The hostname -I command confirmed the successful configuration, displaying the assigned IP address (192.168.208.101).

[root@developer ~]# nmcli conn add con-name "default" ifname "*" type ethernet
Connection 'default' (a1aa6a63-38ca-4358-98fd-3ab9888a26a3) successfully added.
[root@developer ~]# nmcli conn modify "default" ipv4.address "192.168.208.101/24" ipv4.method manual ipv4.gateway "192.168.208.2" ipv4.dns "192.168.208.2"
[root@developer ~]# nmcli conn up "default" 
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)
[root@developer ~]# systemctl restart NetworkManager
[root@developer ~]# hostname -I
192.168.208.101

To grant full sudo privileges to the user “Sanjeeb,” we edited the sudoers file using the visudo command. We added the line sanjeeb ALL=(ALL) ALL to the file, which allows the user "Sanjeeb" to execute any command with elevated privileges (sudo) on any host and from any terminal session. This step ensures that "Sanjeeb" has sufficient permissions to perform administrative tasks on the system. However, it's crucial to exercise caution and restrict sudo access only to trusted users to maintain system security.

[root@developer ~]# visudo

#ADD THE FOLLOWING IN THE FILE
sanjeeb ALL=(ALL) ALL

Installing Git & Setting up SSH

In this section, we’ll walk through the process of installing Git, setting up SSH, and cloning a Flask application from a GitHub repository.

[sanjeeb@developer ~]$ sudo yum -y install git
[sanjeeb@developer ~]$ rpm -q git
git-2.43.0-1.el9.aarch64

[sanjeeb@developer ~]$ ssh-keygen
[sanjeeb@developer ~]$ cd .ssh
[sanjeeb@developer ~]$ ls
id_rsa    id_rsa.pub

#Copy the id_rsa.pub and paste it to the github SSH Keys
[sanjeeb@developer ~]$ cat id_rsa.pub

[sanjeeb@developer ~]$ git clone git@github.com:kcsanjeeb/pythonRepo.git
Cloning into 'pythonRepo'...
The authenticity of host 'github.com (20.205.243.166)' can't be established.
ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
[sanjeeb@developer ~]$ ls
Desktop  Documents  Downloads  Music  Pictures  Public  pythonRepo  Templates  Videos

Configuring Global Parameter for github

[sanjeeb@developer pythonRepo]$ # Configuring GLobal Parameter

[sanjeeb@developer pythonRepo]$ git config --global user.name "sanjeeb"
[sanjeeb@developer pythonRepo]$ git config --global user.email "kcsanjeeb091@gmail.com"
[sanjeeb@developer pythonRepo]$ git config --list

Pushing the changes made to the repo

[sanjeeb@developer flaskproject]$ git add .
[sanjeeb@developer flaskproject]$ git commit -m "from developer"
[master (root-commit) 4209192] from developer
 5 files changed, 26 insertions(+)
 create mode 100644 __pycache__/flaskproject.cpython-39.pyc
 create mode 100644 __pycache__/wsgi.cpython-39.pyc
 create mode 100644 flaskproject.ini
 create mode 100644 flaskproject.py
 create mode 100644 wsgi.py

[sanjeeb@developer flaskproject]$ git push --set-upstream origin master
Enumerating objects: 8, done.
Counting objects: 100% (8/8), done.
Delta compression using up to 2 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 1.22 KiB | 1.22 MiB/s, done.
Total 8 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:kcsanjeeb/flaskproject.git
 * [new branch]      master -> master
branch 'master' set up to track 'origin/master'.

Machine 2: NGINX Machine (Web Server) 🌐

Configuring the Network in Developer Machine

[root@nginx ~]# nmcli conn add con-name "default" ifname "*" type ethernet
Connection 'default' (a1aa6a63-38ca-4358-98fd-3ab9888a26a3) successfully added.
[root@nginx ~]# nmcli conn modify "default" ipv4.address "192.168.208.201/24" ipv4.method manual ipv4.gateway "192.168.208.2" ipv4.dns "192.168.208.2"
[root@nginx ~]# nmcli conn up "default" 
Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/3)
[root@nginx ~]# systemctl restart NetworkManager
[root@nginx ~]# hostname -I
192.168.208.101

Providing Full Privilege to “Sanjeeb” User

[root@nginx ~]# visudo
#ADD THE FOLLOWING IN THE FILE
sanjeeb ALL=(ALL) ALL

Switch to the user “sanjeeb”.

Setting up Nginx Package

Install the Components from the CentOS and EPEL Repositories.

[sanjeeb@nginx ~]$ sudo yum -y install nginx
[sanjeeb@nginx ~]$ rpm -q nginx
nginx-1.22.1-2.el9.aarch64

[sanjeeb@nginx ~]$ sudo systemctl start nginx
[sanjeeb@nginx ~]$ sudo systemctl enable nginx
[sanjeeb@nginx ~]$ sudo systemctl status nginx

You can enable the EPEL repo by typing:

Once access to the EPEL repository is configured on our system, we can begin installing the packages we need. We will install pip, the Python package manager, in order to install and manage our Python components. We will also get a compiler and the Python development files needed to build uWSGI.

[sanjeeb@nginx ~]$ sudo yum install epel-release
[sanjeeb@nginx ~]$ sudo yum install python-pip python-devel gcc
[sanjeeb@nginx ~]$ sudo yum install uwsgi

Check the firewall configuration, and add http service in the firewall and also port.

[sanjeeb@nginx ~]$ sudo firewall-cmd --list-all 
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens160
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
[sanjeeb@nginx ~]$ firewall-cmd --add-service=http
success
[sanjeeb@nginx ~]$ firewall-cmd --permanent --add-port=8000/tcp
success
[sanjeeb@nginx ~]$ sudo firewall-cmd --list-all 
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens160
  sources: 
  services: cockpit dhcpv6-client http ssh
  ports: 8000/tcp 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

We will create a test flask application to check if nginx is working fine:

Creating Test Flask Application

[sanjeeb@nginx ~]$ mkdir flaskproject
[sanjeeb@nginx ~]$ cd flaskproject

# CREATING VIRTUAL ENVIRONMENT
[sanjeeb@nginx flaskproject]$ virtualenv flaskprojectenv
[sanjeeb@nginx flaskproject]$ source flaskprojectenv/bin/activate

# INSTALL FLASK AND UWSGI
(flaskprojectenv) [sanjeeb@nginx flaskproject]$ pip install uwsgi flask
# CREATE NECESSARY FILES
(flaskprojectenv) [sanjeeb@nginx flaskproject]$ vim flaskproject.py

from flask import Flask
application = Flask(__name__)

@application.route("/")
def hello():
    return "<h1 style='color:blue'>Hello There!</h1>"

if __name__ == "__main__":
    application.run(host='0.0.0.0')
# CREATE INI FILE 
(flaskprojectenv) [sanjeeb@nginx flaskproject]$ vim flaskproject.ini

[uwsgi]
module = wsgi

master = true
processes = 5

socket = flaskproject.sock
chmod-socket = 660
vacuum = true

die-on-term = true
# CREATE PYTHON FILE
(flaskprojectenv) [sanjeeb@nginx flaskproject]$ vim wsgi.py

from flaskproject import application

if __name__ == "__main__":
    application.run()
# RUN THE APPLICATION
 (flaskprojectenv) [sanjeeb@nginx flaskproject]$ python wsgi.py

If application is up and running, then, we are ready to accept incoming code from github through jenkins.

Create a systemd unit file

We established streamlined management for the Flask application by crafting a systemd unit file named flaskproject.service within the /etc/systemd/system directory. The unit file was configured to trigger after the network target is reached, ensuring dependency readiness. Within the service definition, we assigned the user sanjeeb and the group nginx for execution. The working directory was set to /home/sanjeeb/flaskproject, and we specified the virtual environment's path using the Environment directive. Furthermore, the ExecStart command was configured to activate the virtual environment and initiate uWSGI to serve the Flask application. Following the setup, we enabled the service to start on boot and initiated it using sudo systemctl enable flaskproject and sudo systemctl start flaskproject commands, respectively. This approach guarantees efficient and automated management of the Flask application as a systemd service, ensuring seamless execution and startup procedures.

[sanjeeb@nginx system]$ cd /etc/systemd/system
[sanjeeb@nginx system]$ vim flaskproject.service

After=network.target

[Service]
User=sanjeeb
Group=nginx
WorkingDirectory=/home/sanjeeb/flaskproject
Environment="PATH=/home/sanjeeb/flaskproject/flaskprojectenv/bin"
ExecStart=/usr/bin/bash -c 'cd /home/user/flaskproject; source flaskprojectenv/bin/activate; uwsgi --socket 0.0.0.0:5000 --protocol=http -w flaskproject'

[Install]
WantedBy=multi-user.target
[sanjeeb@nginx system]$ sudo systemctl start flaskproject 
[sanjeeb@nginx system]$ sudo systemctl enable flaskproject

Configuring Nginx to Proxy Request

[sanjeeb@nginx ~]$ cd /etc/nginx/
vim nginx.conf

# ADD THE BELOW CODE JUST BEFORE THE DEFAULT SERVER BLOCK

server {
    listen 5000;
    server_name _;

    location / {
        include uwsgi_params;
        uwsgi_pass unix:/home/sanjeeb/flaskproject/flaskproject.sock;
    }
}

Setting Permission to the user

[sanjeeb@nginx nginx]$ sudo usermod -a -G sanjeeb nginx
[sanjeeb@nginx nginx]$ chmod 710 /home/sanjeeb

# CHECK THE CONFIGURATION SYNTAX
[sanjeeb@nginx nginx]$ sudo nginx -t 

# START AND ENABLE THE NGINX 
[sanjeeb@nginx nginx]$ sudo systemctl start nginx
[sanjeeb@nginx nginx]$ sudo systemctl enable nginX

# REBOOT AND CHECK THE APPLICATION STATUS IN BROWSER
[sanjeeb@nginx nginx]$ reboot

# CHECK IF PORT IS UP AND RUNNING
[sanjeeb@nginx nginx]$ netstat -tnl

If you want to run the application from external machine, then add port in the firewall.

[sanjeeb@nginx nginx]$ sudo firewall-cmd --permanent --add-port=5000/tcp

Setting up Jenkins

As Jenkins is Java-based, we’ll ensure proper installation of Java alongside configuring Jenkins to ensure seamless functionality.

Let’s proceed with the installation prerequisites. Since Jenkins relies on Java for its execution, our foremost task is to install the appropriate version of OpenJDK:

yum -y install java-17-openjdk

After a successful installation, it’s imperative to validate the installation and configure the JAVA_HOME environment variable:

java -version
ls /usr/lib/jvm/java-17-openjdk-17.0.6.0.10-3.el9.aarch64

With Java seamlessly installed and configured, we’ll progress towards Jenkins installation. This necessitates adding the Jenkins repository and importing the respective repository key:

bashCopy code
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key

Subsequently, we’ll execute the installation of Jenkins utilizing the yum package manager:

yum install jenkins

Following the installation, we’ll proceed to start and enable the Jenkins service to ensure its continuous operation:

systemctl start jenkins
systemctl enable jenkins

As Jenkins inherently operates on port 8080, we’ll verify its operational status and accessibility:

systemctl status jenkins
netstat -tnl | grep 8080

To access Jenkins via the web browser, we’ll obtain the server’s IP address:

hostname -I

Utilizing the obtained IP address along with port 8080, we’ll access Jenkins in the browser. Upon initial access, Jenkins will prompt us to input an administrative password, which can be located within the Jenkins interface.

Upon authentication, we’ll proceed with the initial setup, encompassing the installation of suggested plugins and the configuration of the admin user.

Jenkins dashboard

View admin password

[root@jenkins ~]# cd /var/lib/jenkins/secrets
[root@jenkins secrets]# ls
hudson.console.AnnotatedLargeText.consoleAnnotator
hudson.console.ConsoleNote.MAC
hudson.model.Job.serverCookie
hudson.util.Secret
initialAdminPassword
jenkins.model.Jenkins.crumbSalt
master.key
org.jenkinsci.main.modules.instance_identity.InstanceIdentity.KEY
org.springframework.security.web.authentication.rememberme.TokenBasedRememberMeServices.mac
[root@jenkins secrets]# cat initialAdminPassword 
f91c238bebb14c1cb68ccc206fef8a07

Go to the new item from the sidebar.

Enter the item name and select freestyle project.

Install the plugins named ‘published over ssh’ from Manage Jenkins > Plugins.

Go to Source Code Management, and enter the Git Repository URL , also add the credentials which is in the manage jenkins section inside credentials.

In the configuratoin of the created job, inside the Source code management section, select git and then add the repository and credentials.

In the build trigger section you can do cron job from the poll SCM Section.

In the build Steps, add the SSH Server which is created from the Manage Jenkins > System.

Then Test the configuration, from the Test configuration button , which should show success status. And save and build the project. The console output shows the output of the build steps. Now any thing you push from the developer machine will be reflected in the Webserver through CICD Pipeline.

--

--

Kcsanjeeb

Written by Kcsanjeeb

12 Followers

sanjeebkc.com.np

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams