Continous Integration and Continous Deployment

CI and CD using Jenkins and Git webhooks

This post will help you to automate your deployment using jenkins CI and CD pipeline.

How to trigger the job in 3 steps:

Do it in a few steps ahead

1. Job URL:

Get the Jenkins job URL (ex: https://buildtools.com/job/job-name)

2. API TOKEN:

Get your API token from Jenkins account. (ex:fsdjlfs76dfsf6fsdfsf)

3. Create Job hook URL:

Generate the build hook URL with this pattern : http://<jenkins_username>:<jenkins_API-TOKEN>@<job_url>/build

NOTE: if your job having build parameters use buildWithParameters instead of build.

Now, your job URL is ready to call. wherever u want to trigger you can just call.

Example use cases:

some sample use cases are listed below:

Integrating with Github :

• Login to your GitHub account.
• Click on your repository
• Click on your repository settings (settings can be visible only for repository admin)
• Click on Add Webhook
• paste your Job hook URL in payload URL input box
• select the all events or specific event to trigger the job URL.
• click on Add Webhook
• check your Jenkins job by making some event in GitHub.

WORKFLOW DIAGRAM

Whenever a Pull re quest is raised in github the following script will deploys the code base from the current feature branch. You can define the jenkins to run various api tests, integrations tests jenkins jobs and verify the changes before merging the code to master.

Once the pr is merged into master the following script will deploys the code base into any of the specified environments such as Dev,Test,UAT, Prod.

By default pull request payload will give the events for various payloads such as opened, closed, reopened, edited, assigned, unassigned, review requested, review request removed, labeled, unlabeled, synchronized, ready for review, locked, or unlocked.

Working steps:

• Communicating with Github from python scripts PyGithub package was used.
• PyGithub is a Python (2 and 3) library to use the Github API v3. With it, you can manage your Github resources (repositories, user profiles, organizations, etc.) from Python scripts.
• Firstly, install PyGithub in python packages using below command

pip install pygithub

• Please find attached python script for getting the latest pull request branch/release tag name. This takes the Latest branch name if commit is from pull request branch/release tag otherwise it takes branch name as null.
• Next created the web hook event for each repo in git hub with pull request and Releases events. So whenever any pull request/tags raised/changed our web hook triggers Jenkins job.
• Here one limitation for pull request event has multiple events. Those are edited, assigned, unassigned, review requested, review request removed, labeled, unlabeled, synchronized, ready for review, locked, or unlocked.
• So whenever developer close pull request or edit pull request name then also hook triggers the Jenkins job. Because of this multiple trigger in Jenkins job that triggers the multiple ui/job manager Jenkins job triggers. So this is very time consuming.
• So need to prepare script that hook triggers the ui/job manager Jenkins job only when pull request opened, re-opened and merged events occurred.
• For this, added some code for filtering pull request events from Github in python script that triggers the Jenkins basic job that runs python script. If output of python script returns null then basic job exits and main job (ui/job manager) not triggered otherwise Jenkins job goes further to main ui/ job manager Jenkins job.
• In this way main Jenkins job triggers was limited such that web hook triggers the basic Jenkins job every time but not the main downstream jobs.
• Below is the Jenkins trigger flow for latest pr branch.

Use the following python script to filterthe opened, closed and merged events from the pull request payload.

• Firstly, install PyGithub in python packages using below command

pip install pygithub

• Please find attached python script for getting the latest pull request branch/release tag name. This takes the Latest branch name if commit is from pull request branch/release tag otherwise it takes branch name as null.
• Next created the web hook event for each repo in git hub with pull request and Releases events. So whenever any pull request/tags raised/changed our web hook triggers Jenkins job.
• Here one limitation for pull request event has multiple events. Those are edited, assigned, unassigned, review requested, review request removed, labeled, unlabeled, synchronized, ready for review, locked, or unlocked.
• So whenever developer close pull request or edit pull request name then also hook triggers the Jenkins job. Because of this multiple trigger in Jenkins job that triggers the multiple ui/job manager Jenkins job triggers. So this is very time consuming.
• So need to prepare script that hook triggers the ui/job manager Jenkins job only when pull request opened, re-opened and merged events occurred.
• For this, added some code for filtering pull request events from Github in python script that triggers the Jenkins basic job that runs python script. If output of python script returns null then basic job exits and main job (ui/job manager) not triggered otherwise Jenkins job goes further to main ui/ job manager Jenkins job.
• In this way main Jenkins job triggers was limited such that web hook triggers the basic Jenkins job every time but not the main downstream jobs.
• Below is the Jenkins trigger flow for latest pr branch.

from github import Github
import os
#get the repo
g = Github(os.getenv("git_auth"))
repo=g.get_repo("inmar/" + str(os.getenv("git_repo")))
#get all pull requests from repo
pulls =list(repo.get_pulls(state='open'))
#get brnaches for all pull requests
br_list= [p.head.ref for p in pulls]
#get all issue events like assigned pr,unassigned pr and edit pr name
issue=list(repo.get_issues())
#get all events from repo like push/commit in pr,merge pr into master and release tags created/modified
events=list(repo.get_events())
#list events which are no need to run scripts
events_is=["renamed","assigned","unassigned"]
iss_dat=[]
#get the updated time of latest push event
if(events):
push_dt=events[0].created_at
# check the latest event type from events and take the specific branch
if events[0].type=="PushEvent":
event = events[0].payload["ref"].split("/")[2]
elif events[0].type=="PullRequestEvent":
event = events[0].payload["pull_request"]["head"]["ref"]
elif events[0].type=="CreateEvent":
event = events[0].payload["ref_type"]
else:
event="null"
else:
event="null"
# Check issue events and get issue dates
if(issue):
for j in issue:
main=list(j.get_events())
if (len(main)>0):
iss_event=main[-1].event
iss_dat.append(main[-1].created_at)
else:
iss_event= "null"
if (len(main)>0): iss_dt= max(iss_dat)
else:
iss_event= "null"
# Check when new tag was release/modified
if event == "tag":
if iss_event in events_is and max(push_dt,iss_dt) == iss_dt:
print("null")
else:
print(events[0].payload["ref"])
# Check any pull request created/updated and compare issue events, push events
elif len(pulls)>=1 and event in br_list and iss_event in events_is:
if max(push_dt,iss_dt)==iss_dt:
print("null")
else:
print(event)
#Check when any pull request created/updated
elif len(pulls)>=1 and event in br_list:
print(event)
#Check when any pull request merged with mater or not
elif (len(pulls)==0 or len(pulls)>=1) and event == "master" and events[1].type=="PullRequestEvent" :
push_dt=events[1].created_at
if iss_event in events_is and max(push_dt,iss_dt) == iss_dt:
print("null")
elif events[1].payload["pull_request"]["merged"] == True:
print("master")
else:
print("null")
else:
print("null")

Use the following jenkins handler pipeline to run various jobs such as testing,build and deployment.

pipeline {
agent { docker {
label 'slave_name'
image 'python:3.7'
args '-u root:sudo -v $HOME'
} }
parameters {
string(name: 'git_repo',
description: 'Default repo ',
defaultValue: 'default_value')
}
stages {
stage("python execution"){
steps{
echo "$git_repo"
git branch: 'git_branch', url: 'git_url'
sh 'pip install pygithub'
script{
GIT_BR = sh(script: 'python -c "import jenkins_trigger.jenkins_trigger;"', returnStdout: true)
//GIT_BR="v0.01123"
br=GIT_BR.trim()
echo br
if (br=="null"){
echo "!!!!No latest commit on pull request branch!!!!"
}
}
}
}
stage("scheduling jobs"){
when {
expression{br!="null"}
}
steps {
script{
echo "****Latest pull branch commit on master****"
if (git_repo==git_repo")
{
build job: 'job to run intregration tests', parameters: [[$class:    'StringParameterValue', name: 'git_branch', value: GIT_BR]]
}
if (git_repo=="git_repo")
{
build job: 'job to run apitests, parameters: [[$class:    'StringParameterValue', name: 'git_branch', value: GIT_BR]]
}
build job: 'job to build, parameters: [[$class:    'StringParameterValue', name: 'git_branch', value: GIT_BR]]
}
build job: 'job to deploy, parameters: [[$class:    'StringParameterValue', name: 'git_branch', value: GIT_BR]]
}
}
}
}
}
}

Please feel free to mail abhinav.narraa@gmail.com for any questions.