DevOps Use case: Jira Jenkins Integration
DevOps fever rages on in the entire world. Job Openings are soaring, tool combinations are being explored, and Automation is as always fighting AI for the “buzzword of the year” title. Amidst all of this, certain tools are trying to become de facto for Devops, such as Jenkins for CI/CD pipeline, Jira for project management, Terraform for infrastructure provisioning etc. All of this, to gain upper hand in automation.
While automation is clearly the way to go, we ask a bigger question. Can DevOps also enable ease of use for a project manager? Can we perhaps create a way for a Jira project manager to get a big picture of what the hell is happening in the project? In this DevOps use case, we explore one such particular combination of DevOps tools. This combination, instead of focusing on automation, focuses on information. This combination uses the most standard tools available, namely Jenkins for CI/CD pipelines, and Jira for project management, to build an effective tracking mechanism for a project manager. Furthermore, this use case would give an insight to how a DevOps aim is achieved by designing and integrating different tools based on our objectives.
Unlike other articles, this one is aimed at specific DevOps engineers (sorry Layman folks!). So basic proficiency in DevOps tools such as Jira, Jenkins, Cloudformation, Terraform etc. is required. For this tutorial, user should know how to setup Jira and Jenkins servers, how to install plugins in Jenkins, and how to create pipelines in Jenkins. User should understand what is a base template for cloudformation.
As an added bonus, this tutorial has lots and lots of pictures (and github repo with code and terraform/cloudformation templates, along with steps to create a new pipeline!)
The Use Case: Ease the life of Jira PM
The overall aim that we are trying to achieve is to have a tracking mechanism of all the different Jenkins pipeline for Jira project manager. We are going to heavily customize Jira user interface for this project. Additionally, we are going to modify our Jenkins pipeline to integrate it with Jira. For this use case, we assume a migration project (because that’s where the money is $_$).
Now, there are several hundred Cloud migration steps being executed. Every migration is of different types, like Rehost (lift and shift), Refactor (rearchitect), Replatform (Lift, Tinker and Shift) etc. It also involves a series of steps, such as feasibility analysis, landing zone creation (creating infrastructure), app migration etc.
Overall, for this use case, we are trying to achieve two major objectives related to migration:
- Achieve greater synchronization between Jira PM and Jenkins DevOps: Jira project manager can keep track of all the migration pipelines. At any point, when there’s a build failure and the pipeline has to abort, it’s possible that Jenkins populates all the details in Jira that would help Jira project manager check logs and troubleshoot the error faster. This enables Jira project manager to get a bird’s eye view of all logs and issues associated with migration.
- Automate Jira story updates: Jenkins can automatically update Jira with statuses relevant to the build process (eg: aws_deployed_success). This saves manual overhead of regularly updating the story and keeping track of changes. Additionally, Jira user can fetch the latest updates required for migration such as new migration keys for managed services (eg: new EC2 parameters), new migration processes (Terraform, Cloudformation etc.) from Jenkins.
Design:
To begin with the design, we assume a very simple migration is being performed. In this, a developer updates the application in a github repo. Then, the pipeline automatically fetches the application, creates a WAR file, creates infrastructure for migration (EC2, VPC), and deploys the WAR application there. So essentially, fetch and shift!
We would be needing to collect the parameters for migration (template and input files for Cloudformation, Terraform etc.), and then provision the required infrastructure on Cloud for the migration to be successful.
We assume all the required parameters for migration are kept in a central repo folder. Additionally, we assume there is a recipe with migration instructions in the same repo, which would enable Jenkins DevOps user to perform complex migrations. For some simple migrations, automatic provisioning can be performed without need of this recipe. For some others, these instructions can be included in Jenkins pipeline only.
The overall design is represented below:
The steps for migration are mentioned below:
We are assuming that Jira and Jenkins have already been installed on their respective servers. We are using Jira trial server version, and standard Jenkins installation. Installation is not covered in this tutorial. Additionally, we would be modifying an existing migration Jenkins pipeline. So creating a new pipeline is not covered in this tutorial. We do cover how to modify an existing pipeline to work with Jira.
Jira customisation:
For migration to be successful, we would be needing following bare components in JIRA user interface:
- Base template file: Cloudformation or Terraform base template, which defines how to provision the infrastructure.
- Cloud platform: To select the Cloud platform on which to migrate. There can be several different contexts for it. Like for AWS, it can be EC2, RDS, WAR CREATED, MIGRATE SERVER etc.
- Parameters file or input: The actual input variables for the base template, such as ami size, vpc cidr etc. These can either be provided in input, or as a URL of a location or a repo (github).
- Parameter FLAG: A flag or a counter, to control the flow of information from Jira to Jenkins and vice versa. Suppose, we need certain parameters from Jenkins and then supply these updated parameters back, then we use this flag.
We would also be needing a way to track the status of the migration. For that, we would be updating the workflows of Jira. Traditionally, it comes with “TO DO”, “IN PROGRESS”, and “DONE”. We can update it with different steps, such as “EC2 CREATED”, “WAR CREATED”, “SERVER MIGRATED” etc. This is the heart of Jira-Jenkins integration, as using these workflows, a Jira project manager can track all of his stories easily.
Customize JIRA User Interface:
Step1: From Jira dashboard, click on the “cog wheel” in upper right corner, and then click on Issues.
Step2: Scroll down, and click on Custom Fields under FIELDS section, in left hand side of Jira. Then, click on Add custom field.
Step 3: Create 5 Custom fields, namely:
BUILD STATUS: Text Field (multi line): Leave default value as blank.
CLOUD PLATFORM: Select List (cascading): Enter “AWS” and then under AWS, enter “EC2”, “RDS”, “CREATE WAR”, “DEPLOY WAR”, “SERVER MIGRATED” and “OTHERS” as values. I have also added GCP, as a dummy placeholder.
FILENAME PATH: Text Field (multi line): Enter a default value of “<Insert Base Template GITHUB URL here if required>”
PARAMETERS: Text Field (multi line): Enter a default value of “<Insert KEY-VALUE pairs or GITHUB URL of KEY-VALUE pairs here if required>”
PARAMFLAG: Select List (Single Choice): Enter “PARAMETERS UPDATED” and “WAIT FOR NEW PARAMETERS” as values.
The steps to create these custom fields are pretty straightforward. Do note the custom field ID of each field created, as it would be used with Jenkins integration. This ID can be seen in the URL of each custom field, when configuring or editing it.
<JIRA URL>/secure/admin/ConfigureCustomField!default.jspa?customFieldId=10009
Alternatively, it is visible in the URL when you hover on EDIT field:
The overall Jira GUI should look like this when completed:
Jira status customization:
Now that we have customized Jira dashboard, we work on the Jira status. Statuses are what controls the information flow. A Jira PM can observe the changing status, and using these can determine the state of the migration. Naturally, we have to create these status as relevant to migration as possible.
For this tutorial, we have used following Statuses: REVIEW, UPDATED PARAM, LZONE CREATED, WAR CREATED, EC2 CREATED, RDS CREATED, SERVER MIGRATED, WAR DEPLOYED.
To create a status, click on same “Cog wheel” on right hand side, click on Issues, and then scroll down and select Workflows in the left hand pane. Then, click on edit.
Then, click on Diagram, and then click on +Add Status. Here, enter your status name, and then make sure to tick “Allow all statuses to transition to this one” box.
Voila, now you can see this “test” workflow. We have already created other relevant status as shown below.
Additionally, we can create status in STATUS pane in the left hand side. Here, we can add new categories such as “TO DO”, to differentiate betwern colors or separate different statuses logically. We have categorized “UPDATED PARAM” and “REVIEW” status with “TO DO” category for this demo.
Finally, as a last step, click on the TEXT icon (right beside Diagrams, where you created statuses), and note down the Transition ID (given in 3rd column). These IDs are important, to control the status from Jenkins.
Finally, click on Publish button (very visible button, where you are adding status). A pop up window will come. Here select “Save a backup copy option” if you want to have backup. Click on publish again in the window.
You can now see these workflows in the issues, while selecting workflows:
Jira scrum board customization:
To enable ease to Jira PM, we need to customize Jira scrum board with different cateogries related to the project. In this tutorial, we create additional columns related to migration: LANDING ZONE, APP MIGRATION, and ERROR REVIEW. Adding new columns is very easy. Just click on the BOARDS in the top ribbon, select your board which you want to customize. Click on Board, and configure.
Then, click on “Add Columns” on the right hand side to add new columns.
You can now put different statuses that you created earlier to their respective columns. These statuses would be updated by Jenkins later on, and hence you can observe the issues moving along these columns as the statuses are upgraded. For example here, WAR DEPLOYED, WAR CREATED, and SERVER MIGRATED statuses are grouped in APP MIGRATION. So whenever an issue is transitioned to these statuses, it moves into this column, hence providing a logical way for a Jira PM to decipher the state of his issue.
Additionally, for this tutorial, we have divided an Epic into different stories, and each story has been assigned different subtasks. It is these subtasks that would interact with Jenkins. So Epics->Stories->Subtasks->Calls Jenkins.
Whew, that was a lot of heavy customization of JIRA. There is one last part still left. Triggering Jenkins using Jira.
Jira webhook integration:
Click on the same “Cog wheel” on the right hand side, then click on “System” this time, then, scroll down (in the very bottom actually), and on the left hand side pane, look for “Webhooks”. Click on that. Then, click on “+Create a Webhook” on the right hand side. Here, add following SUFFIX after your Jenkins server url: /jira-trigger-webhook-receiver/. So for example, your Jenkins server is running on http://10.0.0.1:8080, the webhook will be http://10.0.0.1:8080/jira-trigger-webhook-receiver/.
Additionally, tick the checkbox “created” and “updated” under Issue, and “created” under Comment in the Events section. What we are essentially configuring is that whenever new issue is created or updated, or a comment is created, Jenkins is triggered by Jira. So essentially, by updating our issues or by adding comments, we can control Jenkins pipelines. This is how Jira triggers Jenkins pipeline.
Finally, scroll down and click on CREATE to create this webhook. We can integrate multiple Jenkins pipeline using this feature, as we can create webhooks for multiple Jenkins server URLs.
Unfortunately, webhooks is just one part. You would also need TICKET PATTERN. As we know, in Jira, everything is an issue (including stories, epics etc.). So to keep track of it,Jira system generates a prefix before every issue. Think of it as a Primary Key value for Jira.
To get this Ticket pattern, click on the “Cog Wheel” on upper right side, then click on Project. Click on the project which you want to integrate. Then click on “Details” on the left side. The KEY is the TICKET PATTERN required to integrate with Jenkins.
You would also be needing the Jira server URL, Jira server name and the admin username/password. Except the admin username/password (which you should remember), server URL and name can be found in System Settings.
Finally, we can now move to Jenkins side. All of this customization and noting down IDs would payoff, when we design our Jenkins pipeline.
Jenkins side customization:
Jenkins Plugin customization:
As stated before, we are assuming our pipelines are already built. Now, how can we modify it to support Jira? Well, with Jenkins, everything begins (and possibly ends?) with PLUGINS! So let’s download the following necessary plugins for integration: Jira, Jira Pipeline Steps, Jira Trigger Plugin, and jira-ext Plugin.
You also have go to Configure System in Jenkins, and configure these plugins.
For URL, Base URL, or ROOT URL, provide the Jira Url you saved earlier. Do note, provide the URL without the “/” at the end (as given in Jira system settings).
For Name, provide the Jira title you saved earlier. Do note, the name is case sensitive. So provide exact name (in exact case) which is saved in Jira system settings.
For Ticket Pattern, provide the KEY value you got from Jira Project details, suffixed by “-”. So for DEMO key, it is DEMO-. You can provide different KEY values for different projects, just separate each of them by a comma. This step is very important, as KEY values are the unique primary keys of Jira, which helps Jenkins keep track of Jira issues.
Once all configuration is done, it’s a good idea to restart Jenkins for the plugin changes to take effect. In the future steps, if at any point of time if Jenkins and Jira are not interacting, do retrace these steps. The fault can be either in Jenkins Plugin configuration (incorrect URL, incorrect keys, name) or Jira webhook configuration.
Jenkins Pipeline customization:
Now that Jenkins plugins are installed and configured, we work on the pipelines. This is where things get trickier. Although I have implemented more complex pipelines in which an application war is created from github, server is created, and then the application is deployed, for this demo, I show how to create a simple pipeline to provision EC2 instance using both Cloudformation and Terraform.
The above diagram is very important for Jenkins integration (hence zoomed to the max). There are multiple aspects to it, so let’s go into each one. Please keep referring to this diagram as you read further.
First, to enable Jira to trigger Jenkins, we select “Build when an issue is updated in Jira” option in Jenkins, under “Build Triggers” (see diagram). Now, all the customization that we did in Jira, we see its usage here in the Jenkins pipeline.
1.) JQL Filter: This is what triggers Jenkins build from Jira. As you remember, we created a webhook to trigger whenever an issue is updated from Jira. This webhook is of JSON format, and this JQL filter is what decodes this JSON and matches the custom field values. If it all sounds confusing, just think of JQL filter as field matcher. So in this tutorial, the JQL filter given below signifies that trigger this pipeline ONLY WHEN:
status="IN PROGRESS" AND "CLOUD PLATFORM" IN cascadeOption("AWS","EC2") AND PARAMFLAG="Parameters Updated"- The “STATUS” customfield of Jira matches “IN PROGRESS”. Remember how we created different statuses in Jira? This is where it is matched.
- The “CLOUD PLATFORM” field of Jira matches “AWS” and “EC2”. Remember how we created different cascaded values in Jira for AWS? This is where it is matched.
- The “PARAMFLAG” field of Jira matches “Parameters Updated”.
And so on. We can trigger different pipelines, based on direct matching of the values of the customfields we created in Jira. Remember, the values are CASE SENSITIVE. So match the cases before pulling your hair if the pipeline is not triggered.
2) PARAMETER MAPPING: Remember how in Jira we saved different customfield Ids when we were creating them? Its usage comes here. In order to pass the values of the JIRA fields (like URL, filename, parameters etc.), we need to capture the value. When we click on “Add Parameter Mapping”, we map the inputs given in Jira customfield, with the variables here. In a nutshell, we are storing the values passed from Jira’s customfield in a variable of Jenkins, like we do when we pass API values around.
Now, we can use these variables in the Jenkins pipeline to trigger different builds, provision EC2 using cloudformation (template value of which is provided by Jira and stored in Jenkins in these variables), and do a host of different things.
For example, the values of customfield with the id 10009 (which is PARAMETERS in Jira side) would be mapped to the variable JENKINS_PARAMETER, and can be accessed as $JENKINS_PARAMETER in shell scripts or pipelines. Same goes for customfield with the id 10008 (which is FILENAME PATH), whose value is mapped to GIT_BRANCH.
There is also a system generated parameter mapping called “Issue attribute path”. This maps Jira system generated values, and not our own created custom field values, to Jenkins variable. Its value names are also system generated, and are not customfield ids. It is mostly used to map the primary KEY value generated by Jira.
This is important, as only by using this KEY we can update the relevant Jira issue. Otherwise, Jenkins pipeline would never know which issue to update. This key would be used every single time by Jenkins when it is updating Jira from its own end. Just like Jira uses webhooks to disturb Jenkins, Jenkins uses this key value to disturb Jira. Point to note is that only when Jira triggers a build that Jenkins can capture this KEY value. Otherwise, Jenkins can’t update Jira issue from its own end.
In this tutorial, as you can see from the diagram, we map the value of the primary key (in KEY) to a variable JIRA_ISSUE_KEY.
3) Jenkins Pipeline: Given below is a diagram of a sample Jenkins Pipeline:
It first sets up the flag, on whether to use Cloudformation (CF) or Terraform (TF). It then builds our job (to provision EC2 instance), and uses the parameters passed from Jira (in Parameter Mapping) to pass into the build. It then fetches a recipe from a central repo (github in this case), and displays the instructions. The Jenkins user then executes these instructions (simple commands in this case), and if successful, updates the BUILD STATUS accordingly in the pipeline. This is what is transitioned back in Jira, where the status of the issue is updated accordingly (the different statuses we created are updated from here). Jenkins user can further add his or her comments, and this comment is reflected in the Comment Box of the Jira issue. To know which issue to update, the “JIRA_ISSUE_KEY” variable, which contains the PRIMARY KEY value of Jira issue, is used.
Given below is the groovy script which executes the above pipeline:
Few things to note in the code are:
“PARAMETERS” section: It triggers the build “test” and provides Jira parameters (stored as Jenkins variables) to the build.
job: 'test', parameters: [string(name:'FLAG',value: flag),string(name: 'JIRA_ISSUE_KEY', value: JIRA_ISSUE_KEY), string(name:'GIT_BRANCH',value: GIT_BRANCH), string(name:'JENKINS_PARAMETER',value: JENKINS_PARAMETER), string(name:'BUILD',value: BUILD)]“FIELDS: CUSTOMFIELD_” section: It updates the field directly from Jenkins pipeline. Here, it updates the field with id 10011 (the BUILD STATUS field) with text “Build is successful. Find additional comments in Comment Box”.
def fixedInBuild = [fields: [customfield_10011: 'Build is successful. Find additional comments in Comment Box']]“jiraTransitionIssue”: This is where noting down all the transition Id is paying off. From Jenkins pipeline, we can directly update the status of the Jira issue using the transition id. Hence, on different stages of the build, a Jira issue can be upgraded accordingly. This is what enables a Jira PM to have a bird’s eye view of the status of all the issues in his story board. We can safely that the entire objective of this tutorial is achieved using this single function!
def transitionInput = [transition: [id: '51']]
jiraTransitionIssue idOrKey: JIRA_ISSUE_KEY, input: transitionInput, site: 'JIRA'“jiraAddComment”: This enables Jenkins user to add comments in Jira issue’s comment box directly. We first define a variable “messaging”, which stores the comment from the Jenkins user. You can also supply direct text from the pipeline itself.
messaging = input message: 'Any additional comments?', parameters: [text(defaultValue: '', description: 'Enter any additional comments', name: 'COMMENT')]
jiraAddComment comment: messaging, idOrKey: JIRA_ISSUE_KEY, site: 'JIRA'While there are tons of options (and functions) available to control Jira from Jenkins, for the purpose of achieving our objective, these functions suffice. I have also created another Jenkins pipeline that fetches war from repo, creates infra, and then deploys the war. It uses Terraform, Ansible, Maven etc. The scope of creating this pipeline is beyond this tutorial, so lets leave it for now.
For this tutorial, we have also created parameters collection pipeline, which fetches the parameters from a central github repo, and then posts these parameters to Jira user in the form of a comment. Code for that pipeline is given below:
Jira Jenkins Integration:
Finally, we arrive on the most important part, the integration of Jira with Jenkins. All the steps that we did was to create a to and fro mechanism of information flow between Jira and Jenkins. We saw how, using webhooks, Jira can trigger Jenkins whenever an issue is updated. And we saw how Jenkins, using the captured KEY value, updates the Jira issues and statuses using a set of functions. We also saw how we can trigger Jenkins build using JQL filters, and how we can map Jira field values to variables in Jenkins side.
Now, we use all of this knowledge to first create a simplified flow diagram of what is happening in this tutorial:
Let’s try to understand these steps in a more pictorial form for a simple automatic migration.
STEP1: Jira asks for parameters required by updating PARAMFLAG as “Wait for new paramters” and marking the story as “IN PROGRESS”. The platform on which to perform operations is AWS RDS.
STEP2: Now we get these parameters in the comment box. We note down these parameters, and can either supply them in the form of text input, or in the form of URL. We also need to supply the base template to perform migration. Since the current flag is CLOUDFORMATION, we would supply CLOUDFORMATION base template (.yaml in this case) and input files.
STEP3: Now we trigger our build for RDS Provisioning, by supplying the required parameters. We update the PARAMFLAG as “Updated Parameters”, and then again mark the build as IN PROGRESS. Note how the status has already been changed to UPDATED PARAM. This is controlled by Jenkins.
STEP3: Now the appropriate build would be triggered. It would automatically provision RDS, and then update the status back to Jira user in BUILD STATUS field.
The logs can be found in the comment box:
This is for a simple automatic migration (code given in the end). For complex pipelines, a recipe can be created, where the Jenkins DevOps user executes the steps manually following the recipe. This is what we created in the earlier steps.
Additionally, each stage of the pipeline can update the Jira status accordingly, so that Jira PM can keep track of where the issue is at.
Observing the results:
Remember how our initial objective was to ease Jira PM life? Well, now having understood the integration part, we can observe all the subtask status in a single click. Click on the scrum board in the left hand side, to observe something like this:
As you can see, different user stories have different subtasks. Each subtask has its status upgraded by Jenkins. And each of these subtasks are grouped according to their columns (this task we did earlier in Jira customization). As the status is upgraded from Jenkins side, the issues move into their respective columns. You can also observe all the subtasks’ status by clicking on a User story:
Additionally, you can observe status of a single issue by clicking on it individually.
CONCLUSION:
So this is how the integration of Jira and Jenkins eases PM life. We went through some heavy customization of both Jira and Jenkins.
On Jira side, we created different custom fields related to our project. We also created different statuses, and grouped them together in columns of the scrum board. On Jenkins side, we installed several Jira plugins, and created different build triggers with the JQL filters. We captured the values supplied from Jira, and performed our build (of creating EC2 instance). We saw how custom field IDs, the PRIMARY KEY value, and status IDs were used to update Jira issues using pipeline functions.
Finally, we saw everything in action in an updated SCRUM Board, where we saw different issues with different statuses clubbed together. It provided a bird’s eye view of an entire Epic, and the status of different user stories. This overview enables Jira user to observe multiple Jenkins pipeline and several different stories under one glass.
For enterprise production projects, where 1000s of issues are being executed in 100s of Jenkins pipelines, having such an integration would be massively useful. A onetime modification of Jira according to the project is all that is needed. Then, all new Jenkins pipeline can be integrated or even updated, keeping this modification in mind (the customfield ids, the status ids etc.). This can greatly ease the interaction between Jira and Jenkins. Jira user can fetch all the updated parameters/information from Jenkins with a click of a button. Jenkins can also automatically update Jira issue with error logs, status etc.
Overall, a cohesive integration of two of the most popular DevOps tools in the market would only enhance the productivity and increase the delivery speed.
Bonus:
As promised, find below is the shell script for provisioning EC2 or RDS Instance (depends on Jira trigger) using either Cloudformation or Terraform (depends on the flag) automatically. Note these settings are for individual Jenkins job, and not pipeline (whose groovy script has already been shared).
This script is using the parameters supplied from Jira (base template + input file). It verifies whether these parameters are present from a key file. This key file is fetched from a different folder (/var/lib/jenkins/workspace/test_param/provisioning/aws/keys/ in this case). Additionally, it can be fetched from a github repo as well (for this tutorial, it has already been fetched from a different pipeline, hence using folder name).
The variables used are:
JENKINS_PARAMETER: Stores the input parameters of the required base template. Can be either URL or plain text file.
GIT_BRANCH: Stores the URL of the base template file from the Github repo.
BUILD: The current BUILD, whether it is for AWS EC2 or RDS.
FLAG: Controls the build, whether Terraform or Cloudformation. IS CONFIGURED IN JENKINS AND NOT SUPPLIED FROM JIRA.
For terraform, we use inject environment variables plugin to inject the environment variables from the file. Then, using conditional build plugin, we execute the terraform instructions. The condition is that the terraform properties file must exist.
If you don’t want to use pipeline, and use only Jenkins build to update status, you can easily do it with Jira plugins. It is much easier also, as you don’t have to indulge in complicated groovy syntax. Using Jenkins Build, you can directly update Transition using just the name. You can post comments, and can also update a field directly using customfield Id.
The GITHUB URL, where all the Cloudformation and Terraform base template + input files are stored can be found here:
