To Jenkinsfile or not to Jenkinsfile?
Jenkins is our weapon of choice for CI/CD here at Hootsuite. A key player in CI/CD software, Jenkins is open-source, widely used and has an active community.
How does Jenkins fit at Hootsuite? Jenkins plays nicely with git, is supported by plugins, and allows monitoring/collection of metrics. The previously linked blog posts are a great starting point to learn more about Jenkins if you don’t know what it is. This blog post will introduce the concept of a Jenkinsfile and discuss the benefits and drawbacks when working with a Jenkinsfile and how we use Jenkinsfiles here at Hootsuite.
Why use a Jenkinsfile?
If you already are a user of Jenkins, you must be familiar with manually editing configuration steps for each job in the small boxes. For a long time, this was standard procedure for us here at Hootsuite and continues to be in some legacy projects.
Over the years, we’ve encountered a number of pain-points with this way:
- Any developer can edit the configuration of a job, without any approval/change history process. This introduces a major risk if someone were to completely change or even delete a job.
- Often times, build jobs were very similar to each other. For example, there is an individual job set up for each npm package being published, each essentially a copy of the other. However, once a change was introduced in the configuration of one, the others become outdated. It would require a manual effort to keep track of which jobs to update and which jobs correspond to which repository. Wouldn’t it be nice to have some source control and ability to pull configurations from a central repository?
- Developer experience
- Writing code in Jenkins is not the same as writing code in an IDE. the boxes have limited space, no indenting, no syntax checking, etc. This contributes to mistakes being made and lot of copy/paste back and forth. If the developer wants to edit the build scripts, that would require opening Jenkins, finding the job, entering the configuration and finding the correct box. Wouldn’t it be nice if your configuration could live with your code in a github repo?
Enter.. Jenkinsfile! Click the link to read all about it.
With a Jenkinsfile, we can commit the steps of our build pipeline into code, offering many benefits including:
- Version Control
- A Jenkinsfile lives with the code in github. Therefore, like any production code, it can be peer-reviewed, commented and approved before being merged into master and built.
- For Jenkinsfiles that aren’t attached to a specific project, they can be placed in a build-scripts folder and be copied from child projects and used. Because it’s in code, it is easy to visualize what changes were made and search for multiple references.
- Visualization of Stages
- Jenkinsfiles allow visualization of stages, enabling developers to quickly spot which stages are failing more quickly and intuitively.
Enough Talking! Let’s code.
Here is a simple pipeline that checks out a github repository for an npm package and a few commands that build, test and publish this application.
Here is the manual configuration in the Jenkins UI which we will convert:
- Parameter input setting the release type:
- Checking out the git repository
- Build, test and deploy the npm package
- Update the status on a Slack Channel
Tons of opportunities for error right? Let’s go through the Jenkinsfile groovy equivalents step by step:
To allow the user to set a parameter before building, we use this:
Checking out a git repository is simply:
I’ve wrapped this in a “stage” which will show as its own box labelled “checkout” in the pipeline visualization.
Build, test and deploy, again wrapped in stages to visualize:
Finally, notifying the team’s channel on Slack:
Here’s the full Jenkinsfile that does the equivalent of this. Copy/customize this file, then save it as “Jenkinsfile” to the root of the github repository being built.
Great! How would I add this to the build process?
Let’s assume you already have the github and pipeline plugins installed on your Jenkins instance and your/your company’s github organization already lives on jenkins. Simply go to your repository and hit “Scan Repository Now”. This scans each github repository under your username/organization and finds these embedded Jenkinsfiles.
For Jenkinsfile projects, this shows up in the configuration:
The scan should also trigger a build automatically. The first build may fail because the release type parameter isn’t entered. Try a “Build with Parameters” again if that occurs. When you see this…Congratulations on completing your first Jenkinsfile pipeline!
How do I test out changes to my Jenkinsfile?
Say I wanted to add a stage to my Jenkinsfile that runs a linter on my code. Would I have to wait until my changes are merged to master to see if the build process works?
No! Any commits to a branch are picked up and built, including commits to the Jenkinsfile. Therefore, simply pushing a change to the Jenkinsfile on branch will trigger a run-through of the Jenkinsfile, automatically testing your changes, without affecting the main pipeline. Another great reason to use Jenkinsfiles!
For review, here is a cheat-sheet for commonly used commands in Jenkinsfiles:
When are some drawbacks or considerations with Jenkinsfiles?
Before you go crazy converting all your Jenkins jobs to Jenkinsfiles, here are some considerations to think about:
Loss of Plugins
If your build relies heavily on plugins, it will be difficult to integrate those functionalities with Jenkinsfiles.
For jobs that utilize less well-known plugins or in-house developed plugins, they would not have an equivalent Jenkinsfile command out of the box. For example, in our Smoke Test portion of our legacy pipeline, we used the Image Gallery plugin heavily to display screenshots of failed Selenium Tests in the job, but this did not have a Jenkinsfile command. As an alternative, we send screenshots via cURL commands to the respective slack channel rather than display them in the Jenkins UI.
With that being said, very popular workflows such as checking out from github or sending a message to slack have their own commands. For basic pipelines, this should not be an issue.
What if one day we wanted to switch to GoCD? TravisCI? CircleCI? There’s tons of CI/CD software out there and converting all build scripts to Jenkinsfiles restricts us to Jenkinsfile specific commands and using Groovy. As a result, we’ve coupled ourselves to the Jenkins environment.
To work around this issue, we’ve written many of our re-usable build scripts in python or shell.
We then use the shell script function to execute these python or shell scripts from the repository.
Shell and Python are both languages that are widely supported and portable compared to groovy, which hopefully can make moving systems easier if that day comes.
For teams that haven’t used Jenkins at all before, there is also some learning to be done. However, I would argue that with Jenkinsfiles, build pipelines become part of the code, bringing the developer organization one step closer towards Pipeline as Code, contributing to a self-managed developer CI/CD experience. These reasons, along with all the other benefits that Jenkinsfiles provide like security and visualization, are what makes the learning curve absolutely worth it.
About the Author
Connie Ho is a co-op student on the Build Test Deploy team in Production, Operations and Delivery. She is currently studying Computer Science and Microbiology & Immunology at the University of British Columbia. Connect with her on Linkedin.