Supercharge your Jenkins — GitHub Integration ⚙️ 🚀

Published in

albert-health

6 min readAug 11, 2023

In the realm of software development, having the right tools can revolutionize productivity and collaboration. Jenkins and GitHub stand out as two exceptional platforms that have earned developers’ trust for streamlining their processes. An open-source automation server, Jenkins empowers teams with its versatile CI/CD pipelines, while GitHub’s robust version control system ensures seamless code management and collaboration.

The integration of Jenkins and GitHub presents a matchless synergy, automating build triggers and testing in response to code changes. By mastering this integration, developers can unlock the true potential of these tools, simplifying workflows and fostering seamless collaboration among team members. However, achieving this integration can be a daunting task. In this article, we explore how to effortlessly trigger both pull request and development environment builds and tests in Jenkins, and report their status back to GitHub, within a single project.

A huge shoutout to CloudBees and their amazing documentation about GitHub and Jenkins!

Assumptions

Development workflow contains a production branch, a development branch, and separate feature branches.
Features are developed on feature branches, and then they are merged into the development branch with pull requests.
There may be occasional commits to the development branch for small hot-fixes.

Motivation and expected outcome

When a commit is made on the development branch or a pull request to the development branch is opened, a new build process, which includes automated tests, will run on Jenkins.
After the build process is over, its success/fail state will be reported back to GitHub, so buggy parts of the code that make test cases fail do not get merged into development and production branches.

Current Usage of GitHub — Jenkins Integration in Albert Health’s NLP Module

Step 1: “GitHub App”

According to GitHub’s official documentation:

A GitHub App is a type of integration that you can build to interact with and extend the functionality of GitHub. You can build a GitHub App to provide flexibility and reduce friction in your processes, without needing to sign in a user or create a service account.

Why GitHub App? We need the following connections between GitHub and Jenkins:

New commit — webhook (GitHub → Jenkins)
Build success state reporting (Jenkins → GitHub)

Even though the first one can be achieved by GitHub webhook and the second one can be achieved by GitHub Status API, setting up both of these and getting them right is not trivial. GitHub app can do both of these under the hood, and as a plus, it has many more small QoL improvements.

This tutorial from Cloudbees can be followed for GitHub app creation, adding GitHub app to the repository, and adding GitHub app to Jenkins : https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/github-app-auth

Step 2: Multibranch pipeline project creation and configuration

Why multibranch pipeline? It is the most compatible option with GitHub App and has some nice features tailored for projects that need builds from multiple branches.

Steps before “Creating a GitHub Organization project” part can be followed from this tutorial: https://docs.cloudbees.com/docs/cloudbees-ci/latest/cloud-admin-guide/github-branch-source-plugin#_example_multibranch_pipeline_project

A multibranch pipeline can be configured from the left “Configure” tab after multibranch pipeline is clicked.

Branch Sources

Branch sources are integral parts of multibranch pipelines. They are used to discover one or more branches, based on some filtering. One or more branch sources can be added to the multibranch pipeline. Also, adding multiple instances of the same type of branch source is possible. There are 3 kinds of branch sources:

GitHub
Git
Single repository & branch

When “Git” or “Single repository & branch” is selected in a multibranch pipeline project, triggering builds on commits with webhooks is not trivial, so GitHub source has the serious advantage here.

Branch and Pull Request Discovery

For each branch source instance added, one or more branches can be discovered in all 3 kinds of branch sources, but pull requests are only discoverable in GitHub sources.

In each branch source instance added, branches to include can be set from “Discover branches” -> “Strategy” options:

Exclude branches that are also filed as PRs
Only branches that are also filed as PRs
All branches

There are different strategies for discovering pull requests:

Merging the pull request with the current target branch revision: Both commits to “base” branch and “compare” branches after pull request is open will be tracked and included in the build.
The current pull request revision: Only commits to “compare” branch after pull request is open will be tracked and included in the build.
Both: Builds both of them with different build names.

After discovery, branches can be filtered (included and excluded) by “Filter by name (with wildcards)” option.

Branches left after filtering can be built only with Jenkinsfile.

Specific Configuration for Triggering both Pull Request and Development Branch Builds

After creating the multibranch pipeline, two instances of GitHub branch sources should be added by providing the same repository URL and credentials. One is for pull requests and the other one is for building the development branch.
In the first one, for pull requests, under “Behaviors”, “Discover pull requests from origin” should be added and an appropriate option of choice can be selected as a strategy.
In the second one, for development branch, under “Behaviors”, “Discover Branches” should be added, and the strategy “Exclude branches that are also filed as PRs”. Then, “Filter by name (with wildcards)” should be added with include the development branch, like dev and exclude other branches, like main feat/** fix/**

Step 3: Jenkinsfile

When multibranch pipeline is selected as the type of the project, Jenkinsfile is the only option for build configuration. This file can be created at the root of the project directory. Its syntax is mostly like nested curly braces.

Example Jenkinsfile:

pipeline {
    agent any

    environment {
        ENVIRONMENT_FILE = credentials("NLP_SERVICE_TEST_ENV")
    }
    stages {
        stage('Build') {
            steps {
                sh 'cp $ENVIRONMENT_FILE .env'
                sh 'docker build -t nlp_service'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            when {
                not {
                    changeRequest()
                }
            }
            steps {
                echo 'Deploying....'
            }
        }
        stage('Cleanup') {
            steps {
                sh 'docker system prune -f'
            }
        }
    }
}

environment: Environment variables can be set inside this block. Its syntax is the same as .env files. Also, inside this block, secrets added to the Jenkins can be accessed with “credentials()”. Defined environment variables can be used inside other commands with $
stages: This block contains individual stages. Many blocks can be used inside stages, including but not limited to “when” and “steps” blocks. Each stage and its run time will be visualized on the “Status” page of the build.
when: Adds conditionality to the stage. The stage will not run unless the inside of “when” is true.
steps: Many commands can be used inside this block, including but not limited to “echo” and “sh”. Any string that is provided to “sh” will be evaluated as shell commands.

Specific Details from Jenkinsfile of Albert NLP Module

Environment variables are stored in a Jenkins secret file with ID “NLP_SERVICE_TEST_ENV”. In order to be used, it is copied from Jenkins with $ENVIRONMENT_FILE and it is renamed as “.env” while copying.
Deployment of the pull request builds (after tests) was undesirable. So, this is prevented by adding all deployment-related commands to “deploy” stage, and by using “when” block, whether or not the build is from a pull request is checked and negated.

Step 4: Adding a secret file to Jenkins and using it during the build

In Jenkins, to manage credentials, navigate to the home page and click on “Manage Jenkins.” From there, select “Manage Credentials” and choose “Domain (global).” To add new credentials, click on “Add credentials” and opt for “Secret file” as the type. Here, you can choose the scope for the credentials. After specifying the file, upload it from your local computer. Optionally, you can provide an ID for the credentials. Upon successful configuration, the secret file will be copied to the working directory with the desired name you supplied. This process ensures secure access to sensitive files within your Jenkins environment, enhancing the overall security of your projects.