Example of Automated Test Integration in Jenkins: Jenkins + Docker + poetry + pytest + Git

This article might be useful to those who are relatively new to Jenkins. I will provide a simple CD/CI example. It will be a simple python app fetching the ip address, dockerized with Poetry and pytest. After that we will run a Jenkins pipeline, that checksout the app in the master branch, runs the tests and pushes to the release branch.

Part 1: Python script + pytest + pyproject.toml

There will be a very simple python script fetching the ip address using requests library.

Requests library doesn’t go by default in python, so we can see in this example that Poetry will take after the installation. Here is how my test123.py file looks like.

import requests
import time
def main():
    r = requests.get('https://api.ipify.org?format=json')
    print(r.text)
    print(F'time now is {time.time()}')
if __name__ == "__main__":
    main()

Here is my pyproject.toml looks. It can be noticed that we include requests library to poetry.dependencies.

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
[tool.poetry.dependencies]
python = "^3.7"
requests = "^2.28.1"
[tool.poetry.dev-dependencies]
pytest-cov = "^3.0.0"
pytest = "^6.2.5"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

The folder tests contains our single test file my_first_test.py with two simple functions returning successful assertion.

def verify(x):
    assert 1 == 1
def test_system_exit():
    # uncomment the next line to test the pipeline with failed tests
    # raise SystemExit(1)
    assert 1 == 1

The command to run the tests is python -m pytest -v --cov-=. --cov-report=xml:/output/coverage.xml. We will put this command in an entrypoint script that will be executed as our ENTRYPOINT when the Docker container starts.

So entrypoints/pytest_entrypoint.sh will look like

#! /bin/sh
set -e
python -m pytest -v --cov=. --cov-report=xml:/output/coverage.xml

Part 2: Dockerfile

Here is the Dockerfile

Keypoints here:

We use slim as the base image instead of alpine. In alpine version we end up installing many things, eventually resulting in a bigger image and taking a long while to build. Especially, if you have many Python packages to be installed, it’s very likely to end up with a longer and bigger build.

We install poetry directly from the URL as in the documentation.

After the installation, it’s necessary to update PATH with the Poetry bin.

Dependencies are installed directly in the python interpreter of the container.

Part 3: Jenkins pipeline

I assume that you already have a test git repository in this project and Jenkins running. In this example, we create a Jenkins pipeline that takes the project from master branch and in case of successful tests, pushes it to a new release branch. If for example, the latest release branch is release-5, then the script is going to create a branch release-6.

The pipeline has the following stages:

Fetching the content of the master branch.

It can be done automatically if choosing Pipeline script from SCM when creating a Pipeline in Jenking UI. However it wasn’t somehow fetching the right branch in my case and I find checkout function in the script better as it gives more control

Here more details about this function can be found.

So in my case it looked like this

checkout([$class: ‘GitSCM’, branches: [[name: ‘refs/remotes/origin/master’]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: ‘LocalBranch’, localBranch:’**’]], submoduleCfg: [], userRemoteConfigs: [[refspec:”+refs/pull/*:refs/remotes/origin/pr/*”, refspec:”+refs/heads/*:refs/remotes/origin/*”,credentialsId: ‘abc’, url: ‘XXX']]])

Bulding Docker

docker.build(“test-app-image:${env.APP_BUILD_ID}”, “ --network=host -f Dockerfile .”)

In this command the usage of Docker plugin is demonstrated. More details about it can be found in this plugin documentation.

Running container with tests

Now can start the container point to shell script with test commands as the ENTRYPOINT.

docker run --network=host -v ~/Development:/output test-app-image:0.1 /bin/sh entrypoints/pytest_entrypoint.sh

We mount ~/Development and /output for XML coverage test results. According to the command in the entrypoint shell script:

--cov-report=xml:/output/coverage.xml

Starting container with the main python script.

docker run --network=host -v ~/Development:/output test-app-image:0.1 python3 test123.py

As we can see, poetry made its job with all necessary installations and the library requests is installed, test123.py fetches the ip address from the remote URL.

Pushing to a new release branch

A new release branch is defined in the release shell script as

last_release_branch=”$(git branch | grep ‘release-’ | sort — version-sort -r | head -1)”
last_release_number=${last_release_branch##*-}
new_release_branh_name=”release-$((last_release_number+1))”

Then we checkout to this newly created branch.

Afterwards in Jenkins Pipeline we retrieve the name branch by running is NEW_RELEASE_BRANCH="${sh(script:'git rev-parse --abbrev-ref HEAD', returnStdout: true)}" and push the release to it using the Git plugin function withCredentials.

It looks as the following:

withCredentials([ 
    gitUsernamePassword(credentialsId: ‘gitea_auth’, gitToolName: ‘Default’) 
]) { 
    sh “git push — set-upstream origin ${NEW_RELEASE_BRANCH}” 
}

Similarly, the code can be adapted to use tags, e.g. incrementing by 0.1 as in the following snippet

last_tag=”$(git tag — sort=-v:refname | head -1)”
new_tag=`echo $last_tag 0.1 | awk ‘{print $1 + $2}’`
git tag -a “$new_tag” HEAD -m “tagging release”
git push — tags

This is an example of a basic workflow which can be extended in many other way.

Here is my GitHub repo with all the files for this project.

GitHub - iskandre/jenkins-poetry-docker-example