Building and publishing Python packages in your Gitlab CI pipeline
In this tutorial, we’ll look at how to package Python code and push to Packagr using Gitlab CI
In the last tutorial in this publication, we looked at how to create a private package repository (your very own private PyPI.org) and push code to it using twine. This time, we’ll look at how to build this into your CI processes — in this case, we are going to use Gitlab, and it’s excellent built-in continuous delivery/integration tools.
Step 1 — write some code
Presumably, if you’re reading this tutorial, you probably already have some code that you’d like to publish to Packagr. Let’s start by sorting the folder structure properly — create an empty folder, my_repository
and add a subfolder to it, my_package
, into which you’ll put your code:
my_repository/
my_package/
__init__.py
... the rest of your code goes here
Step 2 — add some structure
Next, let’s create a Python file called setup.py
in the my_repository
folder:
from setuptools import setupsetup(name='my_package',
version='0.1.0',
description='Some Python package',
author='Me',
author_email='me@example.com',
license='MIT',
packages=['my_package'],
zip_safe=False)
Step 3 — Create the package
We now have the basic folder structure we need to create a Python package. To do this, open a terminal window from your my_repository
folder and enter the following commands:
pip install wheel
python setup.py sdist bdist_wheel
The first of the two commands above installs wheel
, which allows us to create wheel packages by using the bdist_wheel
argument. If all goes well, we should see that our folder structure now looks something like this:
my_repository/
build/
dist/
my_package-0.1.0.tar.gz
my_package-0.1.0-py3-none-any.whl
my_package/
__init__.py
... the rest of your code goes here
setup.py
Step 4 — test the upload to Packagr
Before going any further, let’s check that we can upload this package to Packagr.
Firstly, you will need to create a Packagr account, if you don’t already have one — Just go to Packagr and follow the simple steps to create an account (a free one is fine for this example). Once you’ve created and verified your account, login and click on create new package. This will display the commands you need to upload packages to your repository:
Let’s try this out now. Open a terminal window and cd into your dist
folder, where your newly created packages are now stored, and enter the commands. If all goes well, the output should look something like this:
$ twine upload --repository-url https://api.packagr.app/63cdQSDO/ *
Enter your username: christopherdavies553@gmail.com
Enter your password:
Uploading distributions to https://api.packagr.app/63cdQSDO/
Uploading my_package-0.1.0-py3-none-any.whl
100%████████████████████████| 3.85k/3.85k [00:00<00:00, 8.83kB/s]
Uploading my_package-0.1.0.tar.gz
100%|███████████████████████| 3.38k/3.38k [00:00<00:00, 3.72kB/s]
If you refresh the Packagr app, you should also see that the package now appears in there:
Step 5 — Creating a repository in Gitlab
We’ve now verified that we are able to package our code and upload it to Packagr using Twine. Let’s now look to automate that process, so that every time you push code to your repository, your code is updated, repackaged and reupload to Packagr. Let’s start by creating a repository on Gitlab — create a Gitlab account, if you don’t already have one, then create a new project at https://gitlab.com/projects/new. I’ve called my Gitlab project my-project
and I’ve made it public for your reference: https://gitlab.com/pypri/my-project
Once you’ve created your project, run the following commands from a terminal window, from your my_repository
folder
git init
git remote add origin git@gitlab.com:pypri/my-project.git
Before we push anything, let’s add two more files:
my_repository/
build/
dist/
my_package/
__init__.py
... the rest of your code goes here
setup.py
.gitignore
.gitlab-ci.yml
.gitignore
is just a simple file that tells git not to commit/push certain files to our repository. We don’t want to commit any packages that we have build locally to version control, so let’s add the build
and dist
folders to our .gitignore
file:
dist
build
The second file .gitlab-ci.yml
is a file that defines the steps to be executed as part of the CI process. In this case, we are going to define one job, called package
, which will package our code and push it to Packagr. Let’s add the following content to this file:
stages:
- package
package:
stage: package
image: python:3.6-alpine
only:
- tags
script:
- pip install twine wheel
- python setup.py sdist bdist_wheel
- twine upload dist/*
The above is fairly self-explanatory — pushing to Gitlab will run the package
job, which builds our package and pushes it to Packagr. Due to the only: — tags
bit, this will only run on tagged commits.
Step 6 — defining the environmental variables
In the .gitlab-ci.yml
file, we haven’t provided any credentials, or the repository URL. However, by setting the following environmental variables, you can provide the necessary information to twine:
TWINE_REPOSITORY_URL
TWINE_USERNAME
TWINE_PASSWORD
You can set these variables by going to Settings > CI/CD > Environmental variables in Gitlab, and populating them as necessary:
Step 7 —managing versioning
If we were to push our changes to Gitlab right now, the CI process would break. This is because in our setup.py
we still have the version set to 0.1.0
. Packagr (like PyPI) doesn’t allow you to reupload the same package version multiple times, so pushing our code as it currently stands would cause twine to return a Conflict for URL error message. When we created the .gitlab-ci.yml
file, we set it to only run the package
command for tagged commits — we will use these tags as the version numbers. We therefore need to update our setup.py
file to set the version number based on the commit tags, like so:
from setuptools import setup
import os
setup(name='my_package',
version=os.environ['CI_COMMIT_TAG'],
...
)
Step 8 — pushing to the repository
Everything should now be ready to go — use the following git commands to push the code to your repository:
git add .
git commit -m "Initial commit"
git tag 0.2.0
git push --tags
If all goes well, your Gitlab CI pipeline should pass correctly, and you should see the new versions show up in Packagr:
Did you find this tutorial helpful? If so, remember to follow the Packagr publication for future tutorials. Alternatively, feel free to reach out at info@packagr.app, or on twitter, @chrisbuildsapps