Building and publishing Python packages in your Gitlab CI pipeline

In this tutorial, we’ll look at how to package Python code and push to Packagr using Gitlab CI

In the last tutorial in this publication, we looked at how to create a private package repository (your very own private PyPI.org) and push code to it using twine. This time, we’ll look at how to build this into your CI processes — in this case, we are going to use Gitlab, and it’s excellent built-in continuous delivery/integration tools.

Step 1 — write some code

Presumably, if you’re reading this tutorial, you probably already have some code that you’d like to publish to Packagr. Let’s start by sorting the folder structure properly — create an empty folder, my_repository and add a subfolder to it, my_package, into which you’ll put your code:

my_repository/
    my_package/
        __init__.py
        ... the rest of your code goes here

Step 2 — add some structure

Next, let’s create a Python file called setup.py in the my_repository folder:

from setuptools import setup
setup(name='my_package',
      version='0.1.0',
      description='Some Python package',
      author='Me',
      author_email='me@example.com',
      license='MIT',
      packages=['my_package'],
      zip_safe=False)

Step 3 — Create the package

We now have the basic folder structure we need to create a Python package. To do this, open a terminal window from your my_repository folder and enter the following commands:

pip install wheel
python setup.py sdist bdist_wheel

The first of the two commands above installs wheel, which allows us to create wheel packages by using the bdist_wheelargument. If all goes well, we should see that our folder structure now looks something like this:

my_repository/
    build/
    dist/
        my_package-0.1.0.tar.gz
        my_package-0.1.0-py3-none-any.whl
    my_package/
        __init__.py
        ... the rest of your code goes here
    setup.py

Step 4 — test the upload to Packagr

Before going any further, let’s check that we can upload this package to Packagr.

Firstly, you will need to create a Packagr account, if you don’t already have one — Just go to Packagr and follow the simple steps to create an account (a free one is fine for this example). Once you’ve created and verified your account, login and click on create new package. This will display the commands you need to upload packages to your repository:

Let’s try this out now. Open a terminal window and cd into your dist folder, where your newly created packages are now stored, and enter the commands. If all goes well, the output should look something like this:

$ twine upload  --repository-url https://api.packagr.app/63cdQSDO/ *
Enter your username: christopherdavies553@gmail.com
Enter your password: 
Uploading distributions to https://api.packagr.app/63cdQSDO/
Uploading my_package-0.1.0-py3-none-any.whl
100%████████████████████████| 3.85k/3.85k [00:00<00:00, 8.83kB/s]
Uploading my_package-0.1.0.tar.gz
100%|███████████████████████| 3.38k/3.38k [00:00<00:00, 3.72kB/s]

If you refresh the Packagr app, you should also see that the package now appears in there:

Step 5 — Creating a repository in Gitlab

We’ve now verified that we are able to package our code and upload it to Packagr using Twine. Let’s now look to automate that process, so that every time you push code to your repository, your code is updated, repackaged and reupload to Packagr. Let’s start by creating a repository on Gitlab — create a Gitlab account, if you don’t already have one, then create a new project at https://gitlab.com/projects/new. I’ve called my Gitlab project my-projectand I’ve made it public for your reference: https://gitlab.com/pypri/my-project

Once you’ve created your project, run the following commands from a terminal window, from your my_repository folder

git init
git remote add origin git@gitlab.com:pypri/my-project.git

Before we push anything, let’s add two more files:

my_repository/
    build/
    dist/
    my_package/
        __init__.py
        ... the rest of your code goes here
    setup.py
    .gitignore
    .gitlab-ci.yml

.gitignore is just a simple file that tells git not to commit/push certain files to our repository. We don’t want to commit any packages that we have build locally to version control, so let’s add the build and dist folders to our .gitignore file:

dist
build

The second file .gitlab-ci.yml is a file that defines the steps to be executed as part of the CI process. In this case, we are going to define one job, called package, which will package our code and push it to Packagr. Let’s add the following content to this file:

stages:
  - package

package:
  stage: package
  image: python:3.6-alpine
  only:
    - tags
  script:
    - pip install twine wheel 
    - python setup.py sdist bdist_wheel
    - twine upload dist/*

The above is fairly self-explanatory — pushing to Gitlab will run the package job, which builds our package and pushes it to Packagr. Due to the only: — tags bit, this will only run on tagged commits.

Step 6 — defining the environmental variables

In the .gitlab-ci.yml file, we haven’t provided any credentials, or the repository URL. However, by setting the following environmental variables, you can provide the necessary information to twine:

• TWINE_REPOSITORY_URL
• TWINE_USERNAME
• TWINE_PASSWORD

You can set these variables by going to Settings > CI/CD > Environmental variables in Gitlab, and populating them as necessary:

Step 7 —managing versioning

If we were to push our changes to Gitlab right now, the CI process would break. This is because in our setup.py we still have the version set to 0.1.0. Packagr (like PyPI) doesn’t allow you to reupload the same package version multiple times, so pushing our code as it currently stands would cause twine to return a Conflict for URL error message. When we created the .gitlab-ci.yml file, we set it to only run the package command for tagged commits — we will use these tags as the version numbers. We therefore need to update our setup.py file to set the version number based on the commit tags, like so:

from setuptools import setup
import os

setup(name='my_package',
      version=os.environ['CI_COMMIT_TAG'],
      ...
      )

Step 8 — pushing to the repository

Everything should now be ready to go — use the following git commands to push the code to your repository:

git add .
git commit -m "Initial commit"
git tag 0.2.0
git push --tags

If all goes well, your Gitlab CI pipeline should pass correctly, and you should see the new versions show up in Packagr:

Did you find this tutorial helpful? If so, remember to follow the Packagr publication for future tutorials. Alternatively, feel free to reach out at info@packagr.app, or on twitter, @chrisbuildsapps