Step-by-Step Guide to Create Python Library using CI/CD Pipeline

Sometimes, you don’t want customers, teammates, and others to reach the source code of the Python project that you create. You need to ensure the sources are encapsulated. So, it is time to create Python packages in accordance with the needs of others that will be used. Besides encapsulation, you supply a reusable project and make others not write the same codes again. Moreover, modules and methods can be used in a practical way.

Simply put, the Python package is created by defining a hierarchical directory structure with necessary files for a Python application which covers modules and sub-packages.

In this article, I’ll show you how to create a Python package step by step with the help of GitLab CI/CD.

The Folder Structure

You will find the explanation of each file in a sample Python project folder you want to pack.

python-packaging-tutorial/
├── LICENSE
├── .gitignore
├── .gitlab-ci.yml
├── pyproject.toml
├── README.md
├── requirements.txt
├── setup.cfg
├── setup.py
├── src/
│   └── example_folder_1/      
│       └── example_folder_2/
│           └── __init__.py
│           └── example_folder_3/
│               └── __init__.py
│               └── example_folder_4/
│                   └── __init__.py
│                   └── example.py/
└── tests/
    └── __init__.py
    └── example_test.py/

We specify how we want to license our project with the content of the classic “LICENSE” file.

The “.gitignore” is a regular file defining the files that Git has to ignore.

On the other hand, the “.gitlab-ci.yml” configuration file is required to employ GitLab CI/CD. For example, in “.gitlab-ci.yml” one can define other configuration files, dependencies, caches, jobs, the location to deploy the application to, how to trigger the jobs, etc.

The “pyproject.toml” describes the build system requirements of Python projects.

Significant information about the project like user guide, maintaining, contributions, etc. are written in the “README.md” file.

We list the specific versions of all the dependencies in “requirements.txt” to be installed by pip.

Different sides of the Python project are configured and several packaging options are determined by “setup.py” and “setup.cfg”. On the “Packaging and distributing projects” page it is stated that “setup.cfg is an ini file that contains option defaults for setup.py commands.”

In the above structure, the number of “example_folder_x” folders can vary according to your application in the “src/” folder. Details of created methods and Python codes take place in “example.py”.

The __init__.py files are used to import modules from related folders, i.e. these files indicate the address of the modules we will add.

All test subjects related to the package are located in the “example_test.py” file in the “tests/” folder.

Configuration Files, Dependencies, and Other Files

In this part, we handle the contents of the files we mentioned above in general.

In “example.py” the methods are defined as we mentioned.

"""
example.py
"""
# Necessary dependencies
# import os
# from PIL import Image

def method_1():
    return "Method 1!"
def method_2():
    return "Method 2!"

Then test cases are described in “example_test.py”.

import unittest
from unittest import TestCase

from src.example_folder_1.example_folder_2.example_folder_3.example_folder_4 import example

class TestMethods(TestCase):
    def test_example(self):
        self.assertEqual(method_1, "Method 1!")
if __name__ == "__main__":
     unittest.main()

In the “requirements.txt” file we indicate the dependencies. For instance, we write the following:

numpy
flask

The sample “pyproject.toml” should look like:

[build-system]
requires = [
    "setuptools>=42",
    "wheel"
]
build-backend = "setuptools.build_meta"

Furthermore, setup configurations need to be set by “setup.py” and “setup.cfg”. You can set the version number you want in the “setup” function by using the “version” option manually for your package.

# setup.py
import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="python-packaging-tutorial",
    version="0.0.1",
    author="Deniz Kenan Kılıç",
    author_email="example@domain.com",
    description="Package deployment of python packaging tutorial",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://medium.com/@deniz.kenan.kilic/a-tutorial-on-creating-python-library-ci-cd-pipeline-8e66022108df",
    project_urls={
        "Bug Tracker": "https://.../python-packaging-tutorial/-/issues",
    },
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    package_dir={"": "src"},
    packages=setuptools.find_packages(where="src"),
    python_requires=">=3.6",

    test_suite='nose.collector',
    tests_require=['nose'],
)

In “setup.cfg”, the exe searches for tests in Python modules that are executable.

[nosetests]
exe=1

After defining all the configurations for setups, we fill the “.gitlab-ci.yml” file to gain the ability to use the CI/CD pipeline. Keyword references for the “.gitlab-ci.yml” are given here.

image: python:3.9.2

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip
    - venv/

before_script:
  - python -V
  - pip install virtualenv
  - virtualenv venv
  - source venv/bin/activate
  - pip install -r requirements.txt

stages:
  - test
  - build
  - deploy

test:
  stage: test
  script:
    - python setup.py test

build:
  stage: build
  artifacts:
    paths:
      - build
      - dist
  script:
    - pip install --upgrade build
    - python -m build

deploy:
  stage: deploy
  script:
    - pip install --upgrade build twine
    - python -m build
    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url https://gitlab.domain.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi dist/*
  allow_failure: false

Sample CI/CD Pipeline

Tips

• All the pipelines, jobs, and status of stages in the pipelines can be monitored in the GitLab sidebar.
• After deploying the project, the packaged project can be found under the “Package Registry” part.
• It is important to have explanatory, clear, and compact documentation. Automatic preparation for documentation is explained step-by-step in this post. (“Documentation hosted on the web, automatically” and “Hosting on ReadTheDocs” titles)
• It is possible to set the version number in the “setup.py” by using the “version” option manually to handle the version control of a project.

Please give me a clap and follow me if you enjoyed this story. 😊