Create, version, and manage your library using Poetry and Google Artifact Registry

Using Poetry and Google Cloud Platform (GCP) for creating, versioning, and managing libraries is crucial for modern software development. Poetry streamlines dependency management, versioning, and virtual environment setup, ensuring consistent and reliable project environments. It simplifies publishing to repositories and handles dependencies efficiently, preventing conflicts and compatibility issues. GCP complements this by providing scalable infrastructure, robust CI/CD integration, secure storage, and comprehensive monitoring and logging. Together, they enhance productivity, reliability, and security, allowing developers to focus on writing quality code. This synergy ensures that libraries are scalable, secure, and efficiently managed, leading to better software quality and faster delivery times.

Assuming you have created repository in github or create a new repository on the command line

echo "# try" >> README.md
git init
git add README.md
git commit -m "first commit"
git branch -M main
git remote add origin git@github.com:your_user_name/try.git
git push -u origin main

Step 1: Prepare Your Python Code with Poetry

Install Poetry (if it is not installed)

curl -sSL https://install.python-poetry.org | python3 -

Create a New Project

poetry new my_library
cd my_library

Add Dependencies

You can use poetry add to add dependencies. Assume I need pandas library for my code.

poetry add pandas
poetry add numpy

It will edit pyproject.toml file to include relevant information which is added.

[tool.poetry]
name = "my-library"
version = "0.1.0"
description = ""
authors = ["Heliya Hasani <heliyahasani@hotmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"
pandas = "^2.2.2"
numpy = "^2.0.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Example code is below. and it is created in example.py

import numpy as np
import pandas as pd

# NumPy operations
array = np.array([1, 2, 3, 4, 5])
print("Array:", array)

matrix = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print("Matrix:\n", matrix)

print("Sum of array elements:", np.sum(array))
print("Mean of array elements:", np.mean(array))
print("Standard deviation of array elements:", np.std(array))

print("Transpose of matrix:\n", np.transpose(matrix))
print("Matrix multiplication:\n", np.dot(matrix, matrix))
# Pandas operations
data = {
    "Name": ["Alice", "Bob", "Charlie"],
    "Age": [25, 30, 35],
    "City": ["New York", "Los Angeles", "Chicago"],
}
df = pd.DataFrame(data)
print("DataFrame:\n", df)

print("Names column:\n", df["Name"])
print("First row:\n", df.iloc[0])

print("Mean age:", df["Age"].mean())
print("DataFrame description:\n", df.describe())

print("People older than 30:\n", df[df["Age"] > 30])

The structure of the code base should look like this.

├── my_library
│ ├── example.py
│ └── __init__.py
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests
 └── __init__.pyThe structure of the repo should look like this.
├── my_library
│ ├── example.py
│ └── __init__.py
├── poetry.lock
├── pyproject.toml
├── README.md
└── tests
 └── __init__.py

Build Your Package

poetry build

Step 2: Version Control with Git

Initialize a Git Repository

git add .
git commit -m "Initial commit"

Create a Repository on GitHub (or any Git provider) and Push

git push -u origin master

Tag Your Versions

git tag v0.1.0
git push origin v0.1.0

Step 3: Enable Artifact Registry API

Google Artifact Registry is a more specialized service designed to store and manage software packages. It provides better integration with CI/CD pipelines and package managers.

gcloud services enable artifactregistry.googleapis.com

gcloud artifacts repositories create my-repo-library \
    --repository-format=python \
    --location=us-central1 \
    --description="My Python library repository"

Configure Poetry to Publish to Artifact Registry

Add the following to your pyproject.toml:

[[tool.poetry.source]]
name = "gcp"
url = "https://us-central1-python.pkg.dev/your-project-id/my-repo-library"

Configure Authentication

gcloud auth application-default login
gcloud auth application-default set-quota-project your-project-id

Publish Your Package

poetry publish --build -r gcp

Install Your Package

pip install --extra-index-url \
https://us-central1-python.pkg.dev/your-project-id/my-repo my-library

How to Update Your Library

Step 1: Update Your Library

1. Make Changes: Make the necessary updates to your library code.
2. Update pyproject.toml: Increment the version number in the [tool.poetry] section of your pyproject.toml file:

[tool.poetry]
name = "my_library"
version = "0.2.0"  # Update the version number
description = "A brief description of your library"
authors = ["Your Name <heliyahasani@hotmail.com>"]

Build the New Version: Build the new version of your library:

poetry build

Step 2: Publish the New Version to Google Artifact Registry

Ensure Configuration in pyproject.toml: Make sure your pyproject.toml includes the Artifact Registry URL(which we added this before so you can skip this):

[[tool.poetry.source]]
name = "gcp"
url = "https://us-central1-python.pkg.dev/your-project-id/my-repo"

Authenticate with Google Cloud: Ensure you are authenticated with Google Cloud:

gcloud auth application-default login
gcloud auth application-default set-quota-project your-project-id

Publish the New Version: Publish the new version to Artifact Registry:

poetry publish --build -r gcp

Step 3: Version Control with Git

Commit Your Changes:

git add . 
git commit -m "Update to version 0.2.0"

Tag the New Version:

git tag v0.2.0 
git push origin v0.2.0

Push Your Changes to the Remote Repository:

git push

Install the New Version

When you want to use the new version, install it directly from the Artifact Registry:

pip install --extra-index-url \
https://us-central1-python.pkg.dev/your-project-id/my-repo my-library==0.2.0

What if I want to use this library in another project ?

Step 1: Ensure Your Library is Published to Google Artifact Registry

Follow the previous steps to publish your library to Google Artifact Registry. Ensure that the library is correctly uploaded and accessible.

Step 2: Configure Authentication in the New Project

Authenticate with Google Cloud: Before you can install packages from Google Artifact Registry, you need to authenticate with Google Cloud in your new project. You can do this using the gcloud CLI.

gcloud auth application-default login
gcloud auth application-default set-quota-project your-project-id

Set Up Poetry Configuration:

1. In your new project, you need to tell Poetry where to find your private package repository (Google Artifact Registry).
2. Update your pyproject.toml to include the Artifact Registry source:

[tool.poetry]
name = "new_project"
version = "0.1.0"
description = "A new project that uses my_library, pandas, and numpy"
authors = ["Your Name <your.email@example.com>"]

[tool.poetry.dependencies]
python = "^3.8"
my-library = "^0.2.0"  # Adjust the version as needed
pandas = "^1.3.0"
numpy = "^1.21.0"

[[tool.poetry.source]]
name = "gcp"
url = "https://us-central1-python.pkg.dev/your-project-id/my-repo"

Step 3: Install Your Library

Now, you can install your library in the new project using Poetry.

Install Dependencies: Run the following command to install the dependencies, including your custom library from Artifact Registry:

poetry install

Poetry will fetch the my-library package from Google Artifact Registry and install it in your new project.Example Project Setup

Use Your Library: You can now import and use my_library in your code.

import my_library
# Example usage of your library
my_library.some_function()

Automating Authentication for CI/CD

If you want to automate this process in a CI/CD pipeline, you can configure your CI environment to authenticate with Google Cloud using service account credentials.

Here’s an example for GitHub Actions:

name: CI

on: [push]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: '3.8'
    - name: Install Poetry
      run: curl -sSL https://install.python-poetry.org | python3 -
    - name: Configure Poetry
      run: poetry config virtualenvs.create false
    - name: Install dependencies
      run: poetry install
    - name: Authenticate to Google Cloud
      uses: google-github-actions/auth@v0.4.0
      with:
        credentials_json: ${{ secrets.GCP_CREDENTIALS }}
    - name: Configure gcloud CLI
      run: gcloud config set project your-project-id
    - name: Install dependencies
      run: poetry install

• Add your Google Cloud credentials as a secret (GCP_CREDENTIALS) in your GitHub repository settings.

This setup will ensure your CI/CD pipeline can authenticate with Google Cloud and install your private library from Artifact Registry.

By following these steps, you can easily use your custom Python library in other projects and automate the installation process in different environments.