Create and Deploy Custom Python Libraries on GCP
CoAuthors Andrew Adcock & Isabella Bresciani
Deploying a custom library in Google Cloud Platform (GCP) is straightforward and can be incredibly useful in various scenarios. By the end of this guide, you’ll have the knowledge to create, package, and deploy one on your own.
What is a Custom Library?
We’re all familiar with importing external libraries that not only simplify our coding tasks but also make our lives easier:
import pandas as pd
import numpy as np
import money as $ # (we wish this one was real!)These libraries didn’t magically appear. Someone wrote the code, encapsulated the logic, and made it available for us to use. At their core, some libraries are simply Python code, packaged and distributed for reuse, while others build upon existing libraries to extend functionality.
Why Create a Custom Library?
Creating a custom library can be a powerful way to streamline your development process, enhance code quality, and address specific needs. While the goal isn’t to reinvent the wheel, there are several compelling reasons to consider building your own library:
- Avoid code repetition (DRY — Don’t Repeat Yourself): When writing code, it’s common to encounter repetitive patterns. Instead of duplicating logic, you can encapsulate it in a reusable library. This not only improves efficiency but also makes maintenance easier. Example: Imagine building a house with bricks. Instead of writing the code for each brick, you can create a “template” as a base.
- Code maintenance: from the example above, If you need to change the color or thickness of the bricks, you only modify the template, and the change is reflected everywhere.
- Database abstraction: Implement the DAO (Data Access Object) pattern to decouple database access logic.
- Standardizing responses: We could deploy a library with a Response Object to Standardize Cloud Function Output.
- Reusability: If your code is useful for future projects or other areas of the business, encapsulating it in a library is a great idea.
- No existing library meets your needs: While rare, it’s possible that no library exists for your specific requirements.
Steps to Create and Deploy a Python Library
Permissions and Access needed:
- You must be authenticated with GCP in your local terminal, for more information follow this step-by-step guide [ Link here ]
- To get started we must first enable the Artifact Registry API in Google Cloud Provider. [Enable Artifact Registry API here].
- You must have Artifact Registry Admin role.
- Grant <project-id>@cloudbuild.gserviceaccount.com service account Artifact Registry Reader permission. Cloud build service account is used to download dependencies, build and deploy the cloud function
1. Install pip dependencies
We will need the following pip libraries to be installed for packaging and uploading our library:
pip install wheel
pip install twine
pip install setuptools2. Setup our python module
Our library will need several files in addition to the code. Below is an example of the final directory structure for this project:
HelloWorld
├── build
├── dist
├── HelloWorldHandler
| ├── __init__.py
| ├── HelloWorldHandler.py
└── setup.py3. Create HelloWorld.py
Inside of the `HelloWorldHandler` directory create `HelloWorldHandler.py` and add the following code. This is a basic example illustrating how your OOP library will be setup.
class HelloWorldHandler:
def say_hello():
return 'Hello World'4. Create init.py
Inside of the `HelloWorldHandler` directory create `__init__.py` and add the following content. This allows for simpler import syntax later on.
from .HelloWorldHandler import HelloWorldHandler
__all__ = ['HelloWorldHandler']5. Create setup.py
Inside of the root (e.g `HelloWorld` ) directory create `setup.py` and add the following content.
from setuptools import setup
setup(
name="hello_world_handler",
version="0.1.0",
packages=["HelloWorldHandler"],
description="A Python package for saying hi.",
install_requires=[]
)- name: This is the unique name of your library.
- version: Used by Artifact Registry for maintaining versioning.
- packages: This is the name of your package and will be referenced in your import (e.g. `from HelloWorldHandler `).
- description: An internal-only description for your library.
- install_requires: If your library has any imports add them here as a list. (Example: “pandas”)
6. Create Artifact Registry
We will use the command line to create an Artifact Registry Repository. You can optionally verify your active project with the command:
“gcloud config get-value project”. To switch to a different project, use:
“gcloud config set project <YOUR_PROJECT_ID> “ (Modify as necessary.)
Then, run the following command to create the Repository:
gcloud artifacts repositories create [REPOSITORY_NAME] \
--repository-format=[FORMAT] \
--location=[LOCATION] \
--description=[DESCRIPTION] - [REPOSITORY_NAME]: This is the reposityory name
- [FORMAT]: This is the type of repository
- [LOCATION]: Google location, in this case it’s us-east1. This will be import in future steps so if you modify please keep that in mind
- [DESCRIPTION]: Internal description of the Artifact Registry Repository
Example:
gcloud artifacts repositories create hello-world-handler \
--repository-format=python \
--location=us-east1 \
--description="Hello World Handler"- gcloud artifacts repositories create hello-world-handler: This creates an Artifact Registry Repository named `hello-world-handler`
- — repository-format=python: Indicates that this is a python library.
- — location=us-east1: Google location of us-east1. This will be import in future steps so if you modify please keep that in mind.
- — description=”Hello World Handler”: Internal description of the Artifact Registry Repository
7. Package the Hello World library
In `HelloWorld` (the root directory) run the following command to create a distribution/wheel file:
python setup.py bdist_wheelThis will generate `dist/` and `/build` directories.
8. Push our library to Artifact Registry
python3 -m twine upload
--repository-url https://[LOCATION]-python.pkg.dev/[PROJECT]/[REPOSITORY_NAME]
dist/[DISTRIBUTION] - verbose- [LOCATION]: In step 4, we declared us-east1, if you modified this use the same location here.
- [PROJECT]: This is your google project ID.
- [REPOSITORY_NAME]: The repository name (we created this in step 4
- [DISTRIBUTION]: The wheel file we just created
Example:
python3 -m twine upload
--repository-url https://us-east1-python.pkg.dev/my-project/hello-world-handler
dist/hello_world_handler-0.1.0-py3-none-any.whl
--verbose9. Create a cloud run function
In Google Console, create a new Cloud Run function and set the Runtime to Python 3.12. Use the same region as in Step 4 (us-east1)
10. Import Library
In requirements.txt we will now add our library reference from our newly created Artifact Registry Repository
Add the following to requirements.txt
--extra-index-url https://[LOCATION]-python.pkg.dev/[PROJECT]/[REPOSITORY_NAME]/simple/
[REPOSITORY_NAME]==0.1.0- [LOCATION]: In step 4 we declared the location as `us-east1`, if you modified this use that same location here.
- [PROJECT]: This is your google project ID.
- [REPOSITORY_NAME]: The repository name (we created this in step 4)
Example:
--extra-index-url https://us-east1-python.pkg.dev/my-project/hello-world-handler/simple/
hello-world-handler==0.1.011. Add contents of main.py
Replace the contents of main.py with the following:
from HelloWorldHandler import HelloWorldHandler
def hello_http(request):
return HelloWorldHandler.say_hello()See it in action
In google console select `Deploy`. When finished the Build and Service tasks complete navigate to the `TESTING` tab in your Cloud Run Function. Scroll down to `CLI test command` and you’ll see a button labeled `RUN IN CLOUD SHELL`. This may take a few minutes to load the first time.
Once loaded, a curl request will be generated. If it does not initiate automatically, you may have to hit enter/return to run the command. You should see the following as proof that your new library is being imported and is functioning correctly.
Subsequent Releases
For subsequent releases update the version parameter in the setup.py file.
from setuptools import setup
setup(
name="hello_world_handler",
version="0.1.1", # increment using semantic versioning
packages=["HelloWorldHandler"],
description="A Python package for saying hi.",
install_requires=[]
)Then, start from step 7 updating the version as needed through out the guide.
— Ref —