How to Manage Dependencies and Build & Publish Python Packages using Poetry

This tutorial covers

1. Poetry Installation
2. Setup project with poetry
3. A brief introduction to pyproject.toml
4. Creating a virtual environment with poetry
5. Dependency management using poetry
6. Create a Python package
7. Build a package with Poetry
8. Publish package with Poetry on PyPI

When building a Python package, developers usually rely on external libraries or dependencies not part of the Python standard library. To manually manage the installation, uninstallation, or updation of these third-party libraries is a tedious task. It is not uncommon for these third-party libraries to have additional external dependencies, making manually managing dependencies even more complicated.

Fortunately, there are dependency management tools for Python that make the management of third-party libraries easier. One such notable tool is Poetry, which not only helps in dependency management but also takes care of creating a virtual environment and building & publishing Python packages.

In this tutorial, we will use Poetry to create, build, and publish a command line tool that receives a country name as input and outputs its corresponding Alpha3 code.

1. Poetry Installation

To install the poetry, you can use the official poetry installer scripts.

# Linux, macOS, Windows (WSL)
curl -sSL https://install.python-poetry.org | python3 -


# Windows (Powershell)
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | py -
# In the above command
# Replace 'py' with 'python', in case you have installed python from Microsoft Store.

To verify the installation

poetry --help

2. Setup project with poetry

To begin, we will start by creating a new project using Poetry.

poetry new alpha3

This will create the following project structure for us.

.
└── alpha3  # project's root directory
    ├── alpha3  # python package
    │   └── __init__.py
    ├── pyproject.toml  # poetry configuration file
    ├── README.md
    └── tests
        └── __init__.py

Poetry, by default, creates with the same name both, the project’s root directory and the Python package directory.

3. A brief introduction to pyproject.toml

While the structure of our newly created project looks just like a regular Python project, there’s an exception — we have a configuration file named pyproject.toml that holds various details about the project's dependencies and other configurations and metadata.

# pyproject.toml

[tool.poetry]
name = "alpha3"
version = "0.1.0"
description = ""
authors = ["Sahil Rohilla <imsahilrl@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Above, we have [tool.poetry] section which contains the general Metadata about the project. This section contains multiple other sections, let us discuss some of them briefly.

[tool.poetry.dependencies]

• Poetry allows us to arrange dependencies in groups.
• The tool.poetry.dependencies is the main group where the dependencies are added by default. All the project’s runtime dependencies must be present in this group.

The below command will add the dependency to the main group.

poetry add dependency-name

[tool.poetry.group.dev.dependencies]

• This section stores all the dependencies which are required only for development or testing purposes.
• Using the --group flag we can add a dependency to the dev group or any other group by just passing the group name.

poetry add dependency-name --group=dev

[tool.poetry.scripts]

In this section, the package entry points are defined to be installed as executables, ensuring their availability on the system or virtual environment PATH.

This is useful when you are building CLI tools.

[tool.poetry.scripts]
cli_tool_name = 'my_package.__main__:main'

[[tool.poetry.source]]

By default, Poetry is configured to look for dependencies and to publish packages on the PyPI repository.

In this section, we can define additional private repositories.

[[tool.poetry.source]]
name = "testpypi"
url = "https://test.pypi.org/legacy/"
priority = "primary"

To add a private repository through the Poetry CLI tool:

poetry source add testpypi https://test.pypi.org/legacy/  # Add repository
poetry config pypi-token.testpypi your_token  # Add repository token to Poetry.

[build-system]

The poetry has its build backend, implemented in poetry-core, that is used for building the distribution packages: wheel and tar.gz (source distribution).

The poetry-core is PEP 517 compliant, which means, it allows tools like pip and build, to easily use poetry-core as the build backend, for building the packages.

The pip and build are known as build frontend tools when speaking in terms of building packages. These tools use a build backend under the hood to build the package.

Apart from the widely used legacy build backends like the distutils and setuptools, there are many other build backends available today, which build frontend tools can choose from. The build frontend tools use whatever build backend that is defined in the [build-system] section, to build the package. This section is not specific to Poetry and is not part of the [tool.poetry] section.

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

4. Creating a virtual environment with poetry

A virtual environment gives you a private space to install and manage your project’s tools and dependencies without affecting your entire computer. This isolation helps prevent conflicts and keeps things tidy.

It just takes one command to activate the virtual environment with the Poetry. If the project’s virtual environment is not already present, the poetry will automatically create one for you, when you first run the following command.

poetry shell

5. Dependency management using poetry

Now that you have activated your virtual environment, let us add some third-party libraries that we will be use in our Python project.

This tutorial requires the following third-party libraries:

• requests==2.31.0 Python HTTP for Humans.
• structlog==23.3.0 Structured Logging for Python

The requests library is used to make HTTP requests in Python. The structlog is used to log output to the command line.

To install the dependency, we can use the poetry add package-name command.

poetry add requests==2.31.0
poetry add structlog==23.3.0


# If you prefer to install the latest version, use
poetry add requests@latest structlog@latest

The [tool.poetry.dependencies] section will reflect the newly added packages.

[tool.poetry.dependencies]
python = "^3.10"
requests = "2.31.0"
structlog = "23.3.0"

When you run the poetry add command for adding a dependency, the poetry does a few things:

• It makes a new entry or updates an existing one for the package with the specified version inside the pyproject.toml.
• It installs the package to the project’s virtual environment. If the package is already present, it will update it or downgrade it according to the version specified.
• It creates or updates the poetry.lock file.

The poetry.lock file is automatically generated by the poetry, and it contains details about both, the direct and transitive dependencies which are already resolved and pinned to a specific version. This ensures that everyone in a team is using the same versions of dependencies, which makes debugging easier.

Furthermore, when other developers while setting up the project install the dependencies using the poetry install command. This will result in faster installation since the poetry would not have to resolve the dependencies again, it will rather use the poetry.lock file which already contains the dependencies in a resolved manner.

6. Create a python package

In this section, we will create a simple Python package to build a command line tool to convert a country name to their corresponding alpha3 code. For instance, the alpha3 code for "United States" will be "USA". For this purpose, we will use opendatasoft API which is publicly available.

We will begin by creating the following new files inside our alpha3 package.

• __main__.py
• get_alpha3.py

.
└── alpha3
    ├── alpha3
    │   ├── get_alpha3.py
    │   ├── __init__.py
    │   └── __main__.py
    ├── poetry.lock
    ├── pyproject.toml
    ├── README.md
    └── tests
        └── __init__.py

Now, let us add the following code to the get_alpha3.py file.

# get_alpha3.py

import structlog
import requests


log = structlog.get_logger()


def get_alpha3_code(country_name):
    """
    To get alpha3code from OpenDataSoft API.
    :param country_name:
    :return: Alpha3code or country name in case alpha3code couldn't be retrieved.
    """
    url = (f"https://public.opendatasoft.com/"
           f"api/explore/v2.1/catalog/datasets/countries-codes/records?"
           f"select=iso3_code&where=label_en='{country_name}'&limit=1")
    try:
        response = requests.get(url)
        response.raise_for_status()
        country = response.json()['results'][0]['iso3_code']
        return country
    except IndexError:
        log.warning(f'Could not find Alpha3code for country {country_name}.')
    except Exception as e:
        log.error(f"Unable to get alpha3 code for {country_name} due to the following error: {e}")

    return None

In the above code:

• we created a function called get_alpha3_code that takes a country name as a parameter.
• Using requests library we make an API call to opendatasoft API. The country name is appended to the URL as a query parameter.
• From the JSON response, we try to access the iso3_code key. If the key is present we return its value which is our alpha3 code.
• In case, the key is missing, we will get Index Error and we will simply log it using the structlog library.
• In case, we have any other type of error, we will catch that in the second except statement and log that as well.

Let us now populate the __main__.py file with the following content.

# __main__.py

import argparse

from .get_alpha3 import get_alpha3_code


def main():
    parser = argparse.ArgumentParser(description='The tools helps to find Alpha3 code for a input country name')

    # Add positional argument to the parser.
    parser.add_argument("country", type=str, help="Name of country")

    # Parse the command-line arguments provided by the user.
    args = parser.parse_args()

    # Make API call 
    alpha3code = get_alpha3_code(args.country)
    
    if alpha3code:
        print(alpha3code)


if __name__ == '__main__':  # pragma: no cover
    main()

In the above code:

• We have imported the argparse, Python's built-in library which is used to develop the command-line tools.
• From the get_alpha3 module, we have imported the get_alpha3_code function.
• Inside our main function, we have created an instance of the ArgumentParser class from the argparse module. We have also provided a short description of the tool for the keyword argument, description.
• We then added a positional argument to the parser. This positional argument is the name of the country the User will provide to the tool.
• After parsing the user response, we make a function call to get_alpha3_code with user-provided country name.
• At last, we simply print the returned value of the function call, if it is not None.

Good work! We have successfully created our Python Package.

To test it, from the root directory execute the package as a module.

# Execute the package as python module.
python -m alpha3 'United States'  # output: USA

Define entry point in pyproject.toml

Before building the package, we need to define an entry point in our pyproject.toml, for our python package to be installed as an executable that is available on the PATH after the user installs it.

[tool.poetry.scripts]
alpha3 = "alpha3.__main__:main"

Add a license

Since we are going to publish this package to PyPI, it is recommended to add a License for it. While adding a license for publishing on PyPI is not strictly required, it is generally a good practice to include it in your project. This helps users know under which terms they are allowed to use, modify, or distribute the code.

Here are the sites where you can browse the licenses to choose from.

• The Software Package Data Exchange (SPDX
• Open Source Initiative (OSI)

For this tutorial, we can go ahead with the MIT License. Create a LICENSE file inside the root folder of your project and copy & paste the following license to it.

MIT License

Copyright (c) [year] [fullname]

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Now let us add the notation for this license to the [tool.poetry] section.

[tool.poetry]
name = "alpha3"
version = "0.1.0"
description = ""
authors = ["Sahil Rohilla <imsahilrl@gmail.com>"]
readme = "README.md"
license = "MIT"  # added license notation

You can find the recommended notation for the most common licenses here.

NOTE

Before moving on to the next stage, it is important to note that the PyPI requires every published package to have a unique name.

If you were following this tutorial step-by-step, most likely the PyPI will reject your package due to a naming conflict. This is because I have already published it with the name alpha3.

To overcome this, we will modify the pyproject.toml to have a distinct name for the published package from the actual one.

[tool.poetry]
name = "unique-name"  # Add a unique name here. This will be the name of your published package.
version = "0.1.0"
description = ""
authors = ["Sahil Rohilla <imsahilrl@gmail.com>"]
readme = "README.md"
license = "MIT"

packages = [{ include = "alpha3" }]  # Add the alpha3 package folder here.

Above, we have added a new field, packages, that specifies which directories should be included in the final distribution.

Since we want a different name for the published package, we must explicitly include the alpha3 directory in the packages field.

7. Build a package with Poetry

To build packages, Poetry uses its build backend implemented in poetry-core.

From your project’s root directory, execute the following command to build the package.

poetry build

Now, the Poetry would have created a dist folder inside your root directory that contains the following distribution packages: source distribution (tar.gz), wheel distribution (.whl)

We can also install this newly created wheel distribution package directly using pip. However, it would be inconvenient to share this wheel file with other developers to install, so let us go ahead and publish this package on the Python Package Index (PyPI) repository.

pip install your-distribution-package.whl

8. Publish Package with Poetry on PyPI

Since in Poetry, the PyPI is configured as the default repository to publish and download packages, we will just provide our authentication token for it.

Create a token on PyPI

• To create an API token on pypi.org, first, you will need to create an account on the platform.
• After creating an account, navigate to the Account settings. There you have to scroll down to the API tokens section and click on the Add API token button.

• Provide any token name, and select the scope as Entire account (all projects), which will be the only available scope value present, unless you have some package already published.
• After clicking on the Create token button, you will see your API token, copy it, and keep it safe.

Add token to Poetry

poetry config pypi-token.pypi your-token

After providing your token to Poetry, execute the following command from the root directory of your project, to publish the package.

poetry publish

# output
(alpha3-py3.10) user@user:~/Desktop/alpha3$ poetry publish
Publishing alpha3 (0.1.1) to PyPI
 - Uploading alpha3-0.1.1-py3-none-any.whl 100%
 - Uploading alpha3-0.1.1.tar.gz 100%

We have now successfully published our Python package to the pypi.org.

To verify, navigate to the Your projects section.

You can now install the package from any pip-configured system using the following command

pip install alpha3

Conclusion

In this tutorial, we have successfully learned how to create a Python package, how to build it, and publish it on PyPI using Poetry. Furthermore, we briefly discussed the various sections in the Poetry configuration file.

LINKS

Projects GitHub Repository: https://github.com/sahilrl/tutorial_poetry_alpha3

Published package on PYPI: https://pypi.org/project/alpha3/

Thank you for reading! Please feel free to ask any query you may have in the comments ))