How to - build and distribute a CLI Tool with Python

In my previous article I wrote about WHY I built a random password generator CLI tool. In this article, I will lay out the details of how we can build a CLI tool from scratch using Python.

Requirements

Python installed on your system (Python3 preferrably because Python2 is so pre-2020s), a text editor of your choice and some enthusiasm.

Before we start, let us also setup a few other things we will need. Open up a terminal and install:

1. virtualenv - pip install virtualenv virtualenv is basically an isolated python env where we can test our tools/scripts etc while not installing them on our system globally.
2. wheel - pip install wheel wheel is a packaging mechanism for python tools/packages.
3. setuptools - pip install setuptools setuptools is a library that we will use to package our tool.
4. twine - pip install twine twine is used for distributing the packages to pypi/test.pypi.

Note: if you think you may have these installed already, then update them to the latest versions by providing the --upgrade flag. Always best to keep your tools updated.

Directory Structure & Initial Setup

Create a directory for the tool. Then create the following folder structure and files inside it

MyTool
    - app
        - __init__.py
        - __main__.py
        - application.py
    - my_tool.py
    - setup.py
    - requirements.txt
    - README.md
    - LICENSE
    - MANIFEST.in

Choose a LICENSE for distributing the package. I used https://choosealicense.com/ for picking a license.

Add the following line to MANIFEST.in to tell setuptools to add the LICENSE file along with the package being distributed
include LICENSE

Write a good README for the project. If you are not sure what to add, you could write it once you are done building your tool when you’ll have better clarity of what the tool’s functionalities are.

We will be using Click for building this CLI tool. Although we can build a simple CLI tool by just reading the arguments using sys.argv using a library such as click will enable us to add more functionality easily and extend our tool.

The dependencies that we need for our tool is specified in requirements.txt
Here we are using Click, so add the following line to the file
click>=7.1.2

Are we done yet? Can we start coding please 😐 ?

No, we still have one more thing to do before we can get into the core of the application. Setting up setup.py . This file is the one that tells setuptools how to package the tool. It took me some research and a bit of trial & error to figure out what goes where to get the tool working fine.

from setuptools import setup, find_packages
with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()
with open("requirements.txt", "r", encoding="utf-8") as fh:
    requirements = fh.read()
setup(
    name = 'mytool',
    version = '0.0.1',
    author = 'John Doe',
    author_email = 'john.doe@foo.com',
    license = '<the license you chose>',
    description = '<short description for the tool>',
    long_description = long_description,
    long_description_content_type = "text/markdown",
    url = '<github url where the tool code will remain>',
    py_modules = ['my_tool', 'app'],
    packages = find_packages(),
    install_requires = [requirements],
    python_requires='>=3.7',
    classifiers=[
        "Programming Language :: Python :: 3.8",
        "Operating System :: OS Independent",
    ],
    entry_points = '''
        [console_scripts]
        cooltool=my_tool:cli
    '''
)

Let us go through some of the confusing parts of this file.

py_modules — This is the place where we tell setuptools what modules it must be including while packaging the tool. In our case, we are asking it to include my_tool.py and the whole app folder. That is where the main functionalities of our tool will lie.

packages —This tells the places setuptools should look for in the current directory to find packages required for the tool. find_packages() without any arguments means look through the whole directory to find any packages required.

entry_points — This is where we tell how our tool can be invoked and what function must be called when it is invoked.
[console_scripts] tells the setuptools that this tool will be used as a CLI tool (like pip or npm).

cooltool=my_tool:cli this tells setuptools that whenever somebody types cooltool in the terminal, call the cli function inside my_tool.py . If we have another function called start and we want that to be invoked, we would write my_tool:start

The code

Okay, now that we have the whole setup done, we can get into what we really set out to do.

my_tool.py

This is the entry point for our app. This is where we need to add our Click integration by writing import click and setup the different commands we want to add in our tool.

We then create a Group object by using @click.group() decorator

@click.group()
def cli():
    pass

This does nothing other than tell click that we have a group of commands and we can add multiple subcommands to this.

@cli.command() decorator is used to tell click that the function below should be executed when it is called as a command.
We can setup the inputs we want for our command via @click.option().

For example if you want a command cooltool hello to output Hello World to the terminal

@cli.command()
def hello():
    click.echo("Hello World")

click.echo is used instead of print to ensure support for both Python2 and Python3 terminals and has a wider range of features than normal print().

Suppose we want to take the name as an input option to the command

@cli.command()
@click.option('-n', '--name', type=str, help='Name to greet', default='World')
def hello(name):
    click.echo(f'Hello {name}')

We are using Python3’s f-Strings to format the string which is faster and more readable and concise. If no option -n is provided, default value World will be used for name.

This was a simple example. If we need to do something a bit more complex ,we can write the code in the application.py file inside app folder and have the file imported into my_tool.py with from app import application and then call the relevant function we need.

Testing the tool

Okay, now that we have a working version ready, let us test it out.
Remember we installed something called virtualenv ? Time to put that to use.

Open up a terminal and navigate to the folder MyTool.
Create a virtual environment by typing in virtualenv <name for virtual env> ex: virtualenv venv

This creates a whole new environment unrelated to the python env on your system. This is really useful for testing purposes as we do not have to install unwanted dependencies on our system.

Activate the virtual env with the command source venv/bin/activate . You should be seeing venv in your terminal.

Let us install the tool in this virtual env now. The command for that is python setup.py develop . This would’ve installed the CLI tool to the virtual environment venv .
We can verify by typing which cooltool and it should output the location of cooltool within the venv folder.
We can test it out by typing cooltool hello -n Python and we should get the output Hello Python on the terminal.

Packaging and Distributing

Now we have a cli tool ready and we want to package and distribute it so that our friends or anyone can use it.

For this, we will again use setuptools but this time instead of asking it to package it for development, we will ask it to package it for distribution.

python setup.py sdist bdist_wheel

This command will create a folder dist in our MyTool directory. You can check to see what binaries have been built using ls command. There should be a .whl, .egg and a .tar.gz file present.

We will upload this to https://test.pypi.org, the test environment for pypi, since our cli tool is just for demo purposes.

Navigate to https://test.pypi.org and create an account. Then create an API token so that you can upload the binaries for distribution. Make sure to copy the token somewhere locally.

We will use twine to upload the binaries to test.pypi.org.

twine upload --repository testpypi --skip-existing dist/*
--repository is important, and skip-existing will be useful when we want to distribute further versions of our tool.

Enter the username as __token__ and the copied complete token as the password.

Done!! We have successfully packaged and distributed a CLI tool using Python.

We can install it using pip install --index-url https://test.pypi.org/simple/ cooltool in our system. This installation would be global and we can run the command from anywhere now.

Next Steps

This was a demo tool we built. We can add more functionality by adding more commands to the tool using Click.

We can also distribute it to the main https://pypi.org repository. Follow the steps we did for distributing in test.pypi including creating an account and token.

Happy coding!

https://bongo.cat/

References

For more details and documentation refer the following links

• Click
• Python Packaging
• Virtualenv
• Setuptools
• Twine
• Wheel

You can also refer my code for the CLI tool I built here.