A Simple Guide for Python Packaging
Code reuse is a very common need. It saves you time for writing the same code multiple times, enables leveraging other smart people’s work to make new things happen. Even just for one project, it helps organize code in a modular way so you can maintain each part separately. When it comes to python, it means format your project so it can be easily packaged. This is a simple instruction on how to go from nothing to a package that you can proudly put it in your portfolio to be used by other people.
When you create a ‘.py’ file, that is a module. You can define classes, functions, variables in that module named as the filename. When you put one or more modules in a folder and add a ‘__init__.py’ file, you create a package named as the folder. So a common package folder structure could be like this:
Build a Package
Now that you have put your code in a package structure, let’s take a look at how to build a package. To do that, the most important thing is the setup configure file ‘setup.py’. You use that to specify metadata. Here is an example of the file:
from setuptools import setup, find_packages
description='The funniest joke in the world',
long_description='Really, the funniest around.',
'Development Status :: 3 - Alpha',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 2.7',
'Topic :: Text Processing :: Linguistic',
keywords='funniest joke comedy flying circus',
More details about setup function can be found here.
To make the package ready to work after installation, you want to tell the setup process what are the other packages it relies on so they can be automatically installed. Simple put that into ‘install_requires=[…]’ within setup function. You can optionally specify a version:
These dependencies are public packages which can be installed via pip. If you are relying on ones not on pypi, use this:
Add Additional Files
Often your package may need to include other types of files. To do that:
- Set ‘include_package_data=True’
- Create a MANIFEST.in file in the folder of setup.py
- Add file path to MANIFEST as ‘include /path/to/test.json’, the path is relative. You can also use search pattern like ‘*.json’.
Now you have everything in place, let’s build the package. Building a package means creating a source code distribution (e.g. tar.gz) or a binary file (e.g. wheel). It’s simple to do.
python setup.py sdist bdist_wheel
This will create both format in ‘./dist’ folder. The files will be named based on the package name and version.
Distribute a Package
Congratulations! You now have a shiny package. Now it’s the time to make it available to other people or your future self. We are gonna upload it to Pypi so you can do your favorite ‘pip install’.
First, you need to create an account on pypi site.
Then, simply run:
twine upload dist/*
Here we assume the files under ‘dist/’ will all be uploaded to pypi. You can of course specify a particular file. Twine is a simple tool to help us interacting with pypi. When you upload for the first time, it will try to register the package on pypi.
Use a Package
At this point, you have published your own package on pypi. To use that, just run
pip install my_package to get it to your machine. Then use ‘import my_package’ in your code.
If you are developing the package locally and want to use it for test. Add the path to the package folder to $PYTHONPATH and it’s good to go. If you don’t want to touch the environment variable, just do
pip install -e . in the root package folder and it will be installed just like a published one. Plus, you don’t need to do anything every time you make changes to your code, it will keep up-to-date.
Here are two great sources to read if you want a bit more details on python packaging:
- official document about module: https://docs.python.org/2/tutorial/modules.html
- python packaging tutorial: https://python-packaging.readthedocs.io/en/latest/index.html