Excited about your python project? Do you want to share it with a wider audience? Packaging and publishing your package is a great way to get other people interested in your project. Here is a simple step by step guide to build and publish your first python package.
Note: We will assume that you already have a project in GitHub that you would like to package and publish.
Step 0: Have your project licensed
Before doing anything else, your project should have a license since it’s going to be open source. Depending on how you’d like your package to be used, many licenses are available. Some usual licenses for open source projects are MIT , BSD or GPL.
Step 1: make your code publish-ready
There are a few preliminary things you need to do to have your project package-able:
- Your project structure needs to be in place. Usually, the root of the repository contains a folder with the name of the repository — this is where the core project code should be. What’s outside of this folder is the extra code that’s necessary to run and build the package (tests, documentation, etc.)
- The core folder should include one (or more) module(s) and an
__init__.pyfile which imports the classes/functions you’ll want the end user to have access to. This file can also include the version of the package, to make it easily accessible to the end user.
- Ideally, a proper logging system (instead of
prints) should be set by using the
- When relevant, your core code should be organized in one or more classes.
Step 2: Create a setup.py with setuptools
Once your project has a set structure, you should then add a setup.py file at the root of the repository. This mostly helps to automate all the publishing and version maintaining processes. Here’s an example of what a setup.py should look like (source here).
A few notes:
- If your package has minimum dependencies, a clean way to handle these is to add them in the setup file, through the
install_requiresargument. If you’d like to share all transitive dependencies, you can also traditionnally point to them in a requirement.txt (however, only minimum dependencies have to be in setup.py)
- If you want metadata (from the repository) to be downloaded when anyone installs the package, you should add these through the
- More information about the
setup()function can be found here
Note: steps 3 to 6 are optional (but highly recommended), however you can jump straight to step 7 if you want to publish your package right now.
Step 3: Set local tests and test coverage checks
If it is not already the case, at this point, your project should definitely have unit tests. Although there are many frameworks that can help you do that, one simple way of doing it is using pytest. All tests should be in a dedicated folder (named
testing/ for example). In that folder, put all the test files you need in order to cover as much of your core code as possible. Here’s an example of how to write a unit test. Here’s also one of our Scitime’s test files.
Once in place, you can run your tests locally by running
python -m pytest from the repository root.
Once your tests are created, you should also be able to estimate the coverage. This is important as you want to maximize the amount of code that’s tested in your project (to reduce the amount of unexpected bugs).
A lot of frameworks are available for this as well, for Scitime we used codecov. You can decide on a threshold of minimum coverage allowed by creating a .codecov.yml file and also decide on what files to include in the coverage analysis by creating a .coveragerc file.
Step 4: Standardize syntax and code style
Step 5: Create a proper documentation
Now that your project has tests and a good to structure, it should be a good time to add a proper documentation. First thing is to have a nice readme file that will show up on your Github repo root. Once this is done, here are a few additional “nice to have”:
- Pull Request & issue templates: when a new PR or issue is created, these files permit you to template the descriptions as you wish. Follow these steps for PRs and these steps for issues to create them. Here are Scitime’s PR template and issue template.
- A contribution guide. This should be a short guide on how you want external users to contribute to your package. Here’s Scitime’s contribution guide.
- A code of conduct (here’s Scitime’s).
- Tags and a description (see screenshot below).
- Badges in the readme file (here’s a nice article on how to set these up).
As the readme file should be quite synthesized, it’s also pretty standard to have a more in-depth documentation. You can use sphinx to do so, and then host the documentation on readthedocs (here’s Scitime’s example). The documentation related files are usually in a
docs/ folder (as done for Scitime). Here’s a nice sphinx & readthedocs tutorial.
Step 6: Set up continuous integration
At this point, your project is not far away from being ready to be published. However, it seems a little overwhelming to have to deal with updating the docs, running the tests and checking the style and coverage after each commit. Fortunately, Continuous Integration (CI) helps you with that. You can use webhooks to GitHub to automate all these things after each commit. Here’s the set of CI tools we used for Scitime:
- For running tests, we used travis-ci and appveyor (for tests on windows platforms). For travis-ci, besides setting up the webhooks on the repository, you also have to create a .travis.yml file in which you can not only run tests but also upload updated coverage outputs and check style and format. Same goes for appveyor by creating a appveyor.yml file
- Codecov and readthdocs also have dedicated webhooks
This should make the whole process of updating the repository a lot easier.
Step 7: Create your first release and publication
At this point, your soon-to-be package should look similar to this:
It’s now ready to be published! The first thing to do is to create your first release on GitHub — this is to keep track of the state of your project at a given point in time, a release should be created each time the version changes. You can follow this to create the release.
- For PyPI, you first need to create an account, and then follow these steps using
twine. This should be fairly straightforward, and PyPI also provides a test environment that can be used right before the actual deploy. The latter basically consists of creating a source distribution (
python setup.py sdist) and uploading it with twine (
twine upload dist/*). Once done, there should be a PyPI page corresponding to your package (here’s Scitime’s) and anyone should be able to install your package running the
- For conda, we recommend publishing your package through conda-forge which is a community that helps you publish and maintain your package through their conda channel. You can follow these steps to add your package to the community, you will then be added to the conda-forge GitHub organization and be able to maintain your package quite easily (here’s Scitime’s conda-forge page) — anyone should then be able to install your package running the
You should now have your package finally published and available to anyone! Although most of the work is done, you’ll still have to maintain your project as you need to make some updates: this basically means changing the versions each time you make significant changes, creating new releases and going through step 7 again.