Creating and sharing private Python packages

How django-carrot uses Packagr to store and distribute development build

The problem

As part of the development process on django-carrot, we regularly need to deploy prerelease builds to development machines for testing before they are released publicly. Previously, this meant adding alpha tags to builds and pushing them to PyPI. However, this inevitably means that at the end of each release, there is a whole bunch of alpha builds that needs to be deleted, so as to keep the package index clean and to prevent people accidentally installing buggy prerelease builds:

The solution

In order to keep things a bit cleaner, I decided I needed a private package index to keep development builds separate from build builds. There are a number of options for this — you could use a self-hosted open source solution such as pypicloud or pypiserver, or a paid cloud solution such as cloudsmithor packagecloud. However, I opted to use Package. Packagr offers a free tier account which lets you create an unlimited number of public Python packages for free. If security is a concern, you can create unlimited private repositories from just £5 per month, which significantly cheaper than either cloudsmith or packagecloud.

The process

First of all, create a python package, if you don’t have one already. Use the following structure to create a simple hello world app

hello/
helloworld/
__init__.py
setup.py

Paste the following code into __init__.py

def hello():
return "Hello, World!"

And in your setup.py:

from setuptools import setup
setup(name='helloworld',
version='0.1',
description='Test',
author='Me',
author_email='me@example.com',
license='MIT',
packages=['helloworld'],
zip_safe=False)

To create the package, run the following command in a terminal:

python setup.py sdist

This will create a folder called dist containing a package called helloworld-0.1.tar.gz:

hello/
dist/
helloworld-0.1.tar.gz
helloworld/
__init__.py
setup.py

This is the package that we are going to push to our private server.


Next, you need to create an account at Packagr — select the free tier if you’re OK with only public packages, or Standard/Advanced for private repositories

Packagr tier options

Complete the simple registration process and login. This will create your private cloud. When you first log in, you will be taken to your package list, which is empty for now:

Click on the Create a new package tab to see the details of your private cloud. This will show you the commands you need to use to upload your package

We now have somewhere to upload our package to. CD into the dist folder that we created earlier, and copy and paste the above commands into your terminal. You will be prompted for credentials — enter the email address and password you used to setup your Packagr account

$ cd dist
$ pip install twine
...
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 pkginfo-1.4.2 requests-2.18.4 requests-toolbelt-0.8.0 tqdm-4.23.4 twine-1.11.0 urllib3-1.22
$ twine upload --repository-url https://api.packagr.app/63cdQSDO/ *
...
Uploading helloworld-0.1.tar.gz
100%|█████████████████████████| 3.34k/3.34k [00:00<00:00, 8.23kB/s]

If all works correctly, you should be able to refresh your page and see the package you just pushed:

Clicking on the package shows the installation command for it:

Running this command in a terminal installs your package. If you chose either of the paid tiers, then anything you upload will be marked as private by default, and you will be prompted to enter your password. If you’re using the free tier, all packages are uploaded as public, meaning they can be installed by anybody who has your index url

$ pip install helloworld -i https://api.packagr.app/63cdQSDO/
...
Successfully installed helloworld-0.1

Notes

  • If using the paid tiers, all packages uploaded to Packagr will be marked as private by default. You can either mark them as public to make them available to anybody, or share your package with other specific people using Access tokens. These are explained in the docs
  • If your package has additional dependencies from PyPI, you should include your repository URL using the extra-index-url argument, instead of the -i switch. Doing this will treat your private cloud repository as an additional index, instead of the only index
  • You can create a pip.conf file to save having to enter credentials for every command, as shown in the below example. (note that the username:password string is only required for private repositories)
[global]
extra-index-url = https://myusername:mypassword@api.packagr.app/my-cloud-id/