Installing private Python packages in Docker images

In this tutorial, we’ll look at how to install a Python package stored in a private repo in a Docker image

Christopher Davies
Oct 11, 2019 · 6 min read

As of October 2019, Packagr.app now provides a private docker image registry to all users

Image for post
Image for post

Let’s say you have a set of common Python utilities that you use across a number of different projects.

For example, let’s say you have some scripts that generate random user data for the purpose of testing various applications, that you share across multiple projects

Copying these scripts across multiple code bases is a nightmare from the perspective of maintenance, but these utilities may contain proprietary code that cannot be put somewhere public. So where can you put them?

One option is to package your utilities into a Python package, store the package in a private Python repository, then have the Docker image that contains your projects install these utilities from your repository. That way, you can easily control versioning of these utilities across all projects that have a dependency on them.

In this tutorial, we’ll look out how to create a Python package, store it securely in a private repository (in this case, Packagr.app) and write a docker image that authenticates your private Python repository, then pulls and installs your package.

For good measure, we’ll also store our built Docker image in Packagr.app

For the purposes of this tutorial, we are going to build a simple tool that generates random user data, which we can use to populate APIs, perform tests, etc. Your utilities might do something totally different, but you can use the same approach to create a distributable package to do whatever you need, and store it privately.

Let’s start by creating our package. This tutorial assumes that you already have python3 and virtualenv installed. Let’s create a folder called data-generatorand a virtual environment:

# create a directory and cd into it
mkdir data-generator && cd data-generator
# create a virtual environment, and activate it
python3 -m venv env
source env/bin/activate

Next, let’s create a subdirectory called package that will store our code

mkdir package

Let’s install Faker, a Python environment that generates random data

pip install Faker

And let’s create a file called process.py in the package folder, which generates a random user every time it is called:

# process.py
from faker import Faker
import json
fake = Faker()

def generate_person():
print(
json.dumps(dict(
name=fake.name(),
email=fake.email(),
password=fake.password(),
address=fake.address()
))
)

The above function simply outputs a dummy person in json format, that looks like this:

{
"name": "Samuel Mendez",
"email": "michelle71@hotmail.com",
"password": "*hMRsQBbK1",
"address": "78287 Morgan Summit\nPhillipsstad, WV 38051"
}

Next, lets create a folder called bin in the root directory, and add a file called make-person. This is a simple script that will let you call your function from the shell

#!/usr/bin/env python
from package import process
process.generate_person()

Finally, lets create a file calledsetup.py in the rootdata-generator directory. This tells

from setuptools import setup

setup(
name='data-generator', # the name of the package
version='1.0',
packages=['package'], # contains our actual code
author='chris',
author_email='chris@packagr.app',
description='a random person generator',
scripts=['bin/make-person'], # the launcher script
install_requires=['faker==2.0.2'] # our external dependencies
)

Our folder structure should now look something like this:

/data-generator
/package
process.py
/bin
make-person
setup.py

We’re now ready to build our package! Enter the following lines into the command line:

# install wheel (to build packages in the bdist_wheel format)
pip install wheel
# create the package
python setup.py bdist_wheel

The process will run, and a few additional folders will be created. The one we’re most interested in is the dist folder, which contains our built package:

/dist
data_generator-1.0-py3-none-any.whl

You can check that our package works as expected by installing and running it as follows:

# install the local package
pip install dist/*
# call the script
make-person
{"name": "Alexandra Nelson", "email": "sgarcia@yahoo.com", "password": "*c706Hvc+H", "address": "625 Powers Orchard\nNorth Bonnietown, IN 04475"}

Now that we’ve built the package, we can upload it to our private repository. If you don’t already have a private repository to upload to, you can create one for free at Packagr.app by following the simple sign up process. Once you’ve logged into your account, click on Create new package in the left hand menu to see your private repository URL — make a note of this, as you’ll need it later:

Image for post
Image for post
Your private repo URL (note that free accounts get a public URL)

Now that you’ve got a private repository, and a package, we can upload files to it. We do this using a tool called twine

# install twine
pip install twine
# upload your built package to your repository (update the URL as necessary)
twine upload --repository-url https://api.packagr.app/63cdQSDO/ dist/*

You’ll be prompted to enter credentials — just use the username and password you signed up for Packagr with. Once you’ve done that successfully, you’ll see your new package in your Packagr dashboard

Image for post
Image for post
Your newly uploaded package

Now that our package is securely stored in our repository, let’s create a docker image capable of installing it. It’s important to remember that you should never store raw credentials in your Dockerfile, or anywhere else in your code for that matter. So, you should pass your credentials as environmental variables at build time. With that in mind, let’s start by creating a file called requirements.txt in an empty folder, remembering to change the repository URL to your own :

--extra-index-url https://api.packagr.app/63cdQSDO/
data-generator==1.0.0
faker==2.0.2

A requirements file is just a list of packages for pip to install. In this case, we are adding our private package, data-generator and the public dependency, faker. Adding the --extra-index-url tells pip to look in our private repo, in addition to the public pypi.org one.

Next, lets create our Dockerfile

FROM python:3.7-alpine
ARG USER
ARG PASS
RUN echo "machine api.packagr.app \
" login ${USER} \
" password ${PASS}" > /root/.netrc
RUN chown root ~/.netrc
RUN chmod 0600 ~/.netrc
COPY requirements.txt /requirements.txt
RUN pip install -r requirements.txt
CMD make-person

Let’s take a detailed look at this, one line at a time:

  • FROM python:3.7-alpine tells Docker the base image to use — in this case we are using the lightweight alpine python distro, mostly to save time/space
  • ARG USER and ARG PASS defines variables that we will provide at build time — specifically, our Packagr username and password
  • RUN echo... creates a file called .netrc which tells the image to use the Packagr username and password when connecting to api.packagr.app. The 2 following lines set the permissions of this file correctly
  • COPY requirements... copies the requirements file to the Docker image
  • RUN pip... installs the dependencies in our requirements file.
  • CMD make-person just calls the script defined in our setup.py at run time

Now that we have our Dockerfile, let’s build the Docker image, substituting in your Packagr username and password:

docker build -t dg-image --build-arg USER=chris@packagr.app --build-arg PASS=changeme .

If all goes well, your docker image should build correctly! We can now upload it to our docker registry. Go back to the Packagr interface, click on Docker registry, and make a note of your Docker registry URL:

Image for post
Image for post
Steps for uploading your Docker image

The first step is to login to docker — as usual, you’ll need to update the URL, username and password to your own

docker login docker.packagr.app/63cdqsdo -u me@example.com -p changeme

Next, you’ll need to tag the image you just build (again, change the URL)

docker tag dg-image docker.packagr.app/63cdqsdo/dg-image:latest

Finally, you can push your tags:

docker push docker.packagr.app/63cdqsdo/dg-image:latest

You should now see your image in your docker registry in Packagr:

Image for post
Image for post

You can pull this docker image from any other machine you are logged into with this command:

docker pull docker.packagr.app/63cdqsdo/dg-image:latest

Packagr

Tutorials for Packagr, your private Python package index

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store