Installing Packages from a Private Repo into your Docker Container

I have spent the last few hours trying to find out the most secure way to install a package from a private repository with python’s pip.

I had a few constraints before I started searching:

• This package had to be installed during build time using pip.
• I didn’t want to pass my gitlab’s username and password in the url/header so using HTTP was out of the question.
• I didn’t want to leave traces of my SSH private key in the container.
• I didn’t want to save my SSH private keys in a local, non-tracked environment file.

With these constraints defined, I eventually settled on this approach by Vladislav Supalov. His approach leverages the fact that Arguments defined in intermediate builds aren’t saved on Docker. This means your private SSH keys won’t be found when someone types docker history <name_of_image> on your images. Warning, note this relies on you using docker file’s ARG command. ENV commands, on the other hand are going to remain in the final container.

I decided that what had to happen in my intermediate build was to download all the packages that are saved on a private repository. To do this, however, I needed to make my SSH private key available during build time. I chose to expose this private key temporarily via the host’s environment variables using export SSH_PRIVATE_KEY. With this in place, it was time to tell Docker about it. Since I was using docker-compose to start up a number of services, I made my service aware of this variable with the code below:

...
web:
  build:
    context: ./web
    args:
      - SSH_PRIVATE_KEY=${SSH_PRIVATE_KEY}
  ...

Most of the work happened in the Docker file:

# this is our first build stage, it will not persist in the final image
# we will use this to install requirements that require ssh access touch
# private repos
FROM python:3.5 as intermediate

# create and change working directory
WORKDIR /code

# Add credentials on build
ARG SSH_PRIVATE_KEY
RUN mkdir /root/.ssh/
# remember to use a temporary variable for this
# This private key shouldn't be saved in env files
RUN echo "${SSH_PRIVATE_KEY}" >> /root/.ssh/id_rsa && chmod 600 /root/.ssh/id_rsa
# make sure your domain is accepted
RUN touch /root/.ssh/known_hosts
RUN ssh-keyscan gitlab.com >> /root/.ssh/known_hosts
COPY ./requirements_private_repos.txt /code/
RUN pip install -r requirements_private_repos.txt
# image that would be used
FROM python:3.5
# copy the python packages from the intermediate image
COPY --from=intermediate /usr/local/lib/python3.5/site-packages/ /usr/local/lib/python3.5/site-packages/
ENV PYTHONUNBUFFERED 1

# change working directory
WORKDIR /code
# install requirements
RUN apt-get update && apt-get install netcat -y
COPY ./requirements*.txt /code/
RUN pip install -r requirements.txt
# copy script that detects context: web, celery worker, celery beat
# and runs the right running command
COPY run_web.sh /code/
RUN chmod +xr run_web.sh
WORKDIR /code/my_project

First, I created an intermediate container. In that container, I created a directory for SSH keys and copied the private keys from the variable SSH_PRIVATE_KEY to a new file in the .ssh directory. Then, I updated my known_hosts file with the public key of the gitlab.com server. This will prevent being prompted to confirm if you want to connect to an unknown host.

At this point, we are ready to install packages from our private repositories. In my case, since I will be making reference to packages in a file, I had to copy that file first before I could start installing the packages.

Once done with installation, I moved to creating the final image. I copied all installed packages from the intermediate container into the final image. With that settled, I could set up whatever else I needed for this container.

Having written the necessary configs for both docker-compose and docker, I was ready to build my image with docker-compose build. This worked with no issues on a mac. However, when I tried running it on an ubuntu server, the SSH_PRIVATE_KEY wasn’t found and a blank string was used instead.

This was because I exported the variable on the ubuntu host without a sudo but ran docker with a sudo:

$ export SSH_PRIVATE_KEY
$ sudo docker-compose build

To fix this, I used:

$ sudo -E docker-compose build

The -E switch preserves user environment, thus making SSH_PRIVATE_KEY available to the docker-compose command.

When you are done, REMEMBER to unset the SSH_PRIVATE_KEY variable.

I will most likely write a script to handle the setting and unsetting of this variable since I can trust that I will always remember this last step.

Anyway, thats how I did it. Itching to hear of a better way to do this.

Thanks for reading.