Vendoring Python dependencies with pip
We have been slowly trying to move our Python development environments from vagrant over to docker. However, we have at least 1 service which relies on a private Python package. We usually install this package via pip install -e git+ssh://git@github.com:org/repo@<tag>, but when trying to do so in docker we get an error since we do not have a valid SSH key to allow pip to access the private GitHub repository.
We tried for the longest time to figure out the best solution. Can we somehow inject our local SSH key into the docker build context, without actually storing the SSH key into the image? Maybe instead we can just create a deploy key in GitHub that we use for projects to build images, that way we can always revoke the SSH key if need be. Neither of these solutions are really that great, and would require some extra tooling around the image creation process.
While reading through the documentation for pip I noticed there was a --download CLI option for pip install. This then led me to find that there is a pip download command which is used to download Python packages locally. With this command, you are able to download a zip of the Python package from GitHub and store it locally. You can also install the package from the zip file instead of via the URL.
$ pip download --no-deps --dest ./vendor -e git+ssh://git@github.com:org/repo@<tag>
$ pip install ./vendor/repo-<tag>.zipYou can even reference the zip file in your requirements.txt file when installing:
# Replace this line
-e git+ssh://git@github.com:org/repo@<tag># With this line
./vendor/repo-<tag>.zip
This solution works really well for our specific use case since we are perfectly fine committing our private dependencies in the GitHub repo as there are not many of them and they are not updated very often.
