How to package a python project with all of its dependencies for offline install.

I’m writing this post to show the way I think is the correct way for installing and packaging python apps with all their dependencies.

Often you need to have a package that you can use to install the app and it’s dependencies without reaching the internet at all, it could be a remote server with only access to internal network for security reasons, and it could be that one of your dependencies is hosted on a private git repository and you don’t want to hard code the git token for installing it over https, for example, a docker container.

My approach is combing SetupTools, DistUtils, and pip Wheel for a final tar.gz format.

I assume you already familiar with the basics of packaging:
https://python-packaging.readthedocs.io/en/latest/minimal.html

basic python packaging requires the minimum of a setup.py file in the root of your project which then you can call on python setup.py sdist

setup.py gist

The cmdclass is an extension to setuptools and is the class that will package the code for distribution but also collects all packages from requirements.txt into the archive.
This script below (package.py) is a setuptools command extension for creating a distribution from a project.
normal python setup sdist will only pack files and folders, while this script go over the
requirements.txt will make a wheel
(zip like archive) of the requirement to be stored at wheelhouse folder and packed together with
the code as a single tar.gz file.

Packaging Caveat:
This script collects the installed (compiled) requirements from where it runs, this means that if your requirements.txt contains any requirements that are based on C-extensions such as psycopg for example, the compiled package will be collected.
When you unpack the archive at the destination if the system is different than the system you collected the package from it will not work as intended and even crash.

let me re-iterate this:
ubuntu != centos != macOS
python2.6 != python2.7

You can however, package on ubuntu when the target system is Debian since they are the same family.

package.py gist

This script can only run from the same family due to C-extensions that needs to be compiled on specific
platform (i.e psycopg etc…).

What this script does:

This script is deleting and recreating wheelhouse folder in the root of your project dir.

It then calls pip wheel -r requirements.txt which collects installed packages from the current environment they are installed into .whl files in the wheelhouse, then the script creates a local requirements.txt file without any http links in it, just the local package name as its stored in the wheelhouse, the reason for this is the way that pip works.

After the package is unpacked at the target destination, the requirements can be installed locally and offline from the wheelhouse folder using the option --no-index on pip install which ignores package index (only looking at --find-links URLs instead).
--find-links <url | path> looks for archive from url or path.

Since the original requirements.txt might have links to a non pip repo such as Github (https) pip will parse the links for the archive from a url and not from the wheelhouse, resulting in a http request.

This functions creates a new requirements.txt (it will backup the original requirements.txt into requirements.orig, and restore after packaging) with t only the name and version for each of the packages, thus eliminating the need to fetch / parse links from http sources and install all archives completely offline from the wheelhouse.

Finally, the script calls setuptools sdist — so you can change this to whatever format you want that is supported by setuptools.

MANIFEST.in
The last part for packaging the wheelhouse we need to tell setuptools to include this folder in the archive using graft wheelhouse:

MANIFEST.in gist

Notice that I prune (delete/skip from the archive) git folder itself.

Putting it all together:
Create the files in the root of your project:

• package.py
• setup.py
• MANIFEST.in

Call python setup.py package :
this will delete and recreate the wheelhouse folder on the root of your project, then it will collect the packages into the wheelhouse folder.
The last step will create the archive in the dist folder on the root of your project.

At this point you may copy the archive into a docker image or into your server any way you like.

Unpacking:
To unpack simply tar zxf archive.tar.gz and then call

pip install -r requirements.txt --use-wheel --no-index --find-links wheelhouse

your packages will be installed completely offline.

Bonus Tip:
If you plan on running this archive in docker container, i suggest you build a python docker image with all gcc OpenSSL and other build libs for installing all the dependancies on and from this container run setup.py package into a shared mount volume / copy to wherever you want. this way your runtime container is lean and does not need all the heavy build libs.