An Approach to Distributing Python Apps
Distributing a Python CLI using PyInstaller
At Retail Insight, we write several self-contained utilities and applications in Python, typically scheduled in our cloud environment, but they may also be run by users on-demand in their local machines. Distributing these applications is a demanding task due to the relationship between a Python application and its hosting environment.
This is a challenge I often face, so I wanted to chart my journey to explore solutions that will make this distribution simpler. In my case, the application must be deployed to several virtual machines in the cloud and installed on the local machines of multiple users, including Python developers, Analysts, and Product Managers.
As I explore the different options to remediate the challenges I face, I will be aiming to:
- Minimize the setup needed to get the app up and running
- Guarantee a uniform end-user experience
- Enable a smooth deployment process
- Address the issue of mismatch between dependencies in the development environment vs production environment
- Reduce the barrier to entry and make the application available to as many users as possible
Where to begin?
Due to the complexities of this project, one of the first things you must do is reflect on some critical questions before beginning, the answers to which will heavily influence your design decisions.
- Where is your software intended to run?
Once you have confirmed the environment in which the application will run, you should ask some functionality questions including:
- Does it have a Python interpreter installed?
- If the interpreter is installed, is it a version type that the app supports?
- If the interpreter is installed and is the right version, are extra packages already installed that might conflict with your applications?
2. Who makes up your intended user base and how can they access and run the latest version of the utility?
Making the application available to as many users as possible is a challenge in Python due to the nature of establishing development environments. A multi-step process which involves:
- Downloading and installing the specific Python version required by the app
- Setting up pip and a virtual environment
- Installing the application along with its dependencies
Now, after answering these questions, it is time to review the distribution landscape for a packaging solution.
Package Distribution Landscape
Selecting a Python distribution package is the first step; the landscape of options you can leverage is vast, yet, my main focus was across the following options:
Wheels:
This is the most common approach used by developers whereby the code is packaged into a wheel binary, dependencies are specified as part of it, and then pushed to an artifact repository. This approach caters to environments that have Python installed and a user base that knows how to install and manage dependencies.
PEX:
Short for Python Executable. Has been used by Twitter to deploy Python apps since 2011. Packages your python application code and its dependencies into a single executable file. This approach assumes that a Python interpreter is installed in the hosting environment.
EXEs:
Short for Executables. Also referred to as “Freezing your code”. Packages your application code, its dependencies, and the Python interpreter into a self-contained executable. The end-user can run the packaged app without installing a Python interpreter or any libraries.
Containers:
Container systems facilitate the distribution of a complete filesystem with everything you need to run your application. It requires installing a container system like docker in the host environment.
Considering these technologies in order of their decreased dependency on the hosting machine would look something like the below. As you move to the right there is a diminishing dependency on an already installed Python interpreter and on installing the application’s dependencies.
In the end, I decided upon an Exe package, this ticks all the boxes and eliminates the setup needed.
Let’s dig a bit deeper into the exe landscape.
EXE
There are plenty of options available here — see table below — so in order to help me choose, I formulated the following questions:
- Does it support Windows, Linux and MacOS?
- Does it support one-file mode (i.e., a single file artifact)?
- Does it support python versions ≥ 3.6, < 3.10?
- Is the project actively maintained? Did it have any releases in the past 6 months?
- Is it easy to setup and use?
After evaluating the type of solution, I landed on PyInstaller as the package of choice. It is actively maintained and with a good level of support. It supports Python 3, one file mode, and all three OS platforms. I have also found it easy to set up and use.
PyInstaller
What does it do?
How does it work?
Usage:
I added the pyinstaller library as a dependency to the application’s pyproject.toml and also committed both a .spec file, which is a python file that can be customized for your build, and a windows specific version_info.txt which will provide the version info needed for windows apps.
I included the step of building the Exe — as below — as part of a GitHub actions workflow, thereby automating the whole process of testing, building, and publishing.
Code breakdown:
--onefile: packages your entire app into a single executable--name: changes the name of your executable and .spec file--hiddenimport: imports that pyinstaller could not detect
Conclusion
As I reflect, my exploration has been a success. The application is running in a UAT environment and will be promoted to production soon! I managed to achieve all of my desired outcomes, but it was not all plain sailing — here are some key learnings I picked up along the way:
- PyInstaller extracts the exe to a temp folder which can add an overhead to the startup time of the application (It was not noticeable in my case).
- Premature termination of the app requires a cleanup to remove the temporary folders created by PyInstaller.
- Pyinstaller supports making executables for Windows, Linux and macOS but it cannot cross compile.
- The size of the exe is larger compared to Python wheels. In my case it is at ~32MB vs ~43 KB respectively.
Retail Insight takes data and turns it into action. Our advanced algorithms unlock valuable insights that drive better decision-making for retailers and CPGs.
Check us out: https://www.retailinsight.io/
