Simplify Your Python Developer Environment

Three tools (pyenv, pipx, pipenv) make for smooth, isolated, reproducible Python developer and production environments.

Using pyenv with pipenv to seamlessly manage multiple environments and pipx for my installed executables gives me a Python setup that's a pleasure to work with. You should use them.

The Issue

Python developer environments are notorious for being kind of a nuclear fallout zone. So much so that even the genius at XKCD has taken to commenting on the issue…

src https://xkcd.com/1987/

which is sad because Python is supposed to be a fun, easy language that enables people to write clean, simple code that can do complex things (as seen in this XKCD comic made years before).

src https://xkcd.com/353/

The main issue we want to avoid in a developer environment is package contamination across products, where a package from one project is used in multiple projects, each requiring a different version of the package. Since Python packaging and pip do not support different versions of a package being installed at the same time, this can turn into a headache very quickly. Or in the case of the packages conflicting at the system level (like wanting a newer version of Python), this may have you starting to look for a new profession, like chicken farming.

An example of this would be working on two different Django applications, one using Django 1.12 and the other using Django 2.0. The 2.0 release of Django changed a lot of how the projects are fundamentally set up and are not backwards compatible.

What is Needed and the Solutions Thus Far

In a perfect world there would be a single tool that allowed for

• Managing and using different Python Versions
• Easy creation around virtual environments that doesn’t take a ton of configuration and headache
• Distinguishing between project based virtual environments and just an isolated environment for a single Python binary/command line tool
• Easy management of packages and their transitive dependencies.

Many developers in the Python community have attempted to solve this problem and some great tooling has come out of it. Most Python developers work with or at least have heard of virtual environments. Unfortunately virtual environment management has become nearly as complicated as Python packaging itself. Do I use virtualenv or python -m venv? source .virtualenv/bin/activate or virtualenvwrapper and workon (which until very recently had been abandoned despite its popularity)? Packaging and installation also have their woes. easy_install vs. pip vs. source installation? Tooling exists around having multiple versions of Python and isolated environments, but the most popular isolated environment for binaries tool has been relatively abandoned for quite some time now.

Unfortunately, a single tool doesn’t exist to solve all of these problems, but a set of a few tools does exist that can accomplish all of this in a nice and relatively painless way.

Three Tools to Solve the Issue

There isn’t a single solution that exists today that can solve all of these issues. However, a lot of work has been done in this space recently and there is a combination of three tools that allows a developer to have a completely isolated and recreate-able environment all the way to the Python language version, not just the packages.

Note: This is for vanilla python development. For data science you will still need to use anaconda or miniconda. However, if you were using pip and a requirements.txt you can replace that section with the pipenv setup described below.

pyenv

About

pyenv is a simple-to-use Python version management tool, originally forked from rbenv and ruby-build and modified for Python. It allows for easy switches between different versions and vendors of Python. It supports regular CPython as well as anaconda, ironpython, jython, micropython, miniconda and pypy to name a few. It accomplishes this through shims placed at the front of the PATH and intercepting commands designated for Python, then passing the command to the version of Python you specified. You can read more about them here.

Some of the impressive features of pyenv include (shamelessly stolen from github):

• Allowing you to change the global version of Python on a per-user basis
• Supports per-project Python versions
• Override the Python version via environment variable
• Search commands from multiple versions of Python at a time

Pros and Cons

Ever since I found this I have thoroughly enjoyed how easy it is to test different versions of Python. When paired with the next two tools it’s absolutely amazing.

The only downside is that it does install all versions from source instead of taking advantage of operating system binaries (would be a pain to maintain) so the build and installation process can take a little bit of time at the beginning.

Things pyenv is

• A way to manage Python versions

Things pyenv isn’t

• A virtual environment manager (there is pyenv-virtualenv for that, but as you’ll see later in the blog we have a better tool for this)

pipx

About

pipx is a tool for running Python binaries (in this case, binaries refer to Python applications that can be run via the command line similar to system level binaries). This tool is a fork of the more popular (but unfortunately somewhat abandoned) pipsi. This tool takes the concept of npx “npm package runner” and ports it to Python. Using pipx you can run a binary a single time in a virtual environment. After the execution the virtual environment and all installed dependencies are removed. If you want to have the binary linger you can chose to install the binary permanently instead. It will still be within an individual virtual environment but will be made globally available with some path swizzling.

Pros and Cons

This tool filled in the gap that was missing with pipenv and virtualenvwrapper. For tools like awscli I would have a separate virtualenv and activate it when I needed to use that tool. After moving to pipenv I lost that ability unless I decided to create a directory for every tool (eww). With pipx I don’t have to worry about those environments anymore and they are now globally available to me via the CLI.

A downside of pipx is it is very new. At the time that I’m writing this it’s only a month old. It is pretty much a fork of pipsi with some added functionality, but I have had to open a bug report on the tool already. It also requires a Python version of 3.6 or higher.

Things pipx is

• A way to manage Python application binaries you want to install locally

Things pipx isn’t

• A virtual environment manager
• Probably not a production tool (I can’t see a use for it that I can’t accomplish with other, more robust tooling)

pipenv

About

pipenv is the packaging tool brought to us by Python wunderkind Kenneth Reitz (the man who brought us requests and the new micro webserver framework python-responder) whose goal is to make tools “for humans”. This tool does what many other interpreted languages have done, which is create a tooling around virtual environments, packaging and lockfiles.

• Node.js: yarn & npm (lockfile)
• PHP: Composer (lockfile)
• Rust: Cargo (lockfile)
• Ruby: Bundler (lockfile)
• Python (old way): pip + virtualenv (no lockfile)
• Python (new way): pipenv (lockfile)

Pipenv features

• Manages virtualenvs for projects
• Add/removes packages from your Pipfile as you install/uninstall packages
• Tracks both main and transitive dependency versions for packages
• Generates Pipfile.lock
• Run security scan on packages in the Pipfile
• Differentiates between production and development dependencies
• Graph dependencies

Setting Up Your Environment

Installation of pyenv

1. Install pyenv from github (recommended way). If you have a Mac you can use brew, but the version may be out of date.

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

2. The installer will output some configuration code at the end that needs to be added to your shell’s rc file.

#ensure the following is in your ~/.<SHELL>rc (.bashrc, .zshrc, .kshrc)
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
export PYENV_ROOT="$HOME/.pyenv" # needed by pipenv
export BYOBU_PYTHON=$PYENV_ROOT/shims/python3 # If you use byobu for managing screen or tmux

3. Once you have done this you can either exit the terminal and relaunch it or source your shell rc file.

# either logout or source your .<SHELL>rc file
source ~/.zshrc 

If you are running in a linux environment you may need to install other packages for pyenv to work fully. Reference the documentation here to ensure you have all the necessary packages.

Installation of pipx

1. Install pipx using pip. You can either use a system level pip installation, or install pip using the system python interpreter or one from pyenv.

# if you don't already have pip installed, but do have a system
# level python this will get you the latest version
python3 -m ensurepip --upgrade
# pipx requires Python 3.6 or higher
pip install pipx

2. The installer will output some configuration code at the end that needs to be added to your shells rc file.

# after the installation has finished you will need to put the following in your .<SHELL>rc (.bashrc, .zshrc, .kshrc)
export PATH=~/.local/bin:$PATH

3. Once you have done this you can either exit the terminal and relaunch it or source your shell rc file.

# either logout or source your .<SHELL>rc file
source ~/.zshrc

Installation of pipenv

1. Since we have pipx installed we can just install pipenv with that. If you have a mac you can install it with brew, but the version may not be the latest.

pipx install pipenv

What Can These Tools Do Alone?

pyenv Usage

• Install a specific version of Python.

pyenv install 3.7.1

• Set local version of Python.

pyenv local 3.7.1

• Set global version of Python.

pyenv global 3.7.1

• View available Python versions.

pyenv install -l

• View current version of the Python interpreter. This is the version of Python your commands will run against.

pyenv version

• View all versions of Python installed.

pyenv versions

• Install and use miniconda3.

pyenv install miniconda3-4.3.30
pyenv local miniconda3-4.3.30

You can pre-setup pyenv environments if you want, but the big advantage of these will come from pipenv. More commands can be found here.

pipx Usage

• One shot call to black (Python code formatter).

pipx black test.py

• Once done if I try to run black again it wont be found since pipx removed the environment it was installed in after execution.

~/# black
bash: black: command not found

• Install black so we have it for future use.

~/# pipx install black

• It has now been installed in a virtualenv and is globally accessible.

pipenv Usage

Pipfile uses TOML as its syntax. Pipenv creates a virtual environment per directory. So running the command in the directory with your code in it will create a virtualenv associated with that directory. You are able to activate the virtual environments from within that directory to work in them, or just run commands as if you were in the virtualenv.

Below is an example Pipfile and Pipfile.lock.

Note: Both the Pipfile and Pipfile.lock were meant to be stored in version control.

Sample Pipfile

[[source]]
# you can have multiple sources for Python repos
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[[source]]
url = "https://pypi.mydomain.com"
verify_ssl = true
name = "myPyPi"
[packages]
flask = "==1.0.1"
requests = "*"
[dev-packages]
pytest = "*"
[requires]
python_version = "3.6"

The above Pipfile creates a lockfile with the hashes of the packages and information about the package indexes it can pull from.

The Pipfile.lock is referenced whenever you attempt to do a pipenv install with a specified Pipfile. If the hashes in the lock file don’t match the packages, pipenv attempts to download it, will inform you of this, and rebuild the lock file. If you are trying to do a deployment and the hashes don’t match, the installation will fail.

Commands

• Install an environment with a predefined Pipfile. By default it will install development and production packages.

pipenv install

• Activate the shell (like you would with virtualenv source /bin/activate or workon with virtualenvwrapper) You must be in the director where the Pipfile was installed from.

pipenv shell

• Install an environment with a predefined Pipfile with production settings. This will not install development packages. If the Pipfile.lock doesn’t match the Pipfile this will fail.

pipenv install --deploy

• Install packages to the system level Python. This is useful for docker deployments, since a docker deployment should already be a single application environment. We don’t gain anything by using virtualenvs here.

pipenv install --system

• Install packages to the system level Python with production settings.

pipenv install --deploy --system

• Install a package. If a Pipfile already exists it will add it to the Pipfile. If you don’t have a Pipfile one will be created for you.

pipenv install requests

• Install a package from a different Python package index (setup for different indexes is shown above in the sample Pipfile).

pipenv install -i myPyPi requests

• Uninstall a package. It will remove it from the Pipfile.

pipenv uninstall requests

• Regenerate the Pipfile.lock (this can also be done by rerunning the install command).

pipenv lock

• Scan for security vulnerabilities (ensure that your Pipfile.lock is up to date).

pipenv check

• Graph dependencies.

pipenv graph

• Ensure that only packages specified in a Pipfile are installed. If you remove a package from a Pipfile and don’t uninstall it the package will linger there until removed.

pipenv clean

• Run a command within the virtualenv.

pipenv run python -c 'import requests'

• If requests is installed this will return nothing.
  As you see, we don’t have it installed in our system level Python so it returns an error.

python -c 'import requests';
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

• Get help, because pipenv can do so much more.

pipenv --help

pipenv is a very powerful tool. I highly recommend checking out their documentation.

Photo by Chris Barton on Unsplash

Bringing It All Together

We’ve seen what these tools can do individually, but bringing them together, even in small ways, makes your Python developer experience much more pleasant.

• Use pipx to install or upgradepipenv. pipenv is meant to be a global binary and pipx makes updating it very easy.

pipx install pipenv
pipx upgrade pipenv

• Integrate pipenv with pyenv so if you want to install an environment with a version of Python that isn't on your system, pyenv will automatically install it and use it for that environment. Note: I changed my requires section in my Pipfile before running this, otherwise it will tell you that while you want python3.7 your Pipfile specifies Python3.6

~/# pipenv --python 3.7.1
Warning: Python 3.7.1 was not found on your system…
Would you like us to install CPython 3.7.1 with pyenv? [Y/n]: y
Installing CPython 3.7.1 with pyenv (this may take a few minutes)…
⠹Downloading Python-3.7.1.tar.xz...
-> https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
Installing Python-3.7.1...
Installed Python-3.7.1 to /root/.pyenv/versions/3.7.1
Creating a virtualenv for this project…
Pipfile: /data/Pipfile
Using /root/.pyenv/versions/3.7.1/bin/python3.7 (3.7.1) to create virtualenv…
⠧Running virtualenv with interpreter /root/.pyenv/versions/3.7.1/bin/python3.7
Using base prefix '/root/.pyenv/versions/3.7.1'
/root/.local/pipx/venvs/pipenv/lib/python3.6/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
New python executable in /root/.local/share/virtualenvs/data-I7nS9QO2/bin/python3.7
Also creating executable in /root/.local/share/virtualenvs/data-I7nS9QO2/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /root/.local/share/virtualenvs/data-I7nS9QO2

• Install a pipenv with miniconda3. Note: I'm not going to pretend to be an expert on conda Python or data science. Documentation about pipenv and other Python distributions can be found here.

# install miniconda3
pyenv install miniconda3-4.3.30
# set it as the python used by current user (local)
pyenv local miniconda3-4.3.30
# Install with pipenv. Since the version of Python was set above it will use the conda Python binary for the virtualenv Python
pipenv install

Conclusion

All three of these tools are pretty amazing, but when we bring them together is when we create a fully isolated and reproducible Python environment.

I now have an easily maintainable per project virtual environment setup in which I can develop with any version of Python or miniconda I want, along with having Python binaries that I depend on like awscli also be isolated and easily maintainable.

And I think that’s just swell.