Simplify Your Python Developer Environment

Three tools (pyenv, pipx, pipenv) make for smooth, isolated, reproducible Python developer and production environments.

Mason Egger
Nov 6, 2018 · 11 min read

Using pyenv with pipenv to seamlessly manage multiple environments and pipx for my installed executables gives me a Python setup that's a pleasure to work with. You should use them.

The Issue

src https://xkcd.com/1987/

which is sad because Python is supposed to be a fun, easy language that enables people to write clean, simple code that can do complex things (as seen in this XKCD comic made years before).

src https://xkcd.com/353/

The main issue we want to avoid in a developer environment is package contamination across products, where a package from one project is used in multiple projects, each requiring a different version of the package. Since Python packaging and pip do not support different versions of a package being installed at the same time, this can turn into a headache very quickly. Or in the case of the packages conflicting at the system level (like wanting a newer version of Python), this may have you starting to look for a new profession, like chicken farming.

An example of this would be working on two different Django applications, one using Django 1.12 and the other using Django 2.0. The 2.0 release of Django changed a lot of how the projects are fundamentally set up and are not backwards compatible.

What is Needed and the Solutions Thus Far

  • Managing and using different Python Versions
  • Easy creation around virtual environments that doesn’t take a ton of configuration and headache
  • Distinguishing between project based virtual environments and just an isolated environment for a single Python binary/command line tool
  • Easy management of packages and their transitive dependencies.

Many developers in the Python community have attempted to solve this problem and some great tooling has come out of it. Most Python developers work with or at least have heard of virtual environments. Unfortunately virtual environment management has become nearly as complicated as Python packaging itself. Do I use virtualenv or python -m venv? source .virtualenv/bin/activate or virtualenvwrapper and workon (which until very recently had been abandoned despite its popularity)? Packaging and installation also have their woes. easy_install vs. pip vs. source installation? Tooling exists around having multiple versions of Python and isolated environments, but the most popular isolated environment for binaries tool has been relatively abandoned for quite some time now.

Unfortunately, a single tool doesn’t exist to solve all of these problems, but a set of a few tools does exist that can accomplish all of this in a nice and relatively painless way.

Three Tools to Solve the Issue

Note: This is for vanilla python development. For data science you will still need to use anaconda or miniconda. However, if you were using pip and a requirements.txt you can replace that section with the pipenv setup described below.

pyenv

About

Some of the impressive features of pyenv include (shamelessly stolen from github):

  • Allowing you to change the global version of Python on a per-user basis
  • Supports per-project Python versions
  • Override the Python version via environment variable
  • Search commands from multiple versions of Python at a time

Pros and Cons

The only downside is that it does install all versions from source instead of taking advantage of operating system binaries (would be a pain to maintain) so the build and installation process can take a little bit of time at the beginning.

Things pyenv is

  • A way to manage Python versions

Things pyenv isn’t

  • A virtual environment manager (there is pyenv-virtualenv for that, but as you’ll see later in the blog we have a better tool for this)

pipx

About

Pros and Cons

A downside of pipx is it is very new. At the time that I’m writing this it’s only a month old. It is pretty much a fork of pipsi with some added functionality, but I have had to open a bug report on the tool already. It also requires a Python version of 3.6 or higher.

Things pipx is

  • A way to manage Python application binaries you want to install locally

Things pipx isn’t

  • A virtual environment manager
  • Probably not a production tool (I can’t see a use for it that I can’t accomplish with other, more robust tooling)

pipenv

About

  • Node.js: yarn & npm (lockfile)
  • PHP: Composer (lockfile)
  • Rust: Cargo (lockfile)
  • Ruby: Bundler (lockfile)
  • Python (old way): pip + virtualenv (no lockfile)
  • Python (new way): pipenv (lockfile)

Pipenv features

  • Manages virtualenvs for projects
  • Add/removes packages from your Pipfile as you install/uninstall packages
  • Tracks both main and transitive dependency versions for packages
  • Generates Pipfile.lock
  • Run security scan on packages in the Pipfile
  • Differentiates between production and development dependencies
  • Graph dependencies

Setting Up Your Environment

Installation of pyenv

curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

2. The installer will output some configuration code at the end that needs to be added to your shell’s rc file.

#ensure the following is in your ~/.<SHELL>rc (.bashrc, .zshrc, .kshrc)export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
export PYENV_ROOT="$HOME/.pyenv" # needed by pipenv
export BYOBU_PYTHON=$PYENV_ROOT/shims/python3 # If you use byobu for managing screen or tmux

3. Once you have done this you can either exit the terminal and relaunch it or source your shell rc file.

# either logout or source your .<SHELL>rc filesource ~/.zshrc 

If you are running in a linux environment you may need to install other packages for pyenv to work fully. Reference the documentation here to ensure you have all the necessary packages.

Installation of pipx

# if you don't already have pip installed, but do have a system
# level python this will get you the latest version
python3 -m ensurepip --upgrade
# pipx requires Python 3.6 or higherpip install pipx

2. The installer will output some configuration code at the end that needs to be added to your shells rc file.

# after the installation has finished you will need to put the following in your .<SHELL>rc (.bashrc, .zshrc, .kshrc)export PATH=~/.local/bin:$PATH

3. Once you have done this you can either exit the terminal and relaunch it or source your shell rc file.

# either logout or source your .<SHELL>rc filesource ~/.zshrc

Installation of pipenv

pipx install pipenv

What Can These Tools Do Alone?

pyenv Usage

pyenv install 3.7.1
  • Set local version of Python.
pyenv local 3.7.1
  • Set global version of Python.
pyenv global 3.7.1
  • View available Python versions.
pyenv install -l
  • View current version of the Python interpreter. This is the version of Python your commands will run against.
pyenv version
  • View all versions of Python installed.
pyenv versions
  • Install and use miniconda3.
pyenv install miniconda3-4.3.30
pyenv local miniconda3-4.3.30

You can pre-setup pyenv environments if you want, but the big advantage of these will come from pipenv. More commands can be found here.

pipx Usage

pipx black test.py
  • Once done if I try to run black again it wont be found since pipx removed the environment it was installed in after execution.
~/# blackbash: black: command not found
  • Install black so we have it for future use.
~/# pipx install black
  • It has now been installed in a virtualenv and is globally accessible.

pipenv Usage

Below is an example Pipfile and Pipfile.lock.

Note: Both the Pipfile and Pipfile.lock were meant to be stored in version control.

Sample Pipfile

[[source]]# you can have multiple sources for Python reposurl = "https://pypi.python.org/simple"verify_ssl = truename = "pypi"[[source]]url = "https://pypi.mydomain.com"verify_ssl = truename = "myPyPi"[packages]flask = "==1.0.1"requests = "*"[dev-packages]pytest = "*"[requires]python_version = "3.6"

The above Pipfile creates a lockfile with the hashes of the packages and information about the package indexes it can pull from.

The Pipfile.lock is referenced whenever you attempt to do a pipenv install with a specified Pipfile. If the hashes in the lock file don’t match the packages, pipenv attempts to download it, will inform you of this, and rebuild the lock file. If you are trying to do a deployment and the hashes don’t match, the installation will fail.

Commands

pipenv install
  • Activate the shell (like you would with virtualenv source /bin/activate or workon with virtualenvwrapper) You must be in the director where the Pipfile was installed from.
pipenv shell
  • Install an environment with a predefined Pipfile with production settings. This will not install development packages. If the Pipfile.lock doesn’t match the Pipfile this will fail.
pipenv install --deploy
  • Install packages to the system level Python. This is useful for docker deployments, since a docker deployment should already be a single application environment. We don’t gain anything by using virtualenvs here.
pipenv install --system
  • Install packages to the system level Python with production settings.
pipenv install --deploy --system
  • Install a package. If a Pipfile already exists it will add it to the Pipfile. If you don’t have a Pipfile one will be created for you.
pipenv install requests
  • Install a package from a different Python package index (setup for different indexes is shown above in the sample Pipfile).
pipenv install -i myPyPi requests
  • Uninstall a package. It will remove it from the Pipfile.
pipenv uninstall requests
  • Regenerate the Pipfile.lock (this can also be done by rerunning the install command).
pipenv lock
  • Scan for security vulnerabilities (ensure that your Pipfile.lock is up to date).
pipenv check
  • Graph dependencies.
pipenv graph
  • Ensure that only packages specified in a Pipfile are installed. If you remove a package from a Pipfile and don’t uninstall it the package will linger there until removed.
pipenv clean
  • Run a command within the virtualenv.
pipenv run python -c 'import requests'
  • If requests is installed this will return nothing.
    As you see, we don’t have it installed in our system level Python so it returns an error.
python -c 'import requests';
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'
  • Get help, because pipenv can do so much more.
pipenv --help

pipenv is a very powerful tool. I highly recommend checking out their documentation.

Photo by Chris Barton on Unsplash

Bringing It All Together

  • Use pipx to install or upgradepipenv. pipenv is meant to be a global binary and pipx makes updating it very easy.
pipx install pipenvpipx upgrade pipenv
  • Integrate pipenv with pyenv so if you want to install an environment with a version of Python that isn't on your system, pyenv will automatically install it and use it for that environment. Note: I changed my requires section in my Pipfile before running this, otherwise it will tell you that while you want python3.7 your Pipfile specifies Python3.6
~/# pipenv --python 3.7.1
Warning: Python 3.7.1 was not found on your system…
Would you like us to install CPython 3.7.1 with pyenv? [Y/n]: y
Installing CPython 3.7.1 with pyenv (this may take a few minutes)…
⠹Downloading Python-3.7.1.tar.xz...
-> https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
Installing Python-3.7.1...
Installed Python-3.7.1 to /root/.pyenv/versions/3.7.1
Creating a virtualenv for this project…
Pipfile: /data/Pipfile
Using /root/.pyenv/versions/3.7.1/bin/python3.7 (3.7.1) to create virtualenv…
⠧Running virtualenv with interpreter /root/.pyenv/versions/3.7.1/bin/python3.7
Using base prefix '/root/.pyenv/versions/3.7.1'/root/.local/pipx/venvs/pipenv/lib/python3.6/site-packages/virtualenv.py:1041: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative usesimport impNew python executable in /root/.local/share/virtualenvs/data-I7nS9QO2/bin/python3.7
Also creating executable in /root/.local/share/virtualenvs/data-I7nS9QO2/bin/python
Installing setuptools, pip, wheel...done.
Virtualenv location: /root/.local/share/virtualenvs/data-I7nS9QO2
  • Install a pipenv with miniconda3. Note: I'm not going to pretend to be an expert on conda Python or data science. Documentation about pipenv and other Python distributions can be found here.
# install miniconda3
pyenv install miniconda3-4.3.30
# set it as the python used by current user (local)
pyenv local miniconda3-4.3.30
# Install with pipenv. Since the version of Python was set above it will use the conda Python binary for the virtualenv Python
pipenv install

Conclusion

I now have an easily maintainable per project virtual environment setup in which I can develop with any version of Python or miniconda I want, along with having Python binaries that I depend on like awscli also be isolated and easily maintainable.

And I think that’s just swell.

Expedia Group Technology

Stories from the Expedia Group Technology teams