Python HOW: Farewell Anaconda! Take Full Control of Your Development Environment

Gabriel Harris Ph.D.

Follow

Dec 9, 2020 · 5 min read

Setting up python with pyenv, venv, pipx, and vsCode on Windows and macOS

Anaconda and Miniconda are amazing python distributions that get you up and running out-of-the-box. Once you start deploying your projects into production, however, you will defiantly need more control. In this article, I go through some of the best tools to make that transition happens! 🤘

All the CLI commands used in this article are in PowerShell for Windows, and zsh for macOS

Before starting up 🚿

If you’ve anaconda or miniconda installed, uninstall it as detailed here. For Windows, also:

• Delete $HOME\Documents\WindowsPowerShell\profile.ps1 if you have previously ran > conda init powershell
• Disable the built-in python launcher: search for Manage app execution aliases ▶️ disable App Installer aliases for python

Install python 🐍

Download the latest 64-bit python Release for Windows or macOS (3.9.0 at the time of writing), and install it (💀without adding python to PATH💀). The installation directory is:

⚠️ Because we haven’t added python to the system’s PATH, each time we want to use it, we have to give the full path. We can indeed do that. However, different projects require different versions of python, which makes this impractical. Solution? 👇

pyenv: python version management tool 🐍🐍

“pyenv lets you easily install and switch between multiple versions of python globally and locally” by using shim executables (more on shims here)

To understand how pyenv works, let’s first install it:

Windows: install pyenv-win with pip in the home directory, add an environment variable PYENV_HOME, and add bin & shims directories to the system’s PATH, and rehash shim:

MacOS: install with brew, add pyenv init to shell (init adds bin and shims to PATH, installs autocompletion, and rehash shim), and install some recommended python build dependencies:

Now we’ve pyenv installed, close and reopen your CLI, then let’s use it to install python versions 3.7.9 and 3.8.6:

pyenv installs python versions in the following directory:

To make version 3.8.6 global for example (i.e. register it with PATH through shims), we use the global command:

Great! now we can make any version of python global 🎉 (we didn’t talk about making it local but you can check the local command)

⚠️ However, not only that each project requires a different versions of python, but it also requires for that version to be isolated in a virtual environment with all the required packages for the project to run. This allows us to take a snapshot of all the project’s requirements for reproducibility and for deployment. Solution? 👇

venv 📩

“The venv module provides support for creating lightweight virtual environments with their own site directories”. The module was introduced in python 3.3, and as of then, it’s included with any installation of python

We can easily create an isolated virtual environment using the venv command. The key point is: the created environment will have its own python binary 🐍 which matches the version of the binary that was used to create it

Here, for example, the virtual environment we create will also have python 3.8.6:

Before installing any dependencies for that project, however, we need to activate the virtual environment. Luckily, activation scripts for powershell, bash, and batch are also created for us in the sub-folder Scripts 💃

To take a snapshot of your project dependencies, use the freeze command in pip:

If you already have the project’s requirements.txt, go through the same steps but instead of installing the dependencies as before, do: pip install -r requirements.txt to install them from the requirements file

Note: to delete the virtual environment, simply delete .venv folder

⚠️ There’re few tools that you should be using for any project, for example, code formatters and linters, that shouldn’t be included as a dependency,. It wouldn’t make sense to install them for each project then delete them from requirements.txt (and their own dependencies!). Solution? 👇

pipx

“pipx focuses on installing and managing Python packages that can be run from the CLI directly as applications”. pipx does that by creating an isolated environment for that package (and its dependencies), that we can then accesses from the CLI globally for any project 🙌

To understand how pipx works, let’s first install it (full instructions here):

Windows: install with pip from the main python installation (i.e. 3.9.0), and run ensurepath to add pipx directories to the system’s PATH:

MacOS: install with brew, and run ensurepath to add pipx directories to the system’s PATH:

The directories that ensurepath actually adds to PATH are its binary directory (so we can run it from the CLI), and the directory where packages’ binaries would go (so we can run them from the CLI), these are:

Now we’ve pipx installed, close and reopen your CLI, then let’s use it to install the following helper packages:

• Black code formatter with “a strict subset of PEP 8 coding style”
• isort for sorting imports alphabetically, and automatically separated into sections and by type
• flake8 code linter for style guide enforcement

Each package will be installed in an isolated virtual environment inside $Home\.local\pipx\venvs, and have its binary in $Home\.local\bin (which is already in PATH thanks to ensurepath). To use any of these applications, we can simply run it in the CLI:

⚠️ If you’re using vsCode, you might be wondering “how to tell vsCode where to find the global packages installed with pipx so I can use them with any project? Solution? 👇

Setting up vsCode

Settings for pipx packages

You can tell vsCode to use all the helper packages we installed using pipx, and more importantly, where to find the binaries 📦 in the settings

vsCode saves settings in a settings.json file, Ctrl+Shift+P to bring the Palette, then search for Preferences: Open Settings (JSON):

• Define the formatting provider as black and provide its binary path (more on formatting here). As black defaults to 88 characters per line, we use the same length for the editor’s ruler (more on line length here)

• Define the imports sorting path for isort (vsCode uses isort as an import sorting provider by default as you can see here). “Black also formats imports, but in a different way from isort defaults which leads to conflicting changes” 😵. To fix this, we can pass few arguments to isort to make it consistent with black (more details here)

• Enable flake8 for linting and provide its binary path (more on linting here). “There are a few deviations that cause incompatibilities with black”. To fix this, we can pass few arguments to make flake8 consistent with black(more details here)

Settings Sync

You can sync your settings, keybindings, and installed extensions across your machines by following the instructions here (you need a Microsoft or a Git account). However, all the binaries paths for the packages we installed using pipx shouldn’t be synced between Windows and macOS. Luckily, we can ignore these by adding them to settings.json:

Note: you can add any settings in settings.json to the list of ignored settings above

What is next?

• Start using Docker 🐋 for Python
• Become a Scikit-learn plumber 🚿

Happy coding!