Wanna learn how to this? keep reading.

Set up a Shell Python Development Environment From Scratch — Part 2

Python, Working with VirtualEnvironments

Posts in the series

1. Part 1, Zsh & Vim configuration:

Set up a Shell Python development environment from scratch— Part 1, Zsh & Vim configuration.

1. Part 2 — Python,Working with VirtualEnvironments

Set up a Shell Python development environment from scratch — Part 2— Python, Working with…

Table of contents

1. Pyenv and Virtualenv plugin: Why use virtual environments at all?
2. Managing virtual environments requirements with pip-tools

1. Pyenv and Virtualenv plugin: Why use virtual environments at all?

First thing first why would you want to work withing virtual environments?, why not just install everything on the system installation of Python?.

There are several reasons for that, the arguably the most important ones are dependency management and project isolation, by creating a new environment from scratch for each single project you can install only the dependencies you actually need without clustering your system environment, and you can install packages in any version that fits your project needs, saving lots of time trying to find issues in dependencies. It keep your system environment free of unnecessary dependencies that would otherwise put at risk system processes that depend on python to run.

Back to preparing the tools for the job, there are many alternatives to doing this, arguably though the most Pythonic ones are Pyenv with Virtualenv plugins. You can manage from the same tool python versions (install and manage different version of Python 2.7, 3.5, 3.9, miniconda, etc.), and at the same time mange virtual environment created with those python versions.

First check that you python installed (remember that ❯ is just a symbol to indicate that the command should be executed in a shell terminal):

❯ python --version

The output should be something similar to below:

❯ python — version
Python 3.9.1

Then install Pyenv and plugins by doing:

❯ curl https://pyenv.run | bash

This last command will download a shell script that install all the dependencies for you. Close the current terminal and open a new one or execute the command below to re-load the terminal.

❯ exec $SHELL

Now let’s do some testing, let’s say that you are interested in testing an old project which depends on python 3.5 to run but your system has python 3.8 only, do you install python 3.5 system wide?, nop, we use pyenv to install and then create a virtual environment for that project to live in.

First let’s check our system installation:

❯ pyenv versions
  system

System is the system’s python installation, and it’s better not touch it at all, now before installing python 3.5, we need to be aware that when installing new python versions, sometimes is possible to find some errors due to missing dependencies in the system (specially older versions of python), so to fix this first we will add to the system all the build dependencies:

❯ sudo apt-get update; sudo apt-get install --no-install-recommends make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

Now we are ready to install python 3.5 by doing:

❯ pyenv install -v 3.5.0

This will download and make available the installed version of python, you can see the full list of available versions by doing:

❯ pyenv install -l
Available versions:
  2.1.3
  2.2.3
  2.3.7
  2.4.0
  2.4.1
...
Etc

Now in order to create a virtual-environment with that new/old version the command is “pyenv virtualenv python_version name_of_venv”, for example for the one we just installed:

❯ pyenv virtualenv 3.5.0 myvenv

Then when you list versions it will show you also the virtual environments created.

❯ pyenv versions
  system
  3.5.0
  3.5.0/envs/myvenv
  myvenv

OK cool now you have created a shiny new virtual environment but, how do you tell your project to run using this virtual environment instead of the system version (or global version, see more details on pyenv github page), the best way is by creating a file in the root of your project directory named .python-version which contains the name of the virtual environment you want to load, if you followed Part 1 of this series on configuring Zsh then the shell will automatically switch to the virtual environment for you upon detecting this file!!, no need to manually activate and deactivate (although you can do that as well if you want).

To create the file you can do “pyenv local name_of_virtual_environment”:

❯ pyenv local myvenv

This will automatically create the file for you, of course you can manually create it as well using vim for instance.

Now whenever you get into project directory with a .python-version file inside you will see how it changes automatically for you.

2. Managing virtual environments requirements with pip-tools

The last piece of advice I want to share is, using pip-tools can simplify the work of managing your dependencies, you always know exactly which files packages you installed yourself and which are dependencies of those packages.

for doing this first we need to switch into our project directory where our .python-version will activate our environment for us, make sure you see the environment name on the right (of left depending your config).

Install pip-tools in your virtual environment:

❯ pip install pip-tools

Next write with vim (or your favourite text editor) a file named requirements.in (is a convention in reality you can name however you want)

Example for vim:

❯ vim requirements.in

once inside the file the name of the packages you want to use in your project, one per line, for example:

numpy
pandas
seaborn
tensorflow
keras

then exit while saving with :wp (assuming you are on vim).

now compile a requirements.txt file that has no dependencies conflicts by using pip-tools:

❯ pip-compile requirements.in

this will create the file requirements.txt for you, finally execute:

❯ pip-sync

What this does is read the requirements.txtfile and install all the dependencies into your virtual environment automatically, just watch the magic happen in the gif below.

Demo of the full process

And that’s it for this series of pots about setting up a developing environment for python 100% in the shell, let me know below in comments what you think, and share your awesome terminal configs.

In the next posts we will use our shiny new environment to develop and end to end machine learning app hosted on Heroku, stay tuned!!