virtualenv demystified

If you have difficulty understanding virtualenv, one reason is it’s misnamed. I don’t see anything virtual about it. It would be more accurate to call it an isolated python environment, which is what I will be calling it in this article.

Installation

On Debian and Ubuntu, you install virtualenv with apt-get install python-virtualenv python3-virtualenv. On Windows, pip install virtualenv.

Python 3.3 and later has the functionality built-in, in the venv standard library module. However, virtualenv is still more popular and we will use that one here, because it also works on Python 2.7 and because it is supported by virtualenvwrapper (we will check it out below).

Basics

Fire up python and give it these commands:

import sys
sys.path

It will give you a list of directories. This is where it searches for modules when you ask it to import something. When you run python interactively the first item is the empty string, which denotes the current directory. On my system, the next item is /usr/lib/python3.4, and so on.

Now, let’s create an isolated python environment:

Unix

virtualenv /tmp/test1

Windows

virtualenv C:\test1

What the above command does is install a slightly modified python interpreter under /tmp/test1/bin (or C:\test1\Scripts). Run it with /tmp/test1/bin/python (or C:\test1\Scripts\python) and check the python path:

import sys
sys.path

As you will see, the path is different from before. Now most of the directories Python will search when you try to import something are under /tmp/test1 (or C:\test1). There are still a few system directories listed, but these are for the standard Python library. Any modules you have installed system-wide are not in the system path.

virtualenv also installs a modified version of pip under /tmp/test1/bin (or C:\test1\Scripts). If you run /tmp/test1/bin/pip install django (or C:\test1\Scripts\pip install django), this will install django somewhere inside /tmp/test1 (or C:\test1).

This is pretty much everything virtualenv does. It installs a modified Python interpreter that has a different python path; and it installs a modified version of pip that installs stuff in the isolated environment directory instead of system-wide.

If you have both Python 2 and Python 3 installed on your system, you can specify which one you want to use with the --python option; for example, in Debian/Ubuntu:

virtualenv --python=/usr/bin/python3

You can remove an isolated environment simply by deleting the directory and its contents.

Activating

It’s hard to type /tmp/test1/bin/python, so there is the activate script:

Unix

source /tmp/test1/bin/activate

Cmd

C:\test1\Scripts\activate

PowerShell *

C:\test1\Scripts\activate.ps1

(* In PowerShell, you may first need to type Set-ExecutionPolicy Unrestricted to allow the execution of unsigned scripts.)

After you run this, if you type python, /tmp/test1/bin/python will run; same with pip.

activate works by changing the shell path. The shell is the software that interprets your commands; for example, when you type python and press enter, something must get your keystrokes and decide to execute python; that something is the shell. If you are on Linux, your shell is probably “bash”; if you are on Windows, it’s “cmd” or the PowerShell. Just as the Python path is a list of directories where Python searches for modules, the shell path is a list of directories where the shell searches for executable files. When you type a command such as python, the shell looks in these directories in order to locate the executable. You can see the shell path with echo $PATH in Unix, echo %path% in Windows cmd, and $Env:path in PowerShell.

So, the only thing the activate script does is change the shell path. You can use deactivate to change it back to what it was.

Virtualenvwrapper

Isolated python environments are so cool that you will soon have too many. Virtualenvwrapper is a set of shell scripts that makes it much easier to manage these environments.

On Debian/Ubuntu, you can install it with apt-get install virtualenvwrapper. On Windows you need pip install virtualenvwrapper-win for cmd and pip install virtualenvwrapper-powershell for PowerShell; I’m not certain how well these are supported, however, especially the PowerShell version, which seems unmaintained since 2012. Note that you must run these pip commands system-wide, not in an isolated environment (i.e. you need to deactivate any isolated environment first).

mkvirtualenv test2

This creates and activates an isolated python environment called test2. It stores it somewhere, but the thing is, you don’t need to know where. In Debian/Ubuntu it’s usually in the .virtualenvs subdirectory of your home directory. You can use deactivate to deactivate it. Activate it again like this:

workon test2

You can check which virtualenvs you have like this:

lsvirtualenv

Finally, remove a virtualenv like this:

rmvirtualenv test2

In production servers I use plain virtualenv; virtualenvwrapper is very handy in development.

System site packages

You can add the --system-site-packages option to virtualenv or mkvirtualenv, like this:

virtualenv --system-site-packages /tmp/test1

or

mkvirtualenv --system-site-packages test2

Now, as before, if you tell it pip install django, it will install django in the isolated environment. The difference is that the python path will also include everything that has been installed system-wide. So if you have installed, for example, requests, system-wide, it will use it (unless you also install requests in the virtualenv, in which case the latter will take precedence).

Many people prefer their isolated environments to be, well, isolated, so they don’t use this option, but I use it often in order to avoid compiling packages. pip install gdal can take a long time because it needs to compile stuff, whereas apt-get install python-gdal is way faster; so I use --system-site-packages in order to be able to use the system-wide gdal (this issue is becoming less important now that most Python packages are distributed compiled, as a Python wheel). In addition, psycopg2 can be tricky to compile, and it is usually better to use the operating system’s version, which is usually guaranteed to work with the operating system’s postgresql. In Windows, however, I normally don’t use --system-site-packages, as there are no site packages packaged with the operating system.

The “go to jail” image is © 2011 Ken Teegardin from seniorliving.org.

This is a republication of an old article first published in my Django Deployment site.