An Introduction to Virtual Environments in Python

I’ve been taking some time off from projects recently, primarily to focus on teaching myself math, but also to learn a new language (Java) and reacquaint myself with my first language: Python. This week I figured I’d tackle a subject which confused me when I first started learning Python, namely virtual environments — what they are and how best to use them.


What Are Virtual Environments (and Why Do We Care?)

The short answer is this: virtual environment’s are Python’s way of separating dependencies between projects. Whereas Ruby’s bundler or JavaScript’s npm come with this feature built-in, Python’s package manager, pip, does not.

If that means nothing to you, then let me give an example: say you’re going to build an awesome web app using Django. You download and install the latest version, which let’s posit is 1.7.11 at the time. You spend a bunch of time working on it until you eventually finish it, and it looks great.

Down the line, somebody is impressed by your handiwork and asks you to build a web app for them too. By this point, a new version of Django has come out — let’s say 2.0.2 — and it has a bunch of new features you’re itching to utilize. So you download and install the new version, and use it to build their app for them. You finish it, everybody’s happy, and all is well and good… at least until you decide you want to make an additional edit to the first app.

See, in the process of upgrading from Django version 1.7.11 to 2.0.2, you accidentally broke your original app. Maybe some of the things you used in 1.7.11 were deprecated, or maybe other packages which worked with the old version of Django haven’t been updated to work with the new version. All you know is that now you have to spend countless frustrating hours trying to put everything back together again… and you can’t just downgrade back to the old version of Django because that’ll break the new app instead. Welcome to dependency hell.

This is the problem that virtual environments solve. More specifically, the solution would be to create one virtual environment for your original web app (with Django 1.7.11 installed) and a separate one for the new web app (with Django 2.0.2). Then the two versions of Django can coexist without interfering with each other.


Getting Started

So how do we work with virtual environments? The traditional tool of choice has been the third-party module virtualenv. However, Python 3.3 introduced a standard library module called venv which, while functionally very similar to virtualenv, is newer and in some ways better optimized. Therefore, venv is what I’ll be using here, with Python 3.6.4.

The first thing to do is to make a folder which will house all of your virtual environments. What you name it is up to you, but for now I’ll call it .venvs:

mkdir ~/.venvs && cd ~/.venvs

The next step is to make the virtual environment itself. Assuming that you’re on Python 3.6 or above, run:

python3 -m venv my-first-environment

Between Python 3.3 and 3.5, the recommended way of creating virtual environments was using the pyvenv command — However, this was changed with 3.6. The -m flag is necessary because we’re running a library module as a script, specifically venv, which creates the virtual environment and gives it whichever name you specify, in this case my-first-environment.

Running this command should generate a folder with an internal file structure that looks something like this:

├── bin
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── easy_install
│ ├── easy_install-3.6
│ ├── pip
│ ├── pip3
│ ├── pip3.6
│ ├── python -> python3.6
│ ├── python3 -> python3.6
├── include
├── lib
│ └── python3.6
│ └── site-packages
└── pyvenv.cfg

The most important things to take note of here are the site-packages sub-directory within lib and the activate scripts within bin. site-packages is where any third-party modules you install will be stored, while activate is what you’ll run to use the virtual environment. Let’s do that now:

source ~/.venvs/my-first-environment/bin/activate

source can be more succinctly expressed as just ., which I’ll be using going forward (the two are equivalent). When you run this, you should see (my-first-environment) pop up to the left of your bash prompt, like so:

This lets you known that the my-first-environment virtual environment is now active. We can verify this by running:

which python3

Which should display something like this:

Note that the version of Python 3 being referenced is the one within the virtual environment. Now any packages installed with pip should be contained within this environment, and inaccessible outside of it.

Exiting the virtual environment is easy. You just run:

deactivate

And there you go. Back to normal.


Saving and Copying Dependencies

Newly created virtual environments start out empty, with nothing installed except for pip and setuptools. But what if you want to copy the packages used by one virtual environment to another? This is necessary for collaboration — there has to be some way for collaborators to see which dependencies the project requires and install them.

The most basic means of accomplishing this is to generate a file called requirements.txt. But to see it in action, we have to install something. Let’s do that, first reactivating our virtual environment like so:

. ~/.venvs/my-first-environment/bin/activate
pip install flask

Flask is a lightweight web framework (akin to Sinatra, for those coming from Ruby). It has some additional dependencies which will also be installed alongside it. You can view those dependencies by running:

pip freeze

This should output something that looks like this:

You can now save this into a requirements.txt file like so:

pip freeze > requirements.txt

If you cat that file (or just open it), you should see that it has the same contents shown above. You can use this file to install the packages listed therein into a new virtual environment. But first, you’ll have to deactivate the current virtual environment and create a new one:

deactivate
python3 -m venv my-second-environment

Then you can source the new virtual environment, and install all the packages listed in requirements.txt:

. ~/.venvs/my-second-environment/bin/activate
pip install -r requirements.txt

There we go! Now my-second-environment should contain all the same dependencies as my-first-environment. You can confirm this by running:

pip list

Making Life Easier With Pipenv

If you’re coming from Ruby or JavaScript, this whole process might seem a little clunky. In those languages, you don’t need to create virtual environments or manually generate requirements.txt files because all that functionality is handled automatically through their respective package managers. Thankfully, a tool has come along which provides a more bundler/npm-like experience for Python: pipenv.

Pipenv is a third-party package, so first you’ll need to install it:

pip install pipenv

Then you can change into your product directory (or create a new directory if you’re starting a new project, as I’m doing here), and instantiate pipenv there:

mkdir ~/dev/projects/my-new-project && cd ~/dev/projects/my-new-project
pipenv install

This will create two new files, a Pipfile and a Pipfile.lock. These function similarly to Ruby’s Gemfile and Gemfile.lock, so if you’re familiar with those you should feel right at home. It’ll also automatically create a new virtual environment for the project, which you can locate with:

pipenv --venv

The Pipfile will be automatically updated whenever you install a new package, so you no longer need to worry about generating or updating your requirements.txt file. When you share the project with someone else, all they need to do (assuming they have pipenv installed) is run:

pipenv install

Likewise, adding new packages to your project is simple. Let’s install flask, as we did before:

pipenv install flask

To show which packages you have installed (and the dependencies of those dependencies) you can run:

pipenv graph

Which should output something like this:

Uninstalling is also a breeze, although bear in mind you’ll need to list any sub-dependencies you want to uninstall as well. In this case:

pipenv uninstall flask click itsdangerous jinja2 markupsafe werkzeug

To activate the virtual environment that pipenv automatically generated, you can run:

pipenv shell

Now any scripts you run will have access to all the packages listed within the Pipfile. Alternatively, you can run scripts without activating the virtual environment first by running commands like so:

pipenv run python my-script.py

If this seems like a lot of typing to have to do every time you want to run something, you can alias it by adding the following line to your ~/.bash_profile:

alias prp="pipenv run python"

Then source it:

. ~/.bash_profile

And now you can run scripts more succinctly:

prp my-script.py

Conclusion

Most of what I covered here is quite new — pipenv debuted only a little over a year ago, and venv won’t work with versions of Python older than 3.3. If you’re working with an older version of Python, you’re going to have to use virtualenv instead of env, which you can read more about here.

Accompanying virtualenv is another popular tool called virtualenvwrapper, which streamlines the use of virtualenv. I choose to focus on pipenv instead, since I prefer it, but if you’d like to learn more about virtualenvwrapper you can do so here.

Lastly, there’s Anaconda. Anaconda is a distribution of Python which focuses primarily on data science and machine learning and comes with its own package management system called conda. If you’re planning on using Anaconda and you’d like to learn more about how to use its package manager, you can read up about that here.