An Introduction to Virtual Environments in Python
I’ve been taking some time off from projects recently, primarily to focus on teaching myself math, but also to learn a new language (Java) and reacquaint myself with my first language: Python. This week I figured I’d tackle a subject which confused me when I first started learning Python, namely virtual environments — what they are and how best to use them.
What Are Virtual Environments (and Why Do We Care?)
If that means nothing to you, then let me give an example: say you’re going to build an awesome web app using Django. You download and install the latest version, which let’s posit is 1.7.11 at the time. You spend a bunch of time working on it until you eventually finish it, and it looks great.
Down the line, somebody is impressed by your handiwork and asks you to build a web app for them too. By this point, a new version of Django has come out — let’s say 2.0.2 — and it has a bunch of new features you’re itching to utilize. So you download and install the new version, and use it to build their app for them. You finish it, everybody’s happy, and all is well and good… at least until you decide you want to make an additional edit to the first app.
See, in the process of upgrading from Django version 1.7.11 to 2.0.2, you accidentally broke your original app. Maybe some of the things you used in 1.7.11 were deprecated, or maybe other packages which worked with the old version of Django haven’t been updated to work with the new version. All you know is that now you have to spend countless frustrating hours trying to put everything back together again… and you can’t just downgrade back to the old version of Django because that’ll break the new app instead. Welcome to dependency hell.
This is the problem that virtual environments solve. More specifically, the solution would be to create one virtual environment for your original web app (with Django 1.7.11 installed) and a separate one for the new web app (with Django 2.0.2). Then the two versions of Django can coexist without interfering with each other.
So how do we work with virtual environments? The traditional tool of choice has been the third-party module virtualenv. However, Python 3.3 introduced a standard library module called venv which, while functionally very similar to virtualenv, is newer and in some ways better optimized. Therefore, venv is what I’ll be using here, with Python 3.6.4.
The first thing to do is to make a folder which will house all of your virtual environments. What you name it is up to you, but for now I’ll call it
mkdir ~/.venvs && cd ~/.venvs
The next step is to make the virtual environment itself. Assuming that you’re on Python 3.6 or above, run:
python3 -m venv my-first-environment
Between Python 3.3 and 3.5, the recommended way of creating virtual environments was using the
pyvenv command — However, this was changed with 3.6. The
-m flag is necessary because we’re running a library module as a script, specifically
venv, which creates the virtual environment and gives it whichever name you specify, in this case
Running this command should generate a folder with an internal file structure that looks something like this:
│ ├── activate
│ ├── activate.csh
│ ├── activate.fish
│ ├── easy_install
│ ├── easy_install-3.6
│ ├── pip
│ ├── pip3
│ ├── pip3.6
│ ├── python -> python3.6
│ ├── python3 -> python3.6
│ └── python3.6
│ └── site-packages
The most important things to take note of here are the
site-packages sub-directory within
lib and the
activate scripts within
site-packages is where any third-party modules you install will be stored, while
activate is what you’ll run to use the virtual environment. Let’s do that now:
source can be more succinctly expressed as just
., which I’ll be using going forward (the two are equivalent). When you run this, you should see
(my-first-environment) pop up to the left of your bash prompt, like so:
This lets you known that the
my-first-environment virtual environment is now active. We can verify this by running:
Which should display something like this:
Note that the version of Python 3 being referenced is the one within the virtual environment. Now any packages installed with pip should be contained within this environment, and inaccessible outside of it.
Exiting the virtual environment is easy. You just run:
And there you go. Back to normal.
Saving and Copying Dependencies
Newly created virtual environments start out empty, with nothing installed except for pip and setuptools. But what if you want to copy the packages used by one virtual environment to another? This is necessary for collaboration — there has to be some way for collaborators to see which dependencies the project requires and install them.
The most basic means of accomplishing this is to generate a file called
requirements.txt. But to see it in action, we have to install something. Let’s do that, first reactivating our virtual environment like so:
pip install flask
Flask is a lightweight web framework (akin to Sinatra, for those coming from Ruby). It has some additional dependencies which will also be installed alongside it. You can view those dependencies by running:
This should output something that looks like this:
You can now save this into a
requirements.txt file like so:
pip freeze > requirements.txt
cat that file (or just open it), you should see that it has the same contents shown above. You can use this file to install the packages listed therein into a new virtual environment. But first, you’ll have to deactivate the current virtual environment and create a new one:
python3 -m venv my-second-environment
Then you can source the new virtual environment, and install all the packages listed in
pip install -r requirements.txt
There we go! Now
my-second-environment should contain all the same dependencies as
my-first-environment. You can confirm this by running:
Making Life Easier With Pipenv
requirements.txt files because all that functionality is handled automatically through their respective package managers. Thankfully, a tool has come along which provides a more bundler/npm-like experience for Python: pipenv.
Pipenv is a third-party package, so first you’ll need to install it:
pip install pipenv
Then you can change into your product directory (or create a new directory if you’re starting a new project, as I’m doing here), and instantiate pipenv there:
mkdir ~/dev/projects/my-new-project && cd ~/dev/projects/my-new-project
This will create two new files, a
Pipfile and a
Pipfile.lock. These function similarly to Ruby’s
Gemfile.lock, so if you’re familiar with those you should feel right at home. It’ll also automatically create a new virtual environment for the project, which you can locate with:
The Pipfile will be automatically updated whenever you install a new package, so you no longer need to worry about generating or updating your
requirements.txt file. When you share the project with someone else, all they need to do (assuming they have pipenv installed) is run:
Likewise, adding new packages to your project is simple. Let’s install flask, as we did before:
pipenv install flask
To show which packages you have installed (and the dependencies of those dependencies) you can run:
Which should output something like this:
Uninstalling is also a breeze, although bear in mind you’ll need to list any sub-dependencies you want to uninstall as well. In this case:
pipenv uninstall flask click itsdangerous jinja2 markupsafe werkzeug
To activate the virtual environment that pipenv automatically generated, you can run:
Now any scripts you run will have access to all the packages listed within the Pipfile. Alternatively, you can run scripts without activating the virtual environment first by running commands like so:
pipenv run python my-script.py
If this seems like a lot of typing to have to do every time you want to run something, you can alias it by adding the following line to your
alias prp="pipenv run python"
Then source it:
And now you can run scripts more succinctly:
Most of what I covered here is quite new — pipenv debuted only a little over a year ago, and venv won’t work with versions of Python older than 3.3. If you’re working with an older version of Python, you’re going to have to use virtualenv instead of env, which you can read more about here.
Accompanying virtualenv is another popular tool called virtualenvwrapper, which streamlines the use of virtualenv. I choose to focus on pipenv instead, since I prefer it, but if you’d like to learn more about virtualenvwrapper you can do so here.
Lastly, there’s Anaconda. Anaconda is a distribution of Python which focuses primarily on data science and machine learning and comes with its own package management system called conda. If you’re planning on using Anaconda and you’d like to learn more about how to use its package manager, you can read up about that here.
In this article, we'll show how to use virtual environments to create and manage separate environments for your Python…realpython.com
In this tutorial, you'll learn about virtual environments. You'll learn about the importance of using virtual…code.tutsplus.com
We are well known for our work with Ruby and Rails here at thoughtbot, but generally we always try to use the most…robots.thoughtbot.com
The module provides support for creating lightweight "virtual environments" with their own site directories, optionally…docs.python.org