The amount of packages available in Python can be quite overwhelming. Photo by chuttersnap on Unsplash

Python Environments and Packages for Beginners

Python for Matlab Users, Part 4: Environments and Packages

This is part 5 in a series on Python for Matlab users. In Part 1 I talked about about why one might use Python over Matlab, part 2 dealt with installing Python, and part 3 discussed which IDE to use.

Remember all problems you’ve had throughout the years regarding developers having different Matlab versions:

You’ve made this great script that you push to Git, but suddenly your colleague calls you up and asks why you pushed code that doesn’t work? You tell her that it works on your computer, and after a 30-minute Teams meeting with screen-sharing, you realise it’s because you used a classifier from the Classification Learner toolbox which is not available in her 2018b Matlab version, while you are rocking the newer 2020a version.

The bad news is that problems like this also occur in Python. The good news is that it’s much easier to handle than in Matlab, since installing a different version of Python is much easier and takes up less space, than having different versions of Matlab on the same PC.

Having installed Conda as described in Step 2, the easiest way to overcome this is to use Conda environments.

What are modules and packages?

A key feature of Python is the use of modules and packages. The reason why we prefer using these is that we don’t have to have a huge installation of all possible features (hint, hint, Matlab) lying on our PC: Instead, we can add features when needed (by importing modules and packages), and only when needed. It also allows for easier organisation of our code, and for re-using features in other contexts.

Modules:
In programming, “ a module is a piece of software that has a specific functionality“ [1]. The Python language itself is quite bare-bone, meaning one would have to import modules to get additional functionalities.
One such module could be the random module, which implements a pseudo-random number generator.

Packages:
Packages are namespaces (or libraries) which contain multiple packages and modules themselves. One such example is the numpy package, which provides support for arrays and matrices and mathematical functions for manipulating them (similar to what is done in Matlab).

Conda Environments

Despite the name, Conda environments don’t have anything to do with global warming. Photo by John O’Nolan on Unsplash

Environments is a feature in Conda which doesn’t really exists in Matlab. But it can prove very useful when working on different project which require different packages (or different versions of packages). So instead of having one version of Python installed with all the necessary packages (as in Matlab), you can make an environment for each project you work on, which would only contain the packages needed for that project. This also helps making the code more portable.

There are a few alternatives to these Conda environments, e.g. Pipenv and VirtualEnv. However, Conda is the best option as it provides a much simpler interface [2]

The following list is based on the great guide made by [3].

1. List available environments

You can list the all available Conda environments on your machine as follows:

conda info --env

2. Create a new environment

You can create a new blank environment called <env_name> with python version x.x very easily in Conda. (To se a list of available python versions first, type conda search "python" or conda search "^python$" and press enter.)

conda create -n <env_name> python=x.x

You can also create a new environment by cloning an existing environment (e.g. base). This is useful if you want to build upon an existing environment instead of starting from scratch.

conda create --name <env_name> --clone base

Another option is to make an environment with the packages included in the default Anaconda installation:

conda create --name <env_name> anaconda python=x.x

Finally, you can also create an environment with a predefined set of packages, e.g. numpy and scipy:

conda create -n <env_name> numpy scipy python=x.x

3. Activate an environment

You can activate an environment by typing the following, where <env_name> is the name of your environment, (e.g. base as done above).

conda activate <env_name>

When activating an environment, you automatically move to that environment with that environments Python version and packages available.

In the environment, you can start Python by typing python in the terminal, just as in the base environment, but you will now be using the environments local python instead of the global base python.

NOTE: If you get an error regarding pip or Python in your environment and the error message is something like “cannot find Python”, then it might be because you have non-ASCII characters in you user folder, which will make it impossible to find any of the packages installed in your environment. To fix this, the only option is to rename you user folder, by following the steps described here. Or just re-install Anaconda somewhere else.

4. Install additional packages in an environment

To see which packages are available in a certain environment with following command (this works with any environment created with Conda):

conda list -n <env_name>

To install a new package in an environment, you type one of the following, depending on whether you want the latest or a certain version (version x.x ) of the package :

conda install -n <env_name> <package_name>
conda install -n <env_name> <package_name>=x.x

5. Exporting an environment

One of the great things about environments is that they can be exported for others (or yourself) to easily get the same setup as you have, with exactly the same Python version and exactly the same packages. You export an environment to an yml-file like this:

conda env export --name <env_name> > envname.yml

This can then be imported on the other PC by simply creating a new env using the yml file.

conda env create -f envname.yml

6. Deactivate an environment

To end a session in the current environment, enter the following. Note that it is not necessary to specify the environment name — whichever is currently active will be deactivated, and the shell variables will be returned to normal.

conda deactivate

7. Delete a no longer needed virtual environment

To delete a Conda environment, enter the following, where <env_name> is the name of the environment you wish to delete.

conda remove --name <env_name> -all
conda env remove -n <env_name>

Installing packages manually

Photo by Zoya Loonohod on Unsplash

You can install additional packages in Python in two ways: Using conda and using pip, which is a package manager commonly used among Python users.

Pip vs Conda is a huge discussion topic in itself, as both are ways of providing packages to your python installation. But Jake VanderPlas summarizes the difference quite well in his blog [4]:

Conda and pip are different projects with different intended audiences: pip installs python packages within any environment; conda installs any package within conda

The difference is also summarized in the table below:

A comparison of conda and pip taken from https://www.anaconda.com/understanding-conda-and-pip/

So, what does this mean for you? Well, it’s actually quite simple:

You should always use Conda whenever possible.

The reason for this is that the packages installed using Conda can be re-used across environments, whereas pip modules are installed globally (if in the base environment) or in the local environment. Also, if pip is used to install software into a conda environment, conda will be unaware of these changes and may make modifications that would break the environment. Therefore, the more packages that can be kept in conda, the simpler and better.

Notice that I said whenever possible in the above sentence. This is because some packages might not be available in Conda, as the number of packages available with pip is much bigger: Conda contains ~1500 packages while pip has over 150000 [5].

So, you should install a package by trying conda install first, and fall back to pip install if the package isn’t part of Anaconda.

There are some other, more technical reasons for using Conda as described by e.g. [6]. See [5] for more info on using pip in a Conda environment

Using Conda

Installing packages in Conda is done by simply typing

conda install <package_name>

Using pip in an environment

To use pip, you must first install pip in your environment using Conda.

conda install -n <env_name> pip
conda activate <env_name>

Then, you can install packages using pip:

pip install <package_name>

In the next part (coming soon!) we will look at how to make numerical simulations using Numpy.

References

[1] https://www.learnpython.org/en/Modules_and_Packages

[2] https://stackoverflow.com/questions/34398676/does-conda-replace-the-need-for-virtualenv

[3] https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/

[4] https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/

[5] More on using pip in Anaconda: https://www.anaconda.com/understanding-conda-and-pip/

[6] https://stackoverflow.com/questions/20994716/what-is-the-difference-between-pip-and-conda

More on environments: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html