Setting up Python environments for your project
A Python environment is a context in which your program runs. It usually features an interpreter and any number of installed packages/libraries. When we download and install Python on our computer, it is usually installed system-wide and we get a global Python environment.
In practice, we probably will have multiple python projects we are developing on our computers. Each one of these projects will have packages/libraries which we need to do our work. These are our project’s dependencies. Most of the time, we will install these packages using pip, which downloads them from the Python Package Index (PyPI) and installs them locally into your /site-packages directory. Sometimes the dependencies of two projects may conflict. For example, Project A might require pandas version ≤ 0.9 while Project B required pandas version==1.2. This can cause problems.
These issues get worse when we have multiple versions of Python installed. There’s even a famous comic about this issue. 😄
To avoid corruption of our global python environment, we need to isolate the environments we use for each of our projects. Thankfully, there are a couple of solutions for this.
Virtual Environments
Virtual environments are python environments that have their interpreter, packages, and scripts installed in isolation from other virtual environments as well as from all global python environments.
To create a virtual environment, first, change the directory in your terminal to the folder in which you want to create your environment. Then run either of the following commands
# Unix/MacOs
python3 -m venv env# Windows
python -m venv env
Note that this command will create a virtual environment using the version of Python in your global environment. You can check which version of python is in your global environment by running the following command before activating any environment
python --versionAfter running the command to create your virtual environment, you should see an env/ folder show up in the directory you are working in. This folder contains the python installation for your new virtual environment. It is named env because we specified env in our command earlier. We can equally use our project name as follows
# Unix/MacOs
python3 -m venv myproject# Windows
python -m venv myproject
This will create a myproject folder in our directory.
Next, we have to activate our newly created environment. Activating an environment allows us to install or use packages in the environment.
# Unix/MacOs
source env/bin/activate# Windows
.\env\Scripts\activate
Once this is done, we can begin work in our environment. As long as our environment is activated, pip will install all packages into our specific environment. You can confirm that your environment is activated by checking the location of your python interpreter
# Unix/MacOs
which python# Windows
where python
This should show you a path to the python executable in our env folder.
Now that we are all set up, we can start to install packages into our environment using pip
pip install TransformersIf you would like to deactivate your environment at any time, simply run the deactivate command in your terminal
deactivateConda Environments
Like virtual environments, conda environments allow us to isolate the python interpreter and installed packages. However, there is one distinction. Conda treats Python itself as a dependency in its environment. This is because Conda also allows us to manage packages from other languages than Python.
To create conda environments, we must install one of the following:
- Anaconda (Installing on Windows, Installing on MacOS)
- Miniconda (Installing on Windows, Installing on MacOS)
To understand the difference between these two, see the image below.
To create an environment, run the following command in your terminal
conda create --name <your_environment_name>This will create an empty environment named as you specified. This environment is empty and does not contain even python. We can ensure Python is installed by running the following command
conda create --name <your_enviroment_name> pythonThis will create an environment using the same Python version you used when you downloaded and installed Anaconda. If you would like to use a different version of Python, say Python 3.8, run your command as follows
conda create --name <your_environment_name> python=3.8You can check that your environment can be created by listing all existing conda environments using
conda env list This will show all conda environments that exist and then asterisk (*) the name of the active environment. It is a good idea to choose an informative environment name when working with conda. Unlike virtual environments, where we can get away with naming all our environments env. This is because all conda environments are created in the same folder, not in our project folder!
To activate our newly created conda environment, run the following command
conda activate <your_environment_name>Now we can start installing packages into our environment. As well as being an environment manager, conda is also a package manager. This means we can install packages using conda as follows
conda install numpy=<insert_version>We can also use pip to install packages into our conda environment
pip install numpy==<insert_version>If a package can be installed using both pip and conda, it is advisable to install using conda.
Lastly, to deactivate your environment, run:
conda deactivateConclusion
There, we have it. We have explored the motivation for isolated environments and looked at two ways we can achieve this. There are still some other interesting tools used in this area of environment management: virtualenv, pyenv, poetry created to address different concerns. You can have fun exploring them and learning when it is best to use each one. :)
