Managing Python packages using the requirements.txt file
One of the most important things to accomplish when writing code is to write code that is able to live after you. I use the phrase “able to live after you” because your code can be the backbone of someone’s project years after you have departed. Your code can also be accessible to others way after you have left the institution you are currently at, and it is no longer easy to contact you to understand your thought process. To promote the readability and reproducibility of our work, we should all strive to effectively comment our code and highlight dependencies whenever we are developing code that has dependencies.
When programming in python, especially for machine learning, you may use many packages beyond the python standard library. However, when you distribute your code, other individuals may not have a similar environment with the same packages as yours. Consequently, when they try to run your code, it might fail. The screenshot below shows a jupyter notebook that depends on the matplotlib package. The notebook clearly states that the module has not been found in the environment.
Of course, one can follow through the error messages and install each of the required packages; however, as projects grow, they tend to have many dependencies, and those dependencies may have dependencies as well. For that reason, relying on the command line error messages to install the right packages is usually inefficient. Instead, there should be a better way for someone to create the same python environment the developer of the code had when they developed the shared code. To do this, an end-user may create a virtual environment and download all required packages to run the code in this environment.
Typically, the file that has information on all dependencies is called the requirements.txt file. One can create this file manually by adding all the packages the code is dependent on; however, this is not always the case.
In the event that one has already written their code but would like to create a requirements.txt file, there are many ways of doing this. In this tutorial, we will highlight a few.
Creating the requirements.txt file
The typical way assumes you have pip installed on your machine, and you download your python packages using pip install. In that case, you would type the following on the command line:
$ pip freeze > requirements.txt
This should create the requirements file for you; however, it might include other packages you might not necessarily need to run your program. This is so because pip freeze saves all packages in the environment, not just those used in your current project. This is not much of an issue. You can always open your requirements.txt file and manually remove any extra packages. The code below shows a sample requirements.txt file gotten through pip freeze > requirements.txt
One thing many people have noticed with some pip packages is that they put @ file:///* after some package name — were * as a wildcard character. Having @ file:///* after the package name instead of the package version is not good formatting for the requirements.txt file. It is an open issue identified in versions of pip such as 20.1 and 21.0.1.
To avoid the @ file:///*, one needs to type:
$ pip list — format=freeze > requirements.txt
instead of pip freeze > requirements.txt
Again, the output requirements.txt file may have some extra packages, and you can always go into the file and manually remove them. The code below shows a modified requirements.txt file gotten through . It no longer has @ file:///*
Specifying package versions
When creating the requirements file, it is important to understand how you can specify the package version. If you create the requirements.txt file using the method above, you will see that a package version follows each package name, and it is written as follows:
package_name==package_version
== stands for equal to; hence, the stipulated package version will be required for the given package. This might cause problems later since certain versions of software might be discontinued. In cases where your code requires an exact version for a given package, the end-user might not be able to run the code when that specific version is no longer available. In light of this, there are more flexible package specifications such as:
~= requires downloading any version compatible with the stipulated version
>= requires downloading any version greater than or equal to the stipulated version
!= the downloaded package should not equal the stipulated package version
You can certainly play around with these things to enhance your understanding of the requirements.txt file. Furthermore, to allow others to run your python projects with ease, we encourage you to include a requirements.txt file whenever you share your code. You might also need this file yourself when your python environment changes.
Installing dependencies
Now let’s assume that the requirements.txt file has been created, or perhaps it’s you who has just downloaded another person’s code, and you are now wondering how to install all the dependencies in the requirements.txt file. Well that is very easy. We encourage creating a virtual environment in any way you want. Many people use conda, and those who do not use conda can use the command line directly. That could be something like:
$ python -m venv myvenv
With the above command, one creates a virtual environment called myvenv
Once you have created the virtual environment, you can activate it by typing the following on the command line:
$ source ./myvenv/bin/activate
Whoa!!! You now have an active virtual environment. Note that the above steps can be done differently if you are using conda or another framework to create a virtual environment. Now, let’s imagine you have your virtual environment and have activated it. In order to install dependencies in the requirements.txt file, please type the following on the command line:
$ pip install -r requirements.txt
Eureka! You should now be able to run the downloaded code without any hiccups.
