Pipenv: A tool all Python developers should be using
This post is about my first experience from a blank canvas to production ready code using pipenv as the package manager plus testing using its virtual environment.
I would like to start with an analogy. I want you to imagine you are working in a restaurant as a waiter, and the chef says “today we will be serving the components of this meal on individual plates”.
The more complex the meal the more plates you will have to carry to the customers table.
You hear the bell, service is up, but the components of the meal are coming out at different times, as well as different tables ordering the same meals. Furthermore, you have to make sure that it gets to the customer correctly with the customers preferences (e.g. steak medium rare) satisfied…sounds like a nightmare right?
Well if you imagine the components of the meal as Python packages, the different tables as virtual environments and the preferences as specific versions of the dependencies then we can start see how similar these scenarios can be. Even if the meal “resolved”, would you leave a tip?
As Python developers we should focus on exactly that…developing in Python.
Scala has SBT, Ruby has RubyGems, C# has NuGet, the list goes on.
Of course we can use pip with virtualenv/pyenv, and they are great tools, but they can be fiddly especially if you have no experience of using them.
And wouldn’t it be nice just to change directory into different projects and have the virtualenv ready to go without having to activate the correct one for that project?
“Starting a pipenv project is as easy seeing off a whole packet of Tunnock’s Caramel Wafers” — Alex Varatharajah
Go to a terminal and make a new directory for your project:
Then do a pip install pipenv then pipenv install pandas --python 3.6 and within seconds you will have created a project with a Python 3.6 virtual environment ready to start working on.
You can access your virtual environment directly using pipenv shell:
Or you can run a program in your virtual environment using pipenv run <your program>
Pipenv creates Pipfile and Pipfile.lock file, replacing requirements.txt, and this is where the dependencies are managed. You’ll want to check your Pipfile.lock into version control for deterministic builds, while using the Pipfile for abstract development.
When installing your dependencies in your virtual environment pipenv install -d --skip-lock (for local development)
and pipenv install --ignore-pipefile (for production) will be your friends.
Obviously, depending on the project you will need more than just a Pipfile and Pipfile.lock as setup.py is still required for package distribution.
During a recent feature delivery which needed good speed to market, my team decided that this was the appropriate point to test this new method.
The risk turned out to make our lives much easier in starting up and getting to MVP as soon as possible. Using this setup, with a couple more pieces thrown into the mix, we had a baby skeleton ready for our data science team to start working on.
One of the primary goals was to have a cookiecutter so we can quickly start new projects, where the whole team can start from the same point with the same dependencies. Without having to worry about being in the correct virtual environment, and keeping track of the new dependencies, as well as pinning dependencies used for the project (unlike using requirements.txt).
A great thing about Pipenv is that it creates a virtual environment that is tracked through hashing. Pipenv also supports different python versions and is compatible with Tox if you wanted to test your code on multiple Python versions.
Pytest then becomes as easy as typing pipenv run pytest from your project directory and it will run your tests in a virtual environment with all the dependencies you installed.
How many times have you run a program for you to see an error E ModuleNotFoundError: No module named '<x>’for you to then check that you are in the right virtual environment or need to install the module etc.
Time could be better spent.
With that said, if you do decide to change the name and structure of folders in the project you will need to remove your virtualenv. This is because the hash is created using the folder and naming structure (hence why pipenv can always locate the virtualenv for a project).
To do this simply use pipenv –-rm:
You will then have to do another pipenv install to create a new virtualenv, you can also do this if you just want to start fresh.
In production you will still want to use pinned dependencies, using Pipenv’s Pipfile.lock pipenv install –ignore-pipfile. This will only install using the working dependencies locked previously from your development process.
Having seen how productive it makes us, we have begun the process of moving our other projects over to Pipenv.
In actual fact pipenv solved a buggy error we were having with pytest-cov, where there were dependency resolution issues between modules in one of our projects that uses an ONZO internal PyPI repo.
The transition has been relatively easy, after setting the dependencies in a Pipfile and creating a Pipfile.lock, we have a Dockerfile that runs Pipenv in our Kubernetes containers as well as a rakefile to run our tests using Pipenv for Pytest and get reports using Pylint, Flake8 and Sonar.
I’ve created a Cookiecutter (which uses Jinja for templating projects), which we will use for all Python based projects we take on in the future.
The project is ongoing and is an open-source data science/R&D starter kit for development, which includes Pipenv, Pandas, and can run Jupyter notebooks. It also has some Postgres test infrastructure with an initial test schema and table that can be used in your tests.
You can find it here: https://github.com/onzo-com/python-cookie-cutter-public
