Preventing Dependency Conflicts in Python
At Neighborhoods.com, one of our core engineering principles is “The Pit of Success” — the idea that work in the present can guide future efforts toward good practice by making bad practice difficult in the first place. Through this, all of our engineers contribute toward an environment where growth and learning are the status quo. In the context of development, this can be seen in conceptual practices such as our usage of decision records, and in technical practices such as automated unit testing. In this post, we will be discussing an example of the Pit of Success that tackles dependency management: a lightweight tool called pipenv-devcheck.
We commonly use Python for much of our analytical and some of our development work. Thanks to PyPi and helpful libraries such as pip and setuptools, Python is fairly good at managing dependencies — it generally can prevent you from installing things that could break your environment. With the addition of Pipenv, a dependency resolution and virtual environment management tool, this ability is amplified considerably. Because Pipenv establishes virtual environments entirely based off a Pipfile that can be stored in GitHub, development teams can rest assured that they are all working in completely identical environments. With these tools at our disposal, we can say goodbye to the ever-painful and mysterious errors that arise from two developers unknowingly using different dependency versions.
What about situations where code is being crafted for use by others, though? Required dependencies for Python packages are automatically installed by pip, specified in a separate file from where Pipenv dependencies are specified. Here, opportunities for discrepancies arise: When a developer changes the the development Pipenv environment, they often must also separately modify the expected user environment to maintain compatibility. This task, while simple, is tedious and manual by nature, and as such is at risk of being forgotten.
So, what can we do to ensure that developers are setting the same expectations for end users as they are for themselves? Although the Pipfile can be stored in GitHub, it is not included in package installations, so end users cannot be expected to have access to it. Even if it was included, it would be unreasonable to expect end users to make use of a dependency management system just to use a package.
Wanting to ensure that we can avoid such issues at Neighborhoods.com, we elected to solve the problem ourselves. Pipenv-devcheck is a lightweight, open source continuous integration tool for catching mistakes in Python package environment specifications. Using native Pipenv code, abstract syntax trees, and regular expressions, pipenv-devcheck parses the developer requirements from Pipenv’s Pipfile and user requirements from the package’s setup.py file, throwing errors if it finds discrepancies between the two. It checks for both matching of the dependency names — ensuring that all dependencies present in one file are present in the other — and for version compatibility — ensuring that end users don’t end up using a version of a dependency that is incompatible with what the developer is building for.
If this tool is added to continuous integration tests, the tests will fail if either of the above checks are not satisfied. This can be used to prevent breaking changes from making their way back into the master codebase, or even as a double-check that all is well before pushing a new version release. This is a clear example of the Pit of Success: with pipenv-devcheck, it directly becomes more difficult to deploy production code with an invalid dependency structure. Instead of consequences resulting from not keeping dependencies updated, a stop is placed on the project, allowing for correction.
In an ideal world, mistakes like dependency mismatches would never happen, and automatic checks for them would be unnecessary. Every developer would double check that they updated their dependencies uniformly across a project, every time. As the developers reading this have likely experienced, however, the reality of writing code is often not so ideal, especially in the professional world. For whatever reason — impending deadlines, the excitement of writing new code, or just old-fashioned forgetfulness — small but important development tasks can relatively easily be neglected, eventually building into a problem that is greater than the sum of its parts. Pipenv-devcheck is a small, simple tool to remove such concerns from your development life.
You can view the project’s homepage here or browse its source code here.
