Detecting outdated Python dependencies in pull requests
At Meilleurs Agents, we have several backend applications, mostly written in Python. With the launch of new products, we now have more and more applications to maintain.
For most of our projects, we use pipenv to manage dependencies and to produce deterministic builds. In order to monitor the state of our dependencies in the pipenv lockfile (outdated dependencies, securities vulnerabilities…), we wanted to add an automatic process to alert developers of the state of an application dependencies.
The goal for developers is to to be able to quickly check if a project requires updates dependencies-wise.
Trying Dependabot
First, we tried Dependabot, as it offered dependencies scanning and opened the associated PRs automatically.
However, we soon realized it didn’t suit our workflow :
- As it opens one PR per update, it triggered a lot of builds on our CI system. On each merge to our main branch, it would trigger another build and a deploy to our staging environment. When there is a lot of small updates, it would be easier to just have one small branch to push and deploy in one go.
- To do this, we then configured it to do PRs targeting a special dependencies branch. This means we could group the updates in this branch and deploy them to the main branch when we want more easily. However, it also means the branch has to be regularly rebased (manually) against the main branch. Then a developer needs to take time for merging and deploying the change.
With both configurations, we found out that developers didn’t really interact with Dependabot PRs. Developers would often continue to do the dependencies updates manually, as the Dependabot PRs needed manual actions anyway.
However, we still wanted to find a way to get alerts on the state of our dependencies. How to know when a new version of a major dependency is released? How to know if an app is starting to get really outdated, and may need some maintenance work?
Including a dependency check in the development workflow
Currently, we are testing a new and simpler solution, without automatic PRs. A Github Action adds a comment on all PRs to report the state of the dependencies.
The Github Action is called deps-report. It parses the lockfile (in our case, Pipfile.lock files) of a project to see which dependencies are used. It then queries the repository specified by using the Simple Repository API (PEP 503) to get the latest version. Thanks to safety-db , it also provides warning in cases of vulnerabilities, even though Github provides a similar function.
Instead of being fully automated like Dependabot, the developers see by themselves if they can include some dependencies upgrades in their PRs. If they choose not to upgrade the dependencies now, they will still see the message on their next PRs. This put the spotlight on the state of the dependencies of the app. Maybe if a developer see that the number of outdated ones starts raising, they will start taking actions to bump the dependencies :)
As the tool is just parsing the lockfile, it doesn’t need to be in a Python environment where the application package is installed, unlike pip list— outdated. It can also be executed locally, to generate reports for example.
It has proven useful for dependencies that are still in a 0.x version without a stable API, like flask-smorest: we can see when a new version was released and quickly adapt to the changes.
