Test Impact Analysis Based Python Test Execution
The missing tool in the Python CI/CD toolkit.
Intro — Why should I care about Test Impact Analysis based testing?
In 2018/2019 I worked for the company creating a well known open source IT infrastructure monitoring software called CheckMK. The company Matthias Kettern GmbH was founded by Matthias Kettner and renamed into tribe29 during this time.
Most of my time I’ve spend maintaining network switch and server monitoring components, called checks. A check extends the systems core functionality with beeing able to monitor additional hardware and/or software. In case of a network switch this is e.g. a switch device and the associated firmware running on it. However I’ve written some software-only checks as well. Each check is a collection of files containing the business logic and the WebUI integration part in one or several Python files whose functionality is integrated into CheckMK using a custom plugin system. Back then there have been a lot of checks… if I remember correctly it were between 1000 and 1300 checks at this time and the number of checks continuously increasing. With maintaining I mean understanding network switch specific and server specific source code logic (not straightforward without Python type hints and without documentation), refactoring code to remove dublication (which requires to have a unit test safety net first of course), fixing bugs (which have been reported before in a quite long list of “known bugs”, there have been little automated tests back then), reviewing source code changes of peers and extending the functionality (the really fun part). Some of my peers have been busy with migrating Python2 to Python3 and adding type hints to the code base. Luckily another Senior Developer added functionality to speed up regression test creation with a “snapshot” approach. The functionality is similar to e.g. how snapshottest speeds up Django API testing. This helped us a lot and enabled us to shrink down the quite extensive list of known bugs down to ZERO in a short time. When I think back on it, we achieved a lot, especially when you consider that we were fewer developers than are employed there today. To be honest, back then the CI/CD environment was stepmotherly treated. The CI/CD environment was existing, there have been some tests and a DevOp began to put significant effort into it. I’m quite sure the CI/CD environment is significantly better today. Back then CheckMK was a monolithic application residing in a single source code versioning repository. The code of the core system was structured very good. After changing or adding features testing on the local machine was no issue by running tests specific to a single check with pytest only. We used a review system to block commits beeing commited into the public GitHub repo until they’ve been reviewed. The more tests we added the more time it took between pushing source code changes to the review repo and seeing the changes in the public GitHub repo. It felt like taking forever. Not because of the reviews but because of the tests taking a very long time cause of the huge repository and given the fact that Python can be slow. This was quite frustrating because we’ve been little people beeing very effective in doing our job (adding tests, fixing bugs, adding features) but at the same time very slow in bringing these source code changes into releases.
I thought about how one could speedup the tests in the CI/CD environment as one would do on the local machine… by selecting and running only the source code which is affected by changes. I’ve known the concept of Test Impact Analysis (TIA) based testing already while working as an Embedded Software Developer for a big robot manufacturing company and setting up and using a TIA capable proprietary software.
What is test impact analysis based testing?
From a high level perspective it’s pretty simple.
“Test Impact Analysis (TIA) is a modern way of speeding up the test automation phase of a build.” — Paul Hammant
… by executing only tests which relate to the production source code changes w.r.t. the “most recent” source code changes.
For a detailed intro into Test Impact Analysis I recommend e.g. the blog post Martin Fowler’s The Rise of Test Impact Analysis.
Do I need Test Impact Analysis based testing?
The question of whether or not TIA based test execution could be interesting for you can be ansered by asking yourself the following questions. Am I blocked by slow test execution in the remote CI/CD environment? If you are working with big monolithic applications chances are high that you’ll be suffering from this. Over the long run the developer satisfaction w.r.t. development workflow is critical. If you are frustrated chances are high that you are less productive. Do I want to save cost by decreasing CI pipeline execution time? Source code hosting providers like GitHub, GitLab, etc. usually provide limited CI pipeline execution quota per month. Do I want to reduce my companies carbon footprint? This sustainability aspect is probably not that obvious in the first place. But if you are honest you can imagine how much energy is wasted for executing tests which will never find a bug. Everyone of us should try to reduce the own carbon footprint as much as we can.
“80% of faulty builds discovered in 1% test time. 90% of faulty builds discovered in 2% test time.” — Dr. Elmar Juergens (Accelerate 2018 — Test Impact Analysis: How to find new bugs 100x faster)
Packages for CI/CD compatible Test Impact Analysis based testing?
I searched for Python packages capable of providing TIA based testing. I’ve looked at the following packages (ordered aphabetically):
- nose-knows
- pytest-incremental
- pytest-knows
- pytest-picked
- pytest-smartcollect
- pytest-testmon
- smother
- samplemod aka “testimpact” script
- watchdog
- when-changed
A more detailed comparison of packages which have been available back then can be found here. Some of these packages were written by really good developers. However in summary all packages available have been uinsuitable. There were a few points which disqualified the packages for usage: All packages addressed the use case of local test execution only. Some of the packages quite good. But there was no single package suitable for use in a CI/CD environment. All packages depend on a mechanism to detect source code changes. Common approaches are to watch the filesystem for changes of the file meta-data (point in time of last modified) or using version control system (VCS) meta-data (which files have been modified w.r.t. the previous commit). The VCS based packages could have been extended to support exeuction in the scope of a CI/CD environment. However there was a lack of interest of the package maintainers to extend the packages functionality to the CI/CD scope. In addition some packages have been unsuitable due to common package evaluation metrics (number of maintainers, number of contributers, number of releases, last release, etc.).
Consequently I decided to start my own package as part time project.
The Test Impact Analysis based test preprocessor: python-tia
The first important design decision was to use VCS meta-data for source code change detection. I had some short conversations with Ned Batschelder, an edX Senior Software Architect and maintainer of Coverage.py. Coverage.py is the most famous Python coverage analysis tool out there. I wanted to use a database (in the first place SQlite) to persist meta-data of which tests execute what production code. As I remember correctly I also had some short e-mail conversation with Ned which is gone unfortunatelly. The result was that Ned was so kind to finally add the “Who tests what?” feature to coverage.py. This would enable to extend the TIA capabilities from the file level into the source code level: If you can only identify file changes you have to execute all the tests which relate to the overall production code in a file. This requires to beeing able to lookup which production code relates to which tests of course. This is the information which is provided by the Coverage.py “Who tests what” feature. If there are many classes/functions in a file (as in the case with CheckMK) a lot more tests need to be executed as needed of course. If you identify not only file changes but their source code level scopes as well using more VCS meta-data (you get the changed file and you can identify the changed production code classes/functions) the granularity of TIA based testing can be optimized.
I recognized that the concept of file level change impact is not only interesting w.r.t. test tool execution only. Analyzers which do not depend on the overall scope of the source code can benefit of “change scoped” execution as well. Of course the time saving is significantly less because analyzers usually execute magnitudes faster than tests. However I liked the idea of creating a package for “Test (or better Change) Impact Analysis based test tool preprocessor”. The project python-tia (GitHub project, documentation website.) was born.
Call to push the topic further
As usual in life things change unpredictably. I was time to leave tribe29 and continue with a journey involving less software maintenance and more system architecture, software architecture and “greenfeelding” :) . Test execution time was no blocker anymore and consequently motivation for continuing python-tia little. BTW: The product CheckMK is great. There will be a major new release CheckMK 2.0 (Python3.8, totally new UI design, etc.) which supports over 1800 checks coming soon.
However in case you suffer from slow CI/CD workflows due to many tests and long test execution times I highly recommend to get into this topic. Probably someone takes over the project or concepts of the python-tia project. I’ve put little effort into implementation but quite some time into design (e.g. documented here) and exploratory coding using Jupyter Notebooks. Probably there are still some notes flying around somewhere which I’d love to hand over to someone continuing this journey. Feel free to write me.
As far as I know there is still no package for CI/CD compatiable Test Impact Analysis based test execution for Python out there.
Conclusion
Test Impact Analysis is a incredible effective measure to reduce test execution time. Python is an easy language which makes it comparably easy to design and implement Test Impact Analysis based test execution. You’ll get the most test execution time savings when using version control system meta-data for identifying file changes, version control system + source code parsing based identification of production source code level changes within a single file (classes, functions, etc.) as well as code coverage information for mapping production source code level changes to test classes/functions.
