Save Thousands of Debugging Hours by flagging flakey tests
FlakeGuard (Open Source Product)
FlakeGuard identifies tests that pass one time, and may fail another time, despite not changing any written code.
Curious about what flaky tests are — checkout this article, before diving into the benefits of utilizing FlakeGuard!
What is FlakeGuard?
- FlakeGuard is a free, open-source tool that allows developers to run Jest tests to automatically detect and report flaky tests in the development process.
- Flaky tests can be flagged, and viewed visually through a variety of charts and metrics, see here.
Why FlakeGuard?
How to use FlakeGuard?
Installation
On your terminal run: npm i flake-guard --save-dev
Run FlakeGuard
To run FlakeGuard: npx flake-guard <filename>with the name of the test file that you want to examine.
FlakeGuard will analyze your E2E tests for flakiness by executing multiple test runs and analyzing the results.
Once all tests have run successfully, simply press enter in the terminal to be redirected to flake-guard’s website.
Configure
The default number of test runs is 10, but this can be adjusted as described below.
To adjust FlakeGuard configuration variables, you can create a file in the root directory of your project calledfg.config.json. Below are the defaults, which can be overridden in your local 'fg.config.json' file.{ runs: 10 }
For example, if you want to increase accuracy, you can increase test runs.
{ runs: 100 }
In general, there is a time versus accuracy tradeoff. More test executions increase accuracy but reduce speed.
How FlakeGuard Works — “Under the hood”:
The flake-guard npm package is automating repeated runs of your test file. It will do a basic parsing of the results locally to log an object in your terminal with all of your test assertions and their pass/fail metrics. It sends off the raw Jest results objects to the FlakeGuard server for further analysis which you can view at
FlakeGuard Web Application:
View analytics including, but not limited to:
* Passing vs failing tests with the color coordinated flake severity table
*Analyze Runtime metrics & API response times
*See flaky test flags
*View pointers showing fail points within test code
*Running test order permutation
*Display flake improvement over time
Other articles:
Flaky Tests — How to Overcome Testing Nightmares
