Handle non-deterministic tests in Intern.js

This short tutorial is about how to separate certain end-to-end or functional tests from the others, that are non-deterministic.

Sometimes it happens, that we have non-deterministic test cases in our functional tests or e2e tests. These really evil tests can do a lot of harm, they can actually make us stop trusting in our whole test suit.

In this cases it makes sense to separate non-deterministic tests from the deterministic ones, e.g create a Quarantine for them.

But how to do it in Intern.js? (Intern.js is a test runner for Javascript)

Let’s say we add the string “[quarantine]” to all of our test descriptions, that we would like to put to quarantine. It would look like:

describe('...', () => {
    it('is a good, deterministic test case', () => {...});
    it('[quarantine] is a non-deterministic one', () => {...});
});

For this to work, we have to use the “grep” option in the Intern.js config.

const argv = require('yargs').argv;
// By default, only runs tests that have no "[quarantine]"
// in their names
const grep = new RegExp('^(?!.*?\\[quarantine\\])');
// Runs the tests in the quarantine only
if (args.quarantine) {
    grep = /\[quarantine\]/;
}
define({
    <...>, // other options
    grep: grep,
});

I think it is a good approach to always keep your functional tests less brittle and to keep trust.

It is also good, to make some rules about how you are handling your tests in the quarantine:
* add a limit of how many tests can be in the quarantine (e.g. max. 3)
* set a maximum time until a test can be in the quarantine (e.g. 1 week)

Special thanks to David Walsh, I have created the Regex based on his article:
https://davidwalsh.name/javascript-regex-string