Testing with Deno — Part 2 Advanced

This is the second part of the testing with Deno series. The first part is here.

Introduction

In the first part of the testing series, we have looked at the basics of Deno’s test runner, including some simple unit tests. Deno’s test runner is more than that. In the simple form, it runs all the tests. There are some advanced features to control the execution of tests to selectively include or exclude tests matching a criterion.

The advanced features are:

• Parallel execution: Tests can run in parallel to save CI/CD time
• Ignore: Certain tests can be ignored (commonly based on environment settings)
• Include: Certain tests can be included (commonly based on the type of testing being done)
• Filter: Run tests whose name matches the filter criteria (commonly used to run a group of related tests)

In this article, we’ll go over the advanced features in detail. For each feature, we’ll also look at some examples.

Module under test

In the first part of this series, we had written a small module a.ts that generates a given number of concatenated UUIDs. The number was capped at 5. We’ll modify that example to allow bigger numbers so that the test execution would take time.

Here is the module under test a.ts that allows up to 1M UUIDs.

import {generate} from "https://deno.land/std/uuid/v4.ts";
export function getLongUuid(len: number): string {
    if(len===0 || len>1000000)
        throw new Deno.errors.InvalidData('len must be between 1 and 100000');
    
    let uuid:string="";
    for(let i=0; i<len; i++)
        uuid+=`${generate()}-`;
    return uuid.slice(0, -1);
}

Parallel Execution

By default, the test runner queues all the test files/suites and executes them one-by-one. This gets slow when the test suite is big. The result is increased CI/CD time.

The test runner has an option to run multiple test files (also called suites) in parallel using test workers. It’s important to note that the test runner doesn’t go inside a file and run the tests cases in parallel. Instead, it runs the suites in parallel.

To run multiple test suites in parallel using workers, a parameter called jobs can be provided to the test command:

deno test --jobs 3

The above command would run up to 3 test suites in parallel. If there are more test suites, they’d be queued and then assigned to the workers as they free up.

Example

The implementation of parallel execution is very simple with a single option to control the number of workers. Now, let’s go over some examples and understand it.

First, let’s run a single test suite (a_test.ts) to test the module under test (a.ts):

//a_test.ts
import {getLongUuid} from "../lib/a/a.ts";
import {assert} from "https://deno.land/std/testing/asserts.ts";
const SIZE_OF_SINGLE_UUID=36,
      U100K=100000, U100K_SIZE=SIZE_OF_SINGLE_UUID*U100K+U100K-1,
      U250K=250000, U250K_SIZE=SIZE_OF_SINGLE_UUID*U250K+U250K-1,
      U500K=500000, U500K_SIZE=SIZE_OF_SINGLE_UUID*U500K+U500K-1,
      U750K=750000, U750K_SIZE=SIZE_OF_SINGLE_UUID*U750K+U750K-1,
      U1M=1000000, U1M_SIZE=SIZE_OF_SINGLE_UUID*U1M+U1M-1;
Deno.test(`len is ${U100K}`, () => {
    assert(getLongUuid(U100K).length===U100K_SIZE);
});
Deno.test(`len is ${U250K}`, () => {
    assert(getLongUuid(U250K).length===U250K_SIZE);
});
Deno.test(`len is ${U500K}`, () => {
    assert(getLongUuid(U500K).length===U500K_SIZE);
});
Deno.test(`len is ${U750K}`, () => {
    assert(getLongUuid(U750K).length===U750K_SIZE);
});
Deno.test(`len is ${U1M}`, () => {
    assert(getLongUuid(U1M).length===U1M_SIZE);
});

A single run of the above suite takes around 9.5 seconds:

deno test a_test.ts 
running 5 tests from file:///Users/mayankc/Work/source/utApp/test/a_test.ts
test len is 100000 ... ok (445ms)
test len is 250000 ... ok (908ms)
test len is 500000 ... ok (1798ms)
test len is 750000 ... ok (2681ms)
test len is 1000000 ... ok (3582ms)
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out (9596ms)

Now, we’ll copy the test suite a_test.ts to a2_test.ts (a simple copy). A sequential run of a_test.ts and a2_test.ts takes around 19 seconds.

deno test
running 5 tests from file:///Users/mayankc/Work/source/utApp/test/a2_test.ts
test len is 100000 ... ok (393ms)
test len is 250000 ... ok (907ms)
test len is 500000 ... ok (1806ms)
test len is 750000 ... ok (2714ms)
test len is 1000000 ... ok (3645ms)
running 5 tests from file:///Users/mayankc/Work/source/utApp/test/a_test.ts
test len is 100000 ... ok (396ms)
test len is 250000 ... ok (908ms)
test len is 500000 ... ok (1826ms)
test len is 750000 ... ok (2724ms)
test len is 1000000 ... ok (3636ms)
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out (19072ms)

This is where parallel execution can help! We’ll run 2 parallel jobs as there are 2 suites only. A parallel execution of a_test.ts and a2_test.ts takes around 9.7 seconds:

deno test --jobs 2
running 5 tests from file:///Users/mayankc/Work/source/utApp/test/a2_test.ts
running 5 tests from file:///Users/mayankc/Work/source/utApp/test/a_test.ts
test len is 100000 ... ok (475ms)
test len is 100000 ... ok (475ms)
test len is 250000 ... ok (898ms)
test len is 250000 ... ok (912ms)
test len is 500000 ... ok (1849ms)
test len is 500000 ... ok (1873ms)
test len is 750000 ... ok (2713ms)
test len is 750000 ... ok (2732ms)
test len is 1000000 ... ok (3623ms)
test len is 1000000 ... ok (3636ms)
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out (9760ms)

It takes almost half the time! This would significantly reduce the CI/CD time when there is a big test suite to execute.

Conditional execution

By default, the test runner would execute all the tests. However, there might be situations when only some need to execute, such as run only some cases on Linux while run different ones on Windows, etc. Or, don’t run integration tests in CI/CD. Or, run test names matching a string.

There are three ways to conditionally execute test cases:

• Ignore: Don’t execute the ones matching given condition
• Include: Only run the ones matching given condition
• Filter: Only run the ones whose name matches given filter criteria

The first two of the above conditions need to be specified in the test definition. The third one i.e. filter goes as an option with the test command. In the first two cases, test function would also go inside the definition object.

interface TestDefinition {
    fn: () => void | Promise<void>;
    name: string;
    ignore?: boolean;
    only?: boolean;
    sanitizeOps?: boolean;
    sanitizeResources?: boolean;
    sanitizeExit?: boolean;
}

Unlike earlier, when name and test function was passed as 1st and 2nd argument, this time the test definition need to be passed as a structured object that contains test name, function, and other options.

Ignore

Any test case can be ignored by using ignore condition. If the condition evaluates to true, the test would be ignored. The test runner records and reports the number of tests that have been ignored.

Here is an example of always ignoring a test:

Deno.test({ name: `len is ${U100K}`,
            ignore: true,
            fn: () => assert(getLongUuid(U100K).length===U100K_SIZE)
});

Here is an example of the same test, except that this time the test is ignored if an environment variable IGNORE_INTEGRATION_TESTS is set:

Deno.test({ name: `len is ${U100K}`,
            ignore: Deno.env.get('IGNORE_INTEGRATION_TESTS') ? true: false,
            fn: () => assert(getLongUuid(U100K).length===U100K_SIZE)
});

Here is a sample run with ignored test case:

> export IGNORE_INTEGRATION_TESTS=1
> deno test --allow-env a_test.ts 
running 5 tests from file:///Users/mayankc/Work/source/utApp/test/a_test.ts
test len is 100000 ... ignored (0ms)
test len is 250000 ... ok (935ms)
test len is 500000 ... ok (1766ms)
test len is 750000 ... ok (2667ms)
test len is 1000000 ... ok (3611ms)
test result: ok. 4 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out (9040ms)

Include

Any test case can be included using only condition. If the condition evaluates to true, then the test case would execute. The test suite would still report failure. The ‘only’ option isn’t a commonly used option.

Here is an example of always including a test (which is true even without specifying only):

Deno.test({ name: `len is ${U100K}`,
            only: true,
            fn: () => assert(getLongUuid(U100K).length===U100K_SIZE)
});
> deno test --allow-env a_test.ts 
Check file:///Users/mayankc/Work/source/utApp/test/a_test.ts
running 1 test from file:///Users/mayankc/Work/source/utApp/test/a_test.ts
test len is 100000 ... ok (396ms)
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out (424ms)
FAILED because the "only" option was used

Filter

This is an option to the test command (not in the test definition like others). This is used to run specific tests whose name matches the given filter criteria. The filter option is very useful in running a subset of tests without making any changes to the test suite.

deno test --filter "filter-string"

The test result would report the number of tests executed, passed, and filtered out. Note that filtered out is different from ignored.

Here is an example of running all the tests whose name contains 100:

deno test --filter 100 a_test.ts 
running 2 tests from file:///Users/mayankc/Work/source/utApp/test/a_test.ts
test len is 100000 ... ok (400ms)
test len is 1000000 ... ok (3666ms)
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out (4124ms)

Here is another example of running all the tests whose name contains 750:

deno test --filter 750 
running 1 test from file:///Users/mayankc/Work/source/utApp/test/a2_test.ts
test len is 750000 ... ok (2749ms)
running 1 test from file:///Users/mayankc/Work/source/utApp/test/a_test.ts
test len is 750000 ... ok (2738ms)
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 8 filtered out (5579ms)

That’s all for the second part of the testing series. There is still a lot to discuss on the test runner. Here is the next plan:

• In the first part, we’ve gone over the basics of test runner
• In the third part, we go over how to convert test results to JSON (useful for CI/CD, stats, etc.)
• In the fourth part, we go over all the asserts in detail
• In the fifth part, we go over doing integration testing with test runner (includes running a load test)