CODEX

We explore how Julia tests are organized differently from what you may be used to in other testing frameworks.

Erik Engheim

Follow

Jan 21 · 7 min read

Julia development practices will likely not be the same as what you have experienced in Java, JavaScript, Python, Ruby and other popular programming languages.

Thus in this story I will do a sort of quick recap of how testing is often done in other languages and compare that with the common practice followed in Julia as exemplified by the Julia standard library.

How to effectively run tests: Julia v1.5 Testing: Best Practices.

No Tests But Nested Test Sets

Testing in Julia is a lot more free form than what you may be used to. For contrast let me show some examples from other tests frameworks.

Pytest

Pytest is a frequently used testing framework for Python. Here one simply prefixes functions that represents tests with test_ as shown below:

# Python: Pytest

def capital_case(x):
    return x.capitalize()

def test_capital_case():
    assert capital_case('semaphore') == 'Semaphore'

In Python one would execute these tests by running:

$ pytest

Go Unit Tests

Go comes with a builtin unit testing framework. Why I am showing all these frameworks instead of jumping straight to Julia? Because these work how people are used to. And I need something to contrast with to help clarify how Julia testing is different.

// Go
package main

import "testing"

func TestSum(t *testing.T) {
    total := Sum(5, 5)
    if total != 10 {
       t.Errorf("Sum was incorrect, got: %d, want: %d.", total, 10)
    }
}

For Go tests you put them in a file which ends with _test such as foobar_test.go. Each test is placed in a function starting with Test.

$ go test

This will look for the files ending with _test.go and treat every function with a Test prefix taking a testing.T type argument as a test.

JUnit

In Java JUnit is commonly used. Each test is a method on a test class as shown below:

// Java: JUnit Tests
public class MyTests {

    @Test
    public void multiplicationOfZeroIntegersShouldReturnZero() {
        MyClass tester = new MyClass(); // MyClass is tested

        // assert statements
        assertEquals(0, tester.multiply(10, 0), "10 x 0 must be 0");
        assertEquals(0, tester.multiply(0, 10), "0 x 10 must be 0");
        assertEquals(0, tester.multiply(0, 0), "0 x 0 must be 0");
    }
}

These classes are then collected into a test suite, thus you can avoid making enormous test classes:

Java: JUnit
@RunWith(Suite.class)
@SuiteClasses({
        MyClassTest.class,
        MySecondClassTest.class })

public class AllTests {

}

Julia Test Sets

In Julia we do away with this artificial separation between test methods, test classes, test suites etc. Instead everything gets mashed into one concept called a testset. These are more flexible than what you find in other frameworks which is why developers coming from other frameworks may feel uncomfortable with the much looser form use in Julia testing.

In Julia everything is much more free form. You tailor the test framework more to your own preferences and style. This is possible because test sets can be nested.

Here is an example from my Little Man Computer (LMC) assembler.

@testset "Disassembler tests" begin
   
    @testset "Without operands" begin
        @test disassemble(901) == "INP"
        @test disassemble(902) == "OUT"
        @test disassemble(000) == "HLT"                                     
    end
    
    @testset "With operands" begin
        @test disassemble(105) == "ADD 5"
        @test disassemble(112) == "ADD 12"
        @test disassemble(243) == "SUB 43"
        @test disassemble(399) == "STA 99"
        @test disassemble(510) == "LDA 10"
        @test disassemble(600) == "BRA 0"
        @test disassemble(645) == "BRA 45"
        @test disassemble(782) == "BRZ 82"                                        
    end   
end

I have have defined test sets for each major component. So e.g. the assembler, disassembler and simulator are all represented by a different test set. Each of these test sets have sub test sets which test aspects of that component.

Thus a Julia @testset corresponds to both a test function as well as a test class, and a test suite.

What to Put in a Test Set

There are a lot of different ideas of what should be in a unit test. Many swear by to having only one specific thing being tested per test. The Julia convention for test sets is that each test set has a collection of related tests. This is best understood by simply looking at real world tests found in the Julia standard library.

The test sets I show as examples here will be shortened a bit by me as there is not particular value in showing the full length of each test.

# Julia: abstractarray.jl tests

A = rand(5,4,3)
@testset "Bounds checking" begin
    @test checkbounds(Bool, A, 1, 1, 1) == true
    @test checkbounds(Bool, A, 5, 4, 3) == true
    @test checkbounds(Bool, A, 0, 1, 1) == false
    @test checkbounds(Bool, A, 1, 0, 1) == false
    @test checkbounds(Bool, A, 1, 1, 0) == false
end

@testset "vector indices" begin
    @test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
    @test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
    @test checkbounds(Bool, A, 1:5, 0:4, 1:3) == false
    @test checkbounds(Bool, A, 1:5, 1:4, 0:3) == false
end

You can see here that tests defined for abstractarray.jl are made so that each test set check a number of related things. Also there are no fixtures. We simply put e.g. the array being tested on called A outside the test sets. That makes it available in every test.

Let us do another example by looking at the test for individual character represented by the Char type.

# Julia: char.jl tests
@testset "basic properties" begin

    @test typemin(Char) == Char(0)
    @test ndims(Char) == 0
    @test getindex('a', 1) == 'a'
    @test_throws BoundsError getindex('a', 2)
    # This is current behavior, but it seems questionable
    @test getindex('a', 1, 1, 1) == 'a'
    @test_throws BoundsError getindex('a', 1, 1, 2)

    @test 'b' + 1 == 'c'
    @test typeof('b' + 1) == Char
    @test 1 + 'b' == 'c'
    @test typeof(1 + 'b') == Char
    @test 'b' - 1 == 'a'
    @test typeof('b' - 1) == Char

    @test widen('a') === 'a'
    # just check this works
    @test_throws Base.CodePointError Base.code_point_err(UInt32(1))
end

@testset "issue #14573" begin
    array = ['a', 'b', 'c'] + [1, 2, 3]
    @test array == ['b', 'd', 'f']
    @test eltype(array) == Char

    array = [1, 2, 3] + ['a', 'b', 'c']
    @test array == ['b', 'd', 'f']
    @test eltype(array) == Char

    array = ['a', 'b', 'c'] - [0, 1, 2]
    @test array == ['a', 'a', 'a']
    @test eltype(array) == Char
end

Again you can see that we group related tests into one test set. The second test set I added as an example because it shows how Julia developers also use a popular advice about testing which is to add tests for bugs which has popped up in the past to avoid regressing on the bug again.

Nesting

A test set with code you are testing can also contain other test sets. Here is an example from the testing of the Dict type used to represent a dictionary in Julia.

# Julia: dict.jl tests

@testset "Dict" begin
    h = Dict()
    for i=1:10000
        h[i] = i+1
    end
    for i=1:10000
        @test (h[i] == i+1)
    end
    for i=1:2:10000
        delete!(h, i)
    end
    
    h = Dict{Any,Any}("a" => 3)
    @test h["a"] == 3
    h["a","b"] = 4
    @test h["a","b"] == h[("a","b")] == 4
    h["a","b","c"] = 4
    @test h["a","b","c"] == h[("a","b","c")] == 4

    @testset "eltype, keytype and valtype" begin
        @test eltype(h) == Pair{Any,Any}
        @test keytype(h) == Any
        @test valtype(h) == Any

        td = Dict{AbstractString,Float64}()
        @test eltype(td) == Pair{AbstractString,Float64}
        @test keytype(td) == AbstractString
        @test valtype(td) == Float64
        @test keytype(Dict{AbstractString,Float64}) === AbstractString
        @test valtype(Dict{AbstractString,Float64}) === Float64
    end
end

You can see here that the dictionary named h tested on inside the inner test set was actually defined first in the outer test set.

Using Loops

It is easy in Julia to repeat tests across different kinds of data using loops.

let x = Dict(3=>3, 5=>5, 8=>8, 6=>6)
    pop!(x, 5)
    for k in keys(x)
        Dict{Int,Int}(x)
        @test k in [3, 8, 6]
    end
end

We can even loop on the whole test set itself. Here we are running the same set of test for both the == and isequal function:

@testset "equality" for eq in (isequal, ==)
    @test  eq(Dict(), Dict())
    @test  eq(Dict(1 => 1), Dict(1 => 1))
    @test !eq(Dict(1 => 1), Dict())
    @test !eq(Dict(1 => 1), Dict(1 => 2))
    @test !eq(Dict(1 => 1), Dict(2 => 1))
end

How to Write Fixtures in Julia

For a lot of testing frameworks we have the concept of fixtures. That means some data which is initialized and used repeatedly in multiple tests. In Julia the belief is that this is just over-engineering and it simpler and more obvious what is going on by being explicit.

Immutable Fixtures

That means either we put the reuse object outside multiple test sets like this:

A = rand(5,4,3)
@testset "Bounds checking" begin
    @test checkbounds(Bool, A, 1, 1, 1) == true
    @test checkbounds(Bool, A, 5, 4, 3) == true
end

@testset "vector indices" begin
    @test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
    @test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
end

Mutable Fixtures

Or if the state is modified so we need to recreate it each time, when we would instead use a function to create it:

test_array() = rand(5,4,3)

@testset "Bounds checking" begin
      A = test_array()
    @test checkbounds(Bool, A, 1, 1, 1) == true
    @test checkbounds(Bool, A, 5, 4, 3) == true
end

@testset "vector indices" begin
      A = test_array()
    @test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
    @test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
end