CODEX
Julia v1.5 Testing: How to Organize Tests
We explore how Julia tests are organized differently from what you may be used to in other testing frameworks.

Julia development practices will likely not be the same as what you have experienced in Java, JavaScript, Python, Ruby and other popular programming languages.
Thus in this story I will do a sort of quick recap of how testing is often done in other languages and compare that with the common practice followed in Julia as exemplified by the Julia standard library.
How to effectively run tests: Julia v1.5 Testing: Best Practices.
No Tests But Nested Test Sets
Testing in Julia is a lot more free form than what you may be used to. For contrast let me show some examples from other tests frameworks.
Pytest
Pytest is a frequently used testing framework for Python. Here one simply prefixes functions that represents tests with test_
as shown below:
# Python: Pytest
def capital_case(x):
return x.capitalize()
def test_capital_case():
assert capital_case('semaphore') == 'Semaphore'
In Python one would execute these tests by running:
$ pytest
Go Unit Tests
Go comes with a builtin unit testing framework. Why I am showing all these frameworks instead of jumping straight to Julia? Because these work how people are used to. And I need something to contrast with to help clarify how Julia testing is different.
// Go
package main
import "testing"
func TestSum(t *testing.T) {
total := Sum(5, 5)
if total != 10 {
t.Errorf("Sum was incorrect, got: %d, want: %d.", total, 10)
}
}
For Go tests you put them in a file which ends with _test
such as foobar_test.go
. Each test is placed in a function starting with Test
.
$ go test
This will look for the files ending with _test.go
and treat every function with a Test
prefix taking a testing.T
type argument as a test.
JUnit
In Java JUnit is commonly used. Each test is a method on a test class as shown below:
// Java: JUnit Tests
public class MyTests {
@Test
public void multiplicationOfZeroIntegersShouldReturnZero() {
MyClass tester = new MyClass(); // MyClass is tested
// assert statements
assertEquals(0, tester.multiply(10, 0), "10 x 0 must be 0");
assertEquals(0, tester.multiply(0, 10), "0 x 10 must be 0");
assertEquals(0, tester.multiply(0, 0), "0 x 0 must be 0");
}
}
These classes are then collected into a test suite, thus you can avoid making enormous test classes:
Java: JUnit
@RunWith(Suite.class)
@SuiteClasses({
MyClassTest.class,
MySecondClassTest.class })
public class AllTests {
}
Julia Test Sets
In Julia we do away with this artificial separation between test methods, test classes, test suites etc. Instead everything gets mashed into one concept called a testset
. These are more flexible than what you find in other frameworks which is why developers coming from other frameworks may feel uncomfortable with the much looser form use in Julia testing.
In Julia everything is much more free form. You tailor the test framework more to your own preferences and style. This is possible because test sets can be nested.
Here is an example from my Little Man Computer (LMC) assembler.
@testset "Disassembler tests" begin
@testset "Without operands" begin
@test disassemble(901) == "INP"
@test disassemble(902) == "OUT"
@test disassemble(000) == "HLT"
end
@testset "With operands" begin
@test disassemble(105) == "ADD 5"
@test disassemble(112) == "ADD 12"
@test disassemble(243) == "SUB 43"
@test disassemble(399) == "STA 99"
@test disassemble(510) == "LDA 10"
@test disassemble(600) == "BRA 0"
@test disassemble(645) == "BRA 45"
@test disassemble(782) == "BRZ 82"
end
end
I have have defined test sets for each major component. So e.g. the assembler, disassembler and simulator are all represented by a different test set. Each of these test sets have sub test sets which test aspects of that component.
Thus a Julia @testset
corresponds to both a test function as well as a test class, and a test suite.
What to Put in a Test Set
There are a lot of different ideas of what should be in a unit test. Many swear by to having only one specific thing being tested per test. The Julia convention for test sets is that each test set has a collection of related tests. This is best understood by simply looking at real world tests found in the Julia standard library.
The test sets I show as examples here will be shortened a bit by me as there is not particular value in showing the full length of each test.
# Julia: abstractarray.jl tests
A = rand(5,4,3)
@testset "Bounds checking" begin
@test checkbounds(Bool, A, 1, 1, 1) == true
@test checkbounds(Bool, A, 5, 4, 3) == true
@test checkbounds(Bool, A, 0, 1, 1) == false
@test checkbounds(Bool, A, 1, 0, 1) == false
@test checkbounds(Bool, A, 1, 1, 0) == false
end
@testset "vector indices" begin
@test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
@test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
@test checkbounds(Bool, A, 1:5, 0:4, 1:3) == false
@test checkbounds(Bool, A, 1:5, 1:4, 0:3) == false
end
You can see here that tests defined for abstractarray.jl
are made so that each test set check a number of related things. Also there are no fixtures. We simply put e.g. the array being tested on called A
outside the test sets. That makes it available in every test.
Let us do another example by looking at the test for individual character represented by the Char
type.
# Julia: char.jl tests
@testset "basic properties" begin
@test typemin(Char) == Char(0)
@test ndims(Char) == 0
@test getindex('a', 1) == 'a'
@test_throws BoundsError getindex('a', 2)
# This is current behavior, but it seems questionable
@test getindex('a', 1, 1, 1) == 'a'
@test_throws BoundsError getindex('a', 1, 1, 2)
@test 'b' + 1 == 'c'
@test typeof('b' + 1) == Char
@test 1 + 'b' == 'c'
@test typeof(1 + 'b') == Char
@test 'b' - 1 == 'a'
@test typeof('b' - 1) == Char
@test widen('a') === 'a'
# just check this works
@test_throws Base.CodePointError Base.code_point_err(UInt32(1))
end
@testset "issue #14573" begin
array = ['a', 'b', 'c'] + [1, 2, 3]
@test array == ['b', 'd', 'f']
@test eltype(array) == Char
array = [1, 2, 3] + ['a', 'b', 'c']
@test array == ['b', 'd', 'f']
@test eltype(array) == Char
array = ['a', 'b', 'c'] - [0, 1, 2]
@test array == ['a', 'a', 'a']
@test eltype(array) == Char
end
Again you can see that we group related tests into one test set. The second test set I added as an example because it shows how Julia developers also use a popular advice about testing which is to add tests for bugs which has popped up in the past to avoid regressing on the bug again.
Nesting
A test set with code you are testing can also contain other test sets. Here is an example from the testing of the Dict
type used to represent a dictionary in Julia.
# Julia: dict.jl tests
@testset "Dict" begin
h = Dict()
for i=1:10000
h[i] = i+1
end
for i=1:10000
@test (h[i] == i+1)
end
for i=1:2:10000
delete!(h, i)
end
h = Dict{Any,Any}("a" => 3)
@test h["a"] == 3
h["a","b"] = 4
@test h["a","b"] == h[("a","b")] == 4
h["a","b","c"] = 4
@test h["a","b","c"] == h[("a","b","c")] == 4
@testset "eltype, keytype and valtype" begin
@test eltype(h) == Pair{Any,Any}
@test keytype(h) == Any
@test valtype(h) == Any
td = Dict{AbstractString,Float64}()
@test eltype(td) == Pair{AbstractString,Float64}
@test keytype(td) == AbstractString
@test valtype(td) == Float64
@test keytype(Dict{AbstractString,Float64}) === AbstractString
@test valtype(Dict{AbstractString,Float64}) === Float64
end
end
You can see here that the dictionary named h
tested on inside the inner test set was actually defined first in the outer test set.
Using Loops
It is easy in Julia to repeat tests across different kinds of data using loops.
let x = Dict(3=>3, 5=>5, 8=>8, 6=>6)
pop!(x, 5)
for k in keys(x)
Dict{Int,Int}(x)
@test k in [3, 8, 6]
end
end
We can even loop on the whole test set itself. Here we are running the same set of test for both the ==
and isequal
function:
@testset "equality" for eq in (isequal, ==)
@test eq(Dict(), Dict())
@test eq(Dict(1 => 1), Dict(1 => 1))
@test !eq(Dict(1 => 1), Dict())
@test !eq(Dict(1 => 1), Dict(1 => 2))
@test !eq(Dict(1 => 1), Dict(2 => 1))
end
How to Write Fixtures in Julia
For a lot of testing frameworks we have the concept of fixtures. That means some data which is initialized and used repeatedly in multiple tests. In Julia the belief is that this is just over-engineering and it simpler and more obvious what is going on by being explicit.
Immutable Fixtures
That means either we put the reuse object outside multiple test sets like this:
A = rand(5,4,3)
@testset "Bounds checking" begin
@test checkbounds(Bool, A, 1, 1, 1) == true
@test checkbounds(Bool, A, 5, 4, 3) == true
end
@testset "vector indices" begin
@test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
@test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
end
Mutable Fixtures
Or if the state is modified so we need to recreate it each time, when we would instead use a function to create it:
test_array() = rand(5,4,3)
@testset "Bounds checking" begin
A = test_array()
@test checkbounds(Bool, A, 1, 1, 1) == true
@test checkbounds(Bool, A, 5, 4, 3) == true
end
@testset "vector indices" begin
A = test_array()
@test checkbounds(Bool, A, 1:5, 1:4, 1:3) == true
@test checkbounds(Bool, A, 0:5, 1:4, 1:3) == false
end