Supercharge your ExUnit skills in 10 minutes 💪

TL;DR: Changes coming in Elixir 1.8 are amazing! I feel like in candy-shop. Seriously, read full article, it gets even better at the end :)

Photo by Jared Sluyter on Unsplash

In this article you will learn how to use full power of mix test runner.

If you look how to write ExUnit test cases— you should look into my another article.Subscribe to this publication and get notified when I publish it.

I love TDD and any tool that makes my TDD process faster and more enjoyable is very welcome. Changes in Elixir 1.8 make TDD develop-save-run tests cycle even faster and better.

1. Running tests

Well, this part is pretty straightforward, or not?

Here are possible ways to run tests (clap as many times as you learned new way of running tests :) ).

Content of test/chocolate_test.exs

• mix test — run all tests. This one is easy.
• mix test test/chocolate_test.exs — run just one single file with tests. This one is also easy.
• mix test test/chocolate_test.exs:4 — run just one single test within test file. Equivalent of mix test test/chocolate_test.exs --only line:4 Same test clause will run if you specify lines 5 and 6. Second test will run if you use lines 8,9,10.
• mix test test/chocolate_test.exs --only module:ChocolateTest — will find and execute only tests defined in test module ChocolateTest . While you can run mix test --only module:ChocolateTest that will be extremely inefficient, because elixir first need to load (and compile) all test files in tests directory and then only leave only one module.
• mix test test/chocolate_test.exs --only test:"test ice cream" — will find and run test by test name. Note that test name actually have “text “ prefix before it — that is added automatically by ExUnit and you need include it in the filter too.
  Using this form without filename may make sense, because you may have similarly named tests — like “successfully starts”. And you can run all of them with one single filter.
  If you group tests using describe — see examples below.

ExUnit test example with describe section and tags

• mix test test/chocolate_test.exs --only has_milk — will run only tests marked with tag has_milk (without any value). In case above it will be 2 test cases.
• mix test test/chocolate_test.exs --only sugar_amount:100 — will run only tests market with tag sugar_amount with specific value.
• mix test test/chocolate_test.exs --exclude test --include sugar_amount:100 --include has_milk— will run only tests marked with all included tags. --exclude test first excludes all tests (because all of them are marked with tag test) and then includes all tests requested by --include option.
• mix test test/chocolate_test.exs --only describe:"sweets" — will execute tests contained in section “sweets”. That comes useful for all test suite, if you have same section in (all) test modules and it tests similar scenarios (like describe:"error messages"to select only tests dealing with errors and so on).
• mix test test/chocolate_test.exs --only test:"test sweets ice cream" — will execute only one specific test from test suite. Note that first word “test” comes from ExUnit itself, “sweets” is section name (describe) and “ice cream” is the test name. You can see full test names when running mix test test/chocolate_test.exs --trace .

All this richness of filters comes from the fact that ExUnit automatically assigns default tags to each test (run iex, type h ExUnitand read section “Known tags”).

2. Speed-up test run on large code-base by using 'stale' mode

Stale is special mode or running mix that builds cross reference of modules and then executes only that tests, which include (rely) on changed modules. This includes modules directly referred by tests and modules referred by that modules and so on. So even if you change module deep in dependency tree — all modules relying on it will be marked as invalidated and tests for them will re-run.

In this case you don’t need to remember which tests cover depend on module you changing right now — mix test will do it for you.

Two modules defined in same source file.

Important notice:

If you have several tests modules in same source file (.exs) or several modules defined in same source file (.ex, like in example above) — then changing content of one of the modules in the file will cause recompilation of all modules defined in that source file and will re-run tests for all of them. So it is better to split modules and define each module and each test module in their own source file. In that way you will benefit most from mix test --stale mode.

You can combine running tests with --listen-on-stdin option and just press Enter to force tests to re-run. Final command line will be

mix test --stale --listen-on-stdin

3. Speed-up test run on large code-base by using ‘failed’ mode

In this mode mix test will re-run only tests that failed before. On the first run it executes whole test suite, then remembers failed tests and re-runs only them.

On a big test base it is really handy when refactoring and fixing broken tests, but it has drawback that it will not tell you if you broke any other tests, which were working before.

You also can combine running tests with --listen-on-stdin option and just press Enter to force tests to re-run. Final command line will be

mix test --failed --listen-on-stdin

4. Use --max-failures to stop on failed test (starting from Elixir 1.8)

This is the most interesting part of upcoming release! I was about to implement it (see this stream), but just discovered it in documentation.

This feature allows you to stop after some number of tests in test suite failed. In TDD mode I usually run it with 1 — so it will stop on first failed test. It is equivalent of Python’s “python unittest — -failfast”.

Here is the example when all tests are run:

Running ExUnit with 2 passing tests, 3 failing tests.

Here is the example with option --max-failures 1 so that test execution stops after first failed test.

5. Run in predictable sequence --seed 0

By default mix test randomizes order in which it runs tests, so that if some tests or test suites rely on order how they are executed — it will reveal itself quite quickly. You can remember number from end of the run and then re-run them exactly in same sequence using mix test --seed THAT_NUMBER .

In other hand when refactoring it if beneficial to have tests executed in same order as they are defined — because usually in TDD first tests are very simple and then they progress in complexity, as software was developed.

Simple tests usually are on the top of the file and it is beneficial to fix them first, before fixing more complex ones. Using option --seed 0 will always execute tests in order they are defined.

Don’t forget to run test suite without this option to benefit from execution order randomization and see if there are flaky tests.

6. Ultimate powerful mix test command-line to run tests

mix test — stale — max-failures 1 — listen-on-stdin — trace — seed 0

What does it really do?

• --stale executes only tests for source and test files which are changed.
• --max-failures 1 stops execution after first failing test.
• --listen-on-stdin allows you press Enter and have tests re-run.
• --trace prints names of tests instead of dots, so you can see which tests are executed in more descriptive way.
• --seed 0 forces tests to execute in same order they are mentioned in source file, so that you will have predictable consecutive runs of the tests.

7. Automatically re-run tests on file-system changes

There is amazing modules called mix-test.watch which enhances mix test with ability to monitor file system for changes and re-running tests when something changes.

Install it using by adding it to deps in your mix.exs .

{:mix_test_watch, "~> 0.8", only: :dev, runtime: false}

Afterwards you can run mix test.watch to monitor all sources and tests for changes and re-running them.

Important: Use --seed 0 option with test.watch so that next run will execute tests exactly in same sequence and if you still did not fixed test case you are working on — you will see it failing.

Combine this module with previous command, especially with --max-failures 1 and you get very powerful tool for TDD.

mix test.watch --stale --max-failures 1 --trace --seed 0

8. Extra super useful clap-worthy TDD setup :) Get instant desktop notifications on test success or failure in a seconds after you save source file

There is another amazing module ex_unit_notifier which sends you desktop notifications after test suite run.

Install it using by adding following dependency to deps in your mix.exs .

{:ex_unit_notifier, "~> 0.1", only: :test}

Also add this line to test/test_helper.exs

ExUnit.configure formatters: [ExUnit.CLIFormatter, ExUnitNotifier]
ExUnit.start

Then run test suite using mix test.watch:

mix test.watch --stale --max-failures 1 --trace --seed 0

Magic! You will see in a seconds after saving file notification popping up with test suite run statistics. Here is notifications after fixing tests one by one until all test suite passes.

About me

I’m Gaspar Chilingarov . I facilitate DevOps transition, help moving legacy applications to cloud and write high-performance Elixir apps.

Need help with your Elixir app or want prototype your next microservice in Elixir? DM me on Twitter or Github.

You can connect with me on Twitter, Facebook, LinkedIn and GitHub.

Found this post useful? Kindly tap the ❤ button below! :) Let’s spread word about Elixir.