Running Clojure tests in Leiningen projects is fairly simple for most use-cases. For a few out-of-the-ordinary use-cases, however, it can become surprisingly difficult to control. In this post, we look at how to write Leiningen test selectors that go beyond simple use-cases.
The Pivotal Role of Test Selectors in Continuous Delivery
Test selectors can be thought of as a way to decide which tests to run because not every test needs to run every time. Martin Fowler, in this essay, elucidates the necessity for having a layered testing architecture, from regularly run fast unit tests to slower and more pervasive integration and system tests:
One of the challenges of an automated build and test environment is you want your build to be fast, so that you can get fast feedback, but comprehensive tests take a long time to run. A deployment pipeline is a way to deal with this by breaking up your build into stages. Each stage provides increasing confidence, usually at the cost of extra time. Early stages can find most problems yielding faster feedback, while later stages provide slower and more through probing.
As a prerequisite to a layered test suite, we need to have the ability to select which tests to run at each stage of the deployment pipeline. In Clojure and Leiningen, such selection of tests can be done through functions called “test selectors” which use metadata of tests to decide if a test should be run or not.
Basic Usage of Leiningen Test Selectors
Let us start off by looking at a few examples of basic use-cases of Leiningen test selectors.
lein testprovides the ability to run all tests in a project.
lein test <selector> <selector-args>allows a custom selector to be defined by a project’s
project.cljfile in the
:test-selectorskey. A selector is a function that can look at a namespace or a test var and decide whether a test should run or not.
- Tests which run with
lein testcan be overridden by defining a
A basic usage of test selectors is available via Leiningen’s help task
lein help test :
:only selector, we can run only certain tests or all tests in a few namespaces. But what if we wanted to run only
integration tests from a select few namespaces? It is not apparent how to write a test selector that would do this (efficiently). In the rest of this post, we’ll build our own
:integration selector step by step and will try to understand how test selectors work.
A Verbose Test Selector
Let’s look at what a var’s metadata looks like to a test selector function by defining a simple selector which selects every test var but prints the metadata of tests that it sees:
lein test :verbose prints the following for one test var:
Any of the keys in this metadata map may be used to select or reject a test var. One can choose the name of the test, its namespace, the metadata in the
ns form of the namespace that the var is located in (inherited metadata), or other user-defined metadata keys like
:integration. In the example above,
:integration are custom metadata added to our test var using the
(deftest ^:integration ^:flaky remote-system-time-difference-test
Leiningen’s Example Test Selectors
Let’s look at how test selectors are defined and used usually. Note that every test selector gets the same input. So in the snippet below, all three of
:integration test selectors get the same metadata map of the test var being considered for selection. We use
:integration as Clojure keyword functions applied on the metadata map.
By default in Leiningen, the
:default test selector is invoked when no test selector is specified in the test run command. It can be overridden in a project like the above example. In the absence of a
:default test selector in the project, it can be assumed to be
(constantly true). In the selectors map above, when running
lein test, only tests that are not marked as
:integration will be run. Test selectors get access to a var’s (inherited) metadata, which includes the flags put in like
^:integration in the
ns form. So, the keys in the
:test-selectors map are used to refer to a selector. This key is used in the command to run selected tests decided by the selector.
As printed out by
lein help test, a test selector is primarily a function, which looks at the metadata of test vars and namespaces and decides whether to run that test or not. This was the only way to select tests in Leiningen for a while, but since version 2.0.0, a test selector works on two levels. On the first level, a test selector decides whether a test namespace should be considered at all in this test run and if yes, it decides which test vars in the namespace should be run. This namespace-selection logic was specifically added to avoid loading namespaces which are not required in a test run, as that can become a bottleneck in projects having a large number of namespaces.
This can raise the question: why would one need to limit the number of namespaces which are loaded? While we cannot specify a safe limit for the number of namespaces to be loaded, it is going to become a limiting factor if the number of namespaces keeps growing. Moreover, there are possibilities where
once-fixtures, ie test setup/teardown functions, can run even if no test is selected in a namespace by the var selector. We proposed a pull request in Leiningen to avoid this particular case.
A test selector can be written in two ways:
- A function: If the test selector is a function, it is applied to the metadata of all test vars (including the metadata inherited by the var from the namespace’s metadata). The tests for which this selector function returns a truthy value will be run.
- A vector of two functions: If the test selector is a vector of two functions, then the first function is used to determine if a test namespace should be loaded based on the namespace symbol (ie, should tests in this namespace be considered?). Among the namespaces which are loaded, each of the vars in those namespaces is passed to the second function, which decides, based on the var’s (inherited) metadata, whether to run the test or not. This second function is similar to the case where the selector is just a function.
Let us write a vector test selector. The first function (the namespace selector) runs before loading the namespace, so we don’t expect to see much metadata present here. We can write a test selector which highlights the differences in the inputs to the namespace selector and the var selector. Our namespace selector will print “Should I select this namespace?” and the var selector will print “Should I select this var?”.
When we run this selector with
lein test :demo-ns-selector, we get the following printed lines for each test and namespace, the second function in this selector prints the exact same thing as our
Should I select this namespace?
jinx.api.core-testShould I select this var?
:test #object[jinx.api.core_test$fn__4206 0x6ce1cc03
Unlike the var selector (the second function), the namespace selector only gets the namespace as a
Symbol as its input (and the command line arguments to the selector function). Its only goal is to ignore all tests in a namespace based solely on the name of the namespace. That is, if we knew how to select/reject tests based on the name of the namespace a test is located in, then we could bypass loading the namespace entirely.
A common usage pattern we have observed is that after making changes to one module, we like to locally run that module’s tests before committing our changes. For example, if we are making changes in the API server module, it is useful to run all integration tests in the module:
Leiningen’s :only Selector
A default :only test-selector is available to run select tests. For example,
`lein test :only leiningen.test.test/test-default-selector` only runs the
This only selector is defined in Leiningen’s source code as follows:
Ignoring the syntax quoting, the first function inside
only-form takes a namespace in a project and checks whether that namespace was part of the command-line arguments to the test command. In the above snippet,
vars are the command line arguments.
The second function is used to decide whether a single test var (in the namespaces that were selected by the first function) and is similar to the test selectors we have seen above. This second function selects either a test in a namespace or all tests in the namespace based on the input.
It can be a little confusing to realise that the second function alone can also choose to not run any test that comes from a certain namespace. This is because it has access to the metadata of a test var, which contains the namespace the test is located in. Although it is possible to only have the second selector function, it can only run after a namespace has been loaded. A shortcoming of the current way of expressing selectors is that there is no way to only have the first selector in the vector-based selector syntax. If a selector is expressed as a vector, the vector has to contain two elements even if the second element is a redundant
An Enhanced Integration Test Selector
Let us revisit the original test selector which we set out to write. Noting down our requirements:
- The selector should only run tests which have the
- If namespaces are passed as arguments to the selector, then only
:integrationtests in those namespaces should be run.
- If test vars of the form
<namespace>/<var>are passed as arguments to the selector, then only those test vars should be run which are
- The rules are composable, meaning that
lein test :integration <ns1> <ns2>/<var1>should run all integration tests in the namespace
ns1and the test
var1in the namespace
ns2if it is marked as an
:integrationtest (either in the
deftestform or in the
- If no namespace or test var is passed as arguments, all
:integrationtests in the project should be run.
These requirements basically want to combine the behaviour of the
:only selector and a selector based on tests’ metadata. Since selectors cannot be composed like functions can be in Clojure code, we can update the
:only selector with our requirements:
Intertwinement between Leiningen and clojure.test
clojure.test is defined as “a unit testing framework” which takes inspiration from the unit testing framework chapter in Practical Common Lisp by Peter Seibel. In effect, the cornerstone of Clojure tests is the
deftest form which is a macro that evaluates to a function with the metadata
:test, the same metadata we have been using thus far to select tests. Other than this ability to define a test,
clojure.test provides two high-level functions which can be used to find and run tests:
run-all-tests: Goes through all namespaces that have been loaded and runs all functions that have the
run-tests: Runs tests (functions with the
:testmetadata) in the current namespace or namespaces passed in as arguments. This is the function Leiningen uses to pass in tests that have been selected by its selector (and removing the
:testmetadata from vars that are not selected).
Tests are selected by Leiningen and handed off to
clojure.test. Leiningen looks at a test’s metadata, passes it to the selector function(s), and upon receiving a truthy value, calls
run-tests to actually run the tests. Here we perhaps need to delve into a somewhat philosophical question: What is the role of a test runner in a testing framework? Borrowing from Lambda Island’s explanation of the testing ecosystem, a test framework can be composed of the following components:
- Test syntax: How to define tests in code?
- Test semantics: How does the system define what a test is?
- Assertion libraries: Ways to write test statements.
- Mocking/stubbing libraries: Utilities to help with test isolation.
- Test fixtures and factories: Utilities for building up test data and establishing application states.
- Progress reporters: Ways to get progress information while tests are running.
- Result reporters: Summarize the results of a test run for human or machine consumption.
- Test runners: Comprehensively load, detect, and run test suites.
clojure.test provides most of the above but it lacks a few test runner capabilities. Leiningen adds the following to the capabilities of
- Ability to discover and load test namespaces, which is a prerequisite for
clojure.testto be able to run tests in those namespaces.
- Selection logic for deciding which tests to run.
Leiningen test selectors are easy to configure for simple use cases like running all integration tests always, but for more complicated use-cases, it is hard to configure and understand. Much of the downsides, though, as admitted by the author of Leiningen, Phil Hagelberg, is because of the lack of support for test selectors in the
clojure.test test runner. From Leiningen’s source code:
This is a massive and terrible monkeypatch to work around the fact that the built-in clojure.test library does not accept patches from outside the core team. At this point the monkeypatching code below should be considered legacy, and rather than trying to improve it, all efforts should be instead directed towards improving external libraries such as https://github.com/circleci/circleci.test, which has a superset of these features and also doesn’t require you to stop using clojure.test to write your tests.
We recommend that projects override the test task with an alias that calls out to a third-party testing library instead.
Other test runners like CircleCI’s test runner and Lambda Island’s Kaocha have solved the problem of selecting tests in more elegant ways, though they come with their own set of tradeoffs which we can delve into in a future post. Some features that could prove to be supremely useful in selecting tests through Leiningen:
- Improving the test selector API: Instead of vectors of functions, a map with expressive keys as to what the selector functions do would help in the readability of test selectors. A caveat of the vector-based selectors is that the vector must have two elements, even if the second element is not necessary, even if it is
- Composing test selector functions: For example, the ability to run
:flakytests in the same run without writing another selector for it.
- Being able to access the metadata of namespaces without fully loading the namespace by only reading the metadata in the
nsform and not the whole file.
- Reading the selector functions from a separate location other than
project.cljwhich can quickly become cluttered due to the functions defined inside the
- The ability to pre-process the selector arguments passed in as command line (the pre-processing, for example, converting the command-line namespaces to a set as done in the
:onlyselector, happens once per namespace but could be computed only once for the whole test run).
- Having access to a namespace’s dependency information while selecting a test namespace similar to a tool like
clojure.tools.namespace.dependency(for example, running all tests that depend on a certain source namespace).
Notwithstanding the alternative test runners and desired improvements, most of our testing requirements are fulfilled by the test selectors in Leiningen and we are thankful that Leiningen provides the ability to select tests even though Clojure’s test runner does not provide much support for it.
The source code and examples used in this post are available here: https://github.com/mourjo/jinx
Addendum: Interoperability of Selectors with Namespaces in the CLI
Even without specifying any selector, it is possible to run tests in given namespaces by passing the namespaces as command line arguments to the
test task itself in Leiningen, for example,
lein test <ns1> <ns2>. So how does that fit into the selector framework? As far as we can tell, this was the old way of isolating tests to a few namespaces, before test selectors started taking shape. Today, a test selector’s logic is applied on top of the namespaces passed to the test command (if no test namespaces are passed, then the test selector logic applies to all test namespaces):
lein test <ns1> <ns2> <test-selector> <selector-agrs>
So the difference between
lein test <ns1> and
lein test :only <ns1> is that
:only will run all tests in the namespace
ns1 whereas the former will run those tests in the namespace
ns1 that are allowed by the
:default selector. Similarly,
lein test :only <ns1> and
lein test <ns1> :all will run all tests in the namespace
ns1, the latter achieves this by using the
:all selector provided by Leiningen along with