The Convoluted Magic of Leiningen Test Selectors

Running Clojure tests in Leiningen projects is fairly simple for most use-cases. For a few out-of-the-ordinary use-cases, however, it can become surprisingly difficult to control. In this post, we look at how to write Leiningen test selectors that go beyond simple use-cases.

A sample integration test build on our CI/CD platform. To run an integration test build like this, we need to select tests that are classified as “integration tests”. This can be done in Leiningen through test selectors.

The Pivotal Role of Test Selectors in Continuous Delivery

Test selectors can be thought of as a way to decide which tests to run because not every test needs to run every time. Martin Fowler, in this essay, elucidates the necessity for having a layered testing architecture, from regularly run fast unit tests to slower and more pervasive integration and system tests:

One of the challenges of an automated build and test environment is you want your build to be fast, so that you can get fast feedback, but comprehensive tests take a long time to run. A deployment pipeline is a way to deal with this by breaking up your build into stages. Each stage provides increasing confidence, usually at the cost of extra time. Early stages can find most problems yielding faster feedback, while later stages provide slower and more through probing.

As shown in this figure, software testing begins with unit tests. Testing of more complex functionality is layered atop unit tests. https://landing.google.com/sre/sre-book/chapters/testing-reliability/

As a prerequisite to a layered test suite, we need to have the ability to select which tests to run at each stage of the deployment pipeline. In Clojure and Leiningen, such selection of tests can be done through functions called “test selectors” which use metadata of tests to decide if a test should be run or not.

Basic Usage of Leiningen Test Selectors

Let us start off by looking at a few examples of basic use-cases of Leiningen test selectors.

• lein test provides the ability to run all tests in a project.
• lein test <selector> <selector-args> allows a custom selector to be defined by a project’s project.clj file in the :test-selectors key. A selector is a function that can look at a namespace or a test var and decide whether a test should run or not.
• Tests which run with lein test can be overridden by defining a :default test selector.

A basic usage of test selectors is available via Leiningen’s help task lein help test :

The lein help test command prints out an example test selector and explains the basic usage of test selectors. Test selectors shown here take as input test metadata map containing keys like :integration and are expected to return truthy values indicating whether a test should run or not.

Using the :only selector, we can run only certain tests or all tests in a few namespaces. But what if we wanted to run only integration tests from a select few namespaces? It is not apparent how to write a test selector that would do this (efficiently). In the rest of this post, we’ll build our own :integration selector step by step and will try to understand how test selectors work.

A Verbose Test Selector

Let’s look at what a var’s metadata looks like to a test selector function by defining a simple selector which selects every test var but prints the metadata of tests that it sees:

Using the :test-selector keys in project.clj, we define a simple selector which selects every test var but prints the metadata of test vars for us to understand what is available for us to select a test on

Running lein test :verbose prints the following for one test var:

The :verbose selector prints metadata of every test to stdout, here is the metadata of one test. Any and all of this information is available to the var-selector and can be used to select or reject a test for the current run.

Any of the keys in this metadata map may be used to select or reject a test var. One can choose the name of the test, its namespace, the metadata in the ns form of the namespace that the var is located in (inherited metadata), or other user-defined metadata keys like :integration. In the example above, :flaky and :integration are custom metadata added to our test var using the ^ notation:

(deftest ^:integration ^:flaky remote-system-time-difference-test
  ;; assertions
)

Leiningen’s Example Test Selectors

Let’s look at how test selectors are defined and used usually. Note that every test selector gets the same input. So in the snippet below, all three of :verbose, :default and :integration test selectors get the same metadata map of the test var being considered for selection. We use :default and :integration as Clojure keyword functions applied on the metadata map.

Our test-selectors map now includes default and integration test selector functions

Test selectors is a map defined in the project.clj file. Keys in this map are used to refer to a selector and that key is used in the command to run tests selected by it.

By default in Leiningen, the :default test selector is invoked when no test selector is specified in the test run command. It can be overridden in a project like the above example. In the absence of a :default test selector in the project, it can be assumed to be (constantly true). In the selectors map above, when running lein test, only tests that are not marked as :integration will be run. Test selectors get access to a var’s (inherited) metadata, which includes the flags put in like ^:integration in the deftest or ns form. So, the keys in the :test-selectors map are used to refer to a selector. This key is used in the command to run selected tests decided by the selector.

Namespace Selectors

As printed out by lein help test, a test selector is primarily a function, which looks at the metadata of test vars and namespaces and decides whether to run that test or not. This was the only way to select tests in Leiningen for a while, but since version 2.0.0, a test selector works on two levels. On the first level, a test selector decides whether a test namespace should be considered at all in this test run and if yes, it decides which test vars in the namespace should be run. This namespace-selection logic was specifically added to avoid loading namespaces which are not required in a test run, as that can become a bottleneck in projects having a large number of namespaces.

This can raise the question: why would one need to limit the number of namespaces which are loaded? While we cannot specify a safe limit for the number of namespaces to be loaded, it is going to become a limiting factor if the number of namespaces keeps growing. Moreover, there are possibilities where once-fixtures, ie test setup/teardown functions, can run even if no test is selected in a namespace by the var selector. We proposed a pull request in Leiningen to avoid this particular case.

A test selector can be written in two ways:

• A function: If the test selector is a function, it is applied to the metadata of all test vars (including the metadata inherited by the var from the namespace’s metadata). The tests for which this selector function returns a truthy value will be run.
• A vector of two functions: If the test selector is a vector of two functions, then the first function is used to determine if a test namespace should be loaded based on the namespace symbol (ie, should tests in this namespace be considered?). Among the namespaces which are loaded, each of the vars in those namespaces is passed to the second function, which decides, based on the var’s (inherited) metadata, whether to run the test or not. This second function is similar to the case where the selector is just a function.

Let us write a vector test selector. The first function (the namespace selector) runs before loading the namespace, so we don’t expect to see much metadata present here. We can write a test selector which highlights the differences in the inputs to the namespace selector and the var selector. Our namespace selector will print “Should I select this namespace?” and the var selector will print “Should I select this var?”.

A test selector to print the name of the namespace being evaluated in order for us to understand what we can select on at the namespace selector level.

When we run this selector with lein test :demo-ns-selector, we get the following printed lines for each test and namespace, the second function in this selector prints the exact same thing as our :verbose selector:

Should I select this namespace?
jinx.api.core-test
Should I select this var?
{:integration true, 
 :test #object[jinx.api.core_test$fn__4206 0x6ce1cc03 
...}

Unlike the var selector (the second function), the namespace selector only gets the namespace as a Symbol as its input (and the command line arguments to the selector function). Its only goal is to ignore all tests in a namespace based solely on the name of the namespace. That is, if we knew how to select/reject tests based on the name of the namespace a test is located in, then we could bypass loading the namespace entirely.

A common usage pattern we have observed is that after making changes to one module, we like to locally run that module’s tests before committing our changes. For example, if we are making changes in the API server module, it is useful to run all integration tests in the module:

The api-integration selector combines namespace and test selection: First selects the namespaces that start with the prefix “jinx.api”, indicating that all namespaces related to the api module will be selected, and in those namespaces, integration tests are selected by the second selector.

Leiningen’s :only Selector

The :only test selector is provided by Leiningen out of the box and as described by the lein help test command, it only runs tests or namespaces passed in as arguments:

A default :only test-selector is available to run select tests. For example,
`lein test :only leiningen.test.test/test-default-selector` only runs the
specified test.

This only selector is defined in Leiningen’s source code as follows:

The :only selector as defined in leiningen.test namespace

Ignoring the syntax quoting, the first function inside only-form takes a namespace in a project and checks whether that namespace was part of the command-line arguments to the test command. In the above snippet, vars are the command line arguments.

The second function is used to decide whether a single test var (in the namespaces that were selected by the first function) and is similar to the test selectors we have seen above. This second function selects either a test in a namespace or all tests in the namespace based on the input.

It can be a little confusing to realise that the second function alone can also choose to not run any test that comes from a certain namespace. This is because it has access to the metadata of a test var, which contains the namespace the test is located in. Although it is possible to only have the second selector function, it can only run after a namespace has been loaded. A shortcoming of the current way of expressing selectors is that there is no way to only have the first selector in the vector-based selector syntax. If a selector is expressed as a vector, the vector has to contain two elements even if the second element is a redundant (constantly true).

An Enhanced Integration Test Selector

Let us revisit the original test selector which we set out to write. Noting down our requirements:

• The selector should only run tests which have the :integration metadata.
• If namespaces are passed as arguments to the selector, then only :integration tests in those namespaces should be run.
• If test vars of the form <namespace>/<var> are passed as arguments to the selector, then only those test vars should be run which are :integration tests.
• The rules are composable, meaning that lein test :integration <ns1> <ns2>/<var1> should run all integration tests in the namespace ns1 and the test var1 in the namespace ns2 if it is marked as an :integration test (either in the deftest form or in the ns form).
• If no namespace or test var is passed as arguments, all :integration tests in the project should be run.

These requirements basically want to combine the behaviour of the :only selector and a selector based on tests’ metadata. Since selectors cannot be composed like functions can be in Clojure code, we can update the :only selector with our requirements:

The integration selector which satisfies our requirements: select tests which are marked as integration in the namespaces or test vars passed in as command line arguments like the :only selector but only run those tests which have the :integration metadata.

Intertwinement between Leiningen and clojure.test

Clojure’s clojure.test is defined as “a unit testing framework” which takes inspiration from the unit testing framework chapter in Practical Common Lisp by Peter Seibel. In effect, the cornerstone of Clojure tests is the deftest form which is a macro that evaluates to a function with the metadata :test, the same metadata we have been using thus far to select tests. Other than this ability to define a test, clojure.test provides two high-level functions which can be used to find and run tests:

• run-all-tests: Goes through all namespaces that have been loaded and runs all functions that have the :test metadata.
• run-tests: Runs tests (functions with the :test metadata) in the current namespace or namespaces passed in as arguments. This is the function Leiningen uses to pass in tests that have been selected by its selector (and removing the :test metadata from vars that are not selected).

Tests are selected by Leiningen and handed off to clojure.test. Leiningen looks at a test’s metadata, passes it to the selector function(s), and upon receiving a truthy value, calls clojure.test's run-tests to actually run the tests. Here we perhaps need to delve into a somewhat philosophical question: What is the role of a test runner in a testing framework? Borrowing from Lambda Island’s explanation of the testing ecosystem, a test framework can be composed of the following components:

1. Test syntax: How to define tests in code?
2. Test semantics: How does the system define what a test is?
3. Assertion libraries: Ways to write test statements.
4. Mocking/stubbing libraries: Utilities to help with test isolation.
5. Test fixtures and factories: Utilities for building up test data and establishing application states.
6. Progress reporters: Ways to get progress information while tests are running.
7. Result reporters: Summarize the results of a test run for human or machine consumption.
8. Test runners: Comprehensively load, detect, and run test suites.

clojure.test provides most of the above but it lacks a few test runner capabilities. Leiningen adds the following to the capabilities of clojure.test:

• Ability to discover and load test namespaces, which is a prerequisite for clojure.test to be able to run tests in those namespaces.
• Selection logic for deciding which tests to run.

Concluding Thoughts

Leiningen test selectors are easy to configure for simple use cases like running all integration tests always, but for more complicated use-cases, it is hard to configure and understand. Much of the downsides, though, as admitted by the author of Leiningen, Phil Hagelberg, is because of the lack of support for test selectors in the clojure.test test runner. From Leiningen’s source code:

This is a massive and terrible monkeypatch to work around the fact that the built-in clojure.test library does not accept patches from outside the core team. At this point the monkeypatching code below should be considered legacy, and rather than trying to improve it, all efforts should be instead directed towards improving external libraries such as https://github.com/circleci/circleci.test, which has a superset of these features and also doesn’t require you to stop using clojure.test to write your tests.

We recommend that projects override the test task with an alias that calls out to a third-party testing library instead.

Other test runners like CircleCI’s test runner and Lambda Island’s Kaocha have solved the problem of selecting tests in more elegant ways, though they come with their own set of tradeoffs which we can delve into in a future post. Some features that could prove to be supremely useful in selecting tests through Leiningen:

• Improving the test selector API: Instead of vectors of functions, a map with expressive keys as to what the selector functions do would help in the readability of test selectors. A caveat of the vector-based selectors is that the vector must have two elements, even if the second element is not necessary, even if it is (constantly true).
• Composing test selector functions: For example, the ability to run :integration and :flaky tests in the same run without writing another selector for it.
• Being able to access the metadata of namespaces without fully loading the namespace by only reading the metadata in the ns form and not the whole file.
• Reading the selector functions from a separate location other than project.clj which can quickly become cluttered due to the functions defined inside the defproject form.
• The ability to pre-process the selector arguments passed in as command line (the pre-processing, for example, converting the command-line namespaces to a set as done in the :only selector, happens once per namespace but could be computed only once for the whole test run).
• Having access to a namespace’s dependency information while selecting a test namespace similar to a tool like clojure.tools.namespace.dependency (for example, running all tests that depend on a certain source namespace).

Notwithstanding the alternative test runners and desired improvements, most of our testing requirements are fulfilled by the test selectors in Leiningen and we are thankful that Leiningen provides the ability to select tests even though Clojure’s test runner does not provide much support for it.

The source code and examples used in this post are available here: https://github.com/mourjo/jinx

Addendum: Interoperability of Selectors with Namespaces in the CLI

Even without specifying any selector, it is possible to run tests in given namespaces by passing the namespaces as command line arguments to the test task itself in Leiningen, for example, lein test <ns1> <ns2>. So how does that fit into the selector framework? As far as we can tell, this was the old way of isolating tests to a few namespaces, before test selectors started taking shape. Today, a test selector’s logic is applied on top of the namespaces passed to the test command (if no test namespaces are passed, then the test selector logic applies to all test namespaces):

lein test <ns1> <ns2> <test-selector> <selector-agrs>

The tests that are actually run by Leiningen are an intersection of the tests in the namespaces passed in the command line (or all if nothing is passed) and the tests selected by the test selector (or the :default selector if none is mentioned in the command line).

So the difference between lein test <ns1> and lein test :only <ns1> is that :only will run all tests in the namespace ns1 whereas the former will run those tests in the namespace ns1 that are allowed by the :default selector. Similarly, lein test :only <ns1> and lein test <ns1> :all will run all tests in the namespace ns1, the latter achieves this by using the :all selector provided by Leiningen along with :only.