Dynamic Testing in JUnit 5; a Practical Guide

“Two men in a steelworks with a crane moving the material over their heads” by Ant Rozetsky on Unsplash

JUnit 5 introduced several novel features, one of which is dynamic testing. I have been presenting on JUnit 5 fairly regularly, but have had trouble explaining dynamic testing. I demonstrated using a simple example, but would have a hard time providing an use case for dynamic testing. That was the case until somewhat recently when I saw this question on Stack Overflow.

When learning a new technology, I have always benefitted from following a guide that includes a practical example, as it helps me better visualize how I might use a technology to solve challenges I’m facing at my job. I’m sure I’m not the only person who learns this way, so in this article we will learn how to write dynamic tests, look at a practical example of a dynamic test, and learn about their role in our automated test suites.

Building a Factory for Tests

In JUnit 5 the first step when writing a dynamic test is annotating a method with @TestFactory. Unlike a method annotated with @Test, a method annotated with@TestFactory is a method for creating tests. This is conceptually different from the normal way of writing tests, which would be explicitly writing out the test within the method body, i.e. writing a test statically. This difference in mindset is why it can be difficult to figure out when to use dynamic tests.

To ease in to the process of learning dynamic tests, let’s first take a look at a really simple one:

We already covered @TestFactory above, lets take a look at the other elements of a dynamic test.

Dynamic tests must return either a Collection, Iterable, Iterator, or Stream, of DynamicNode types. DynamicNode is an abstract class with two default implementations of DynamicTest and DynamicContainer. We will cover DynamicContainer later, DynamicTest however is the executable test composed in a @TestFactory method that will be executed by JUnit.

For building DynamicTests the JUnit team has provided some factory methods, in the above example I am using dynamicTest(String displayName, Executable executable). displayName is the name that will show up in test reports¹ and executable is a method reference or lambda of the code that will actually be executed in the test.

Writing dynamic tests as can be seen above is pretty simple. However the above example doesn’t provide much guidance on how dynamic tests might be used to ensure our applications are being properly covered by automated tests. Let’s take a look at a more practical application of a dynamic test.

Writing Dynamic Tests to Reduce Toil

It’s not uncommon the need to write a service that communicates with multiple clients, and those clients communicating in different data formats. To demonstrate the power and utility of dynamic tests we are going to walk through writing a dynamic test that scans an application for controllers, check if those controllers have a POST endpoint, and then verify the POST endpoint can consume both XML and JSON.

Writing out tests to ensure every POST (and PUT) endpoint in a RESTful API accepts all required content types can be tedious and error prone. It is easy to imagine a scenario where a developer doesn’t properly configure an endpoint and then either not write the tests that validate the endpoint for content types it accepts or have a copy/paste error (doesn’t update the test to call the correct endpoint). There is a good possibility this bug won’t be caught until it reaches production. Let’s look at how a dynamic test can make this process easier, checking all POST endpoints to verify they accept required content types, and as new POST endpoints are added, automatically checking them as well.

Below is a dynamic test that accomplishes this, lets walk through what this code is doing:

The dynamic test can be broken down into roughly three steps:

1. Scan the classpath for controller classes
2. Mock out controller dependencies
3. Build the tests to call the POST endpoints

To handle the first steps we will use of Spring’s ClassPathBeanDefinitionScanner. The classpath scanner has been configured to look for any class annotated with @RestController and to scan from the root package of the project. This should find any controllers within the project, ClassPathBeanDefinitionScanner can be easily configured though to narrow or widen its search criteria as might be needed in your project.

Once all the controllers in the project are found, we must figure out any dependencies the controllers require and create mocks, in this case dummies, of those dependencies. While this task could be accomplished if field injection was being used, it’s made easier by using constructor injection. Using constructor injection instead of field injection helps to define the required elements a class must have in order to be in a usable state. It’s for this reason Pivotal engineers and developer advocates recommend using constructor injection.

For the third step, we need to create the tests themselves. The first part of this step is building the URL that MockMvc will call. The code is checking to see if a @RequestMapping has been defined at the class level providing the base of the URL for the controller. Next we will check if any methods in the controller have been annotated with @PostMapping. We are assuming proper RESTful practices, so the POST endpoint should reside at the base of the controller URL.

Finally we are using mockMvc to execute the POST call and what we will be checking for is that a 415, media type not accepted, is not being returned. By checking that a 415 is not being returned, we make the process of writing the dynamic test easier as we are able to use dummies for our controller dependencies. If we were to check for a 200 (OK), this would not only add a lot of overhead, as the mock dependencies would need to have behavior, but also distract from the purpose of the test, we are checking that POST endpoints are accepting required content types, not testing any of the code within the endpoints themselves. Applying the single responsibility principle to tests as well, makes writing tests easier and automated test suites cleaner.

Executing the above code will return four hopefully passing tests that look like this:

This isn’t too bad of test reporting, but as a RESTful API increases in complexity it might become a little difficult to read. Let’s look at how DyanmicContainers can be used to make the test report a little easier to read and scale better as new controllers and POST endpoints are added.

I cut out some of the code above for brevity, but what has been changed is in the method signature a Collection of DynamicContainers is being returned instead of DynamicTests. Next we have a variable dynamicContainers that is also a List of DynamicContainers. Within the for loops where the tests are being created there is now a second List, dynamicTests, and we are adding the tests we are building to that list. Finally we are adding dynamicTests to dynamicContainers.

With those changes made, when we run the test again the report we get should look something like this:

Now the test for each content type is nested under the the base URL of the controller. This will help keep the report readable as new controllers and POST endpoints are added.

When to Use Dynamic Testing

The benefit dynamic testing offers over statically writing tests doesn’t come in the form of time saved, but from avoiding toil and saving mental capacity. In the example above of verifying that POST endpoints can consume JSON and XML, it would be laborious writing out a set of tests verifying as such each time a new POST endpoint is added. This task would often be accomplished instead by copying and pasting the tests that were used to verify another endpoint. As mentioned earlier it’s pretty easy to imagine then a developer forgetting to update those tests to call the correct endpoint.

This leads to what I think is the biggest tell (or smell if you like) for when to use dynamic tests, copying and pasting tests from other test classes. Other common testing needs for example security and database communication often have a series of standard test; if this role is present allow, if not deny, if all the required fields are present update the table, if not throw an error. Developers often save time by copying, pasting, and updating tests, but it would be better to write out a single dynamic test that will then automatically generate new tests verifying common behavior as new endpoints, tables, security roles, etc. are added.

Conclusion

It’s exciting to see so many new possibilities that come from testing with JUnit 5. In an earlier article I covered how test interfaces introduced in JUnit 5 could be used to make testing easier while also encouraging developers to follow architectural patterns. As the JUnit team continues adding new features to JUnit 5 I’m sure we will find new an exciting ways to write tests that weren’t possible before.

The code in this article can be found here:

Simple dynamic test: https://github.com/wkorando/junit-5-simple-demonstrator

Practical dynamic test: https://github.com/wkorando/WelcomeToJunit5

1. There is currently a bug in surefire where the display name of dynamic tests is not being used.