Writing Expressive Tests

Write tests that will ease your mind

One of my biggest gripes with so many tests I see nowadays is how they fail the intent they are meant to further: Give clear feedback on errors.

You ask, how so? I tell you, how many times have you seen a test failing and not know what was wrong until you dived deep into the test and had to read it line by line until you found the issue? And how many of those times was the issue the test and not the implementation?

I have experienced this too many times, and over time, I have found that extensibility and expressiveness can only improve test frameworks.

To show that, I will provide some examples of how Hamcrest (and Hamcrest Compose) and AssertJ can help you write more expressive tests over JUnit and how easy it is to set them up.

Express what you expect clearly

One helpful tip for writing tests is to make them easy to read and understand. For now, we will use the above example and keep the implementation details hidden because what matters the most in a good test is the superficial layer with which the developers interact.

Naming

In the example above, we have one test case and this test case… Instead of me explaining here, what about we read the test’s name?

The test name is the first place in which the developer should express his expectations, a good test name cuts short the need for future explanations, better yet if written using real English, with spaces and punctuation.

Remember: Although not as directly as in Kotlin, it is still possible to write more human-friendly descriptions with @DisplayName while using Java with JUnit5.

Assertion

An assertion should be as direct as possible about what is happening. In this case, our description will have a set structure: [$randomUUID] $name: $points. This means we can’t validate the UUID because it is intended to be a random identifier.

In this case, we have three examples:

• JUnit: We need to coerce our result to ensure it fits an equal assertion. For this coercion, we have to filter out data that we can’t validate, which in this case would be a random UUID at the start of the description. Without it, we can validate the expected text without the presence of the UUID.
• Hamcrest Regex: Similar to JUnit, but in this case we still validate the whole description, including the presence of said UUID, but not its value.
• Hamcrest Custom Matcher: The next level after Hamcrest Regex hides away the fact that UUID exists from the test body and keeps it in another method.

In the end, they are 90% similar, with differences mostly in readability and the amount of information exposed.

Also, we should consider if we are also testing unrelated matters when writing a test. Is it relevant for me to check every field of an object? Usually the answer is no. If we are testing a feature that adds points to a contestant, there is no need to check if the participant name has changed, unless we have a really good reason for it.

Test Results

And then, there is another place where clarity is essential: The test results. An engineer should be able to use the error message to understand the error. Here are the error logs for them when someone uses the wrong number of decimal places in the description:

We have a much more complete error message for our Hamcrest examples. Also, after testing a case where no UUID was printed, I discovered that my JUnit test case had a bug because the replace function returns the same string if there is no possible substitution, giving me a false positive: The test succeeded when it should fail!

Of course, this is not a problem exclusive to JUnit! But the fact that it happened only when we had to coerce our result indicates that more expressiveness can reduce false positives.

Examples

Asserting a complex object

Now that we have discussed the advantages of being as expressive as possible for a basic test, we can explore tests with multiple assertions to validate the consistency of a result.

I know this is a taboo for some people. Still, there can be value for that, as long as we choose our test cases well and be sure that we are validating effects from the same action, even more so when working on integration tests, where it could be too time-consuming to run the same setup multiple times to execute one validation per test.

Again, AssertJ and Hamcrest are similar, but in this case, it is more important to show both of them here for comparison, although they look very similar, they have different results when considering that both statements should fail:

As can be seen by that message, AssertJ and JUnit stop at the first step and don’t report the next failures; while using Hamcrest Compose, we can receive a complete report about failures.

Just as we don’t want to run the slow test multiple times for multiple assertions, we don’t wish to rerun the test after solving the problem in the test failure message just to discover that we have another issue that could have been reported on the first execution.

On AssertJ, there is a way to use that, using SoftAssertions, but it does come with some consequences:

• No more custom Asserts, unless you also extend SoftAssertions to add your Custom Assertions, as I have been using previously.
• If the user forgets to call assertAll at the end, the test can give a false positive, as the assertions were never executed.

Although I have listed some disadvantages above, SoftAssertions can still be a powerful tool. It just comes as a sadness that we need to choose either to use SoftAssertions or our beautiful custom assertions that AssertJ is so proud of allowing developers to use.

Asserting multiple items

Another useful test case is checking a list after processing. In this case, we should get good feedback on which items don’t match.

I would argue that for this test, we just need to check for the pairing of a name to a score so we know that the right participant has been updated and that it has been updated with the right score.

Again, we encounter problems with assertions that check multiple items in JUnit, as we have to coerce the data so we can check it. Another solution would be to coerce the data to a single value, like the description, or to just concatenate the name to the score and check it, but those are not as expressive as having the expected operators.

AssertJ and Hamcrest take a more expressive approach, using an assertion lambda and a custom matcher.

Now let’s explore an error scenario in which the function forgets to correct the score before sorting the list:

In this case, we can see that AssertJ has the most complete error report by default while missing the fact that it didn’t look at the score after the names validation failed, which could be solved with a SoftAssert.

On Hamcrest, we can see a more direct error message, but not without its issues:

• The contains matcher fails on the first item of the list instead of going through the whole list.
• The allOf matcher also skips the score check if the name is wrong, while the compose matcher formats in a non-intuitive way.

Those can be easily remedied by creating your own contains and allOf matchers, but this is something that should be expected out of the box.

But how do I do those?

After focusing so much on the surface level, now I would like to focus on how easy it is for the users to get to this level of expressiveness.

AssertJ

The easiest way is AbstractObjectAssert class, which will give you all the basics and allow you to create your new assertions as new methods. My implementation is shown below:

With a structure like that, we can already use our custom assertThat to make assertions to Participants. Just take care to remember to import it, or your test will not see the new methods you have created. If you would like to use SoftAssertions too on top of it, you would also need to create ParticipantSoftAssert separately, adding your desired assertions. Also for assertions in lists, a lambda would need to be created, as seen in participantWith.

Hamcrest Compose

As for Hamcrest Compose, everything we need is to create helper methods in the current test class or in a Util file:

With this structure, we just need to call those functions as a second argument for our assertThat, and we are set.

Looking at both, we can see a difference in philosophies. While Hamcrest prioritizes extensibility, AssertJ prioritizes discoverability. Both can do both, but their focuses are in different places.

One pet peeve I have with AssertJ is that I believe the overhead of extending it discourages developers from creating more expressive assertions, while this is a lot lighter and more flexible in Hamcrest.

As an aside, if you do want not to use Hamcrest Compose, there is also a way to explore the same effect as hasFeature with hasProperty. But with the caveat that you throw away type safety with that. For example:

This would allow for the same comparison using only the main library but allow us to use incompatible matches for validations, like using a Matcher<Double> to validate a String property. Another disadvantage of this matcher is that it doesn’t allow for validations of things other than fields.

Conclusion

Those are some concepts I use when setting up expressive tests. There are multiple smaller tricks to make tests more readable, but information clarity and completeness are the most important ways for me to judge the quality of a test.

Your time as a developer is valuable, and every minute you spend writing expressive tests can be recovered later for all future developers. This will save time when a test invariably breaks during a refactor or when a new feature affects another.

I have also shown the same samples for two different tools that can be used for those validations, AssertJ and Hamcrest, which are almost identical at the superficial level.

Although I prefer Hamcrest to AssertJ, one important remark when choosing it is that it doesn’t seem to have had updates in the last two years. Most people would consider it a dead project, but I believe its base is still solid enough, even without any new updates.

For the code resources, you can visit this repository to see the implementation details and structure, with more test samples that would be too excessive for this article.

Do you think you have what it takes to be one of us?

At WAES, we are always looking for the best developers and data engineers to help Dutch companies succeed. If you are interested in becoming a part of our team and moving to The Netherlands, look at our open positions here.

WAES publication

Our content creators constantly create new articles about software development, lifestyle, and WAES. So make sure to follow us on Medium to learn more.

Also, make sure to follow us on our social media:
LinkedIn — Instagram — Twitter — YouTube