Practical shift-left on testing
Should You Unit-Test in ASP.NET Core?
If you find the opinions I express in this post to be radical, you will find the follow-up outrageous.
The Startup pattern, larger units, fewer test doubles and a bit of BDD thinking.
If you have not yet read Andrew Lock’s post Should you unit-test API/MVC controllers in ASP.NET Core? then I suggest you read it without undue delay. In this post, Andrew brilliantly argues that you should not unit test controller classes.
The astute reader will notice that the title of my post is an abbreviation of the title of Andrew’s post. This is a case of less is more, since I believe that Andrew’s arguments can be stretched much more, as hinted by my shorter title.
But before I begin with a stretching exercise, let’s first note that a unit test in Andrew’s post will have a controller class as its subject under test (SUT) and all the dependencies of the controller class are substituted by test doubles, whereas an integration test will also have a controller class as a SUT, but special dependencies, specifically the ASP.NET middleware, are not substituted by test doubles.
Both types of tests are isolated and run in-memory. In the case of integration tests this is achieved with the magic of an in-memory test server and the WebApplicationFactory<> generic class from the Microsoft.AspNetCore.Mvc.Testing package.
In my world, an isolated, in-memory test is simply a unit test, perhaps with a large unit. But I will keep the above distinction between unit and integration for the duration of this post, since it is quite common in the .NET Core world.
Let’s see what happens if we write isolated, in-memory tests but avoid substituting production code with test doubles as much as possible.
What is important is to try to stretch Andrew’s arguments and make a case for not unit testing at all. More precisely, let’s see what happens if we write isolated, in-memory tests but avoid substituting production code with test doubles as much as possible.
The discussion does not have to be restricted to ASP.NET Core services, but let’s keep a narrow focus for now.
What would be the SUT in such tests? What would we have to mock?
Well, the entire service would be the SUT, of course. And since we have restricted ourselves to isolated, in-memory tests, we will have to use test doubles for out-of-memory dependencies, such as other services, databases etc.
This is not much of a restriction, since many microservices depend on a data storage and few other services, so we will often end up with less than a handful of test doubles for the entire set of tests.
It is even, in many cases, possible to run such dependencies in-memory. For example, if you use the Entity Framework for database access, you can use an in-memory database in your tests. This approach has both pros and cons, which I will not focus on in this post.
Wait, but why?
Before I show the details of how to practically have the entire service as the SUT, let’s discuss why I see it as a worthy goal.
First of all, there is a problem with having too isolated tests. If the SUT is a single class or method, then the dependencies will most often be part of an internal implementation of your service. If you substitute internal dependencies with test doubles, then you will have to redo these when you refactor your internal implementation. And we all know, that you have to do that, or your code will rot.
That problem is amplified by the number of test doubles you have. You are likely to have several test doubles per test. And if you are serious about low-level unit testing, you are likely to have many tests as well. That is a lot of test maintenance work you are looking into! If you apply the Object Mother pattern in some form, you may be able to handle all these test doubles slightly better. But since they will tend to be low-level and specific, you will probably just make disaster slightly less disastrous.
But why is a lot of work disastrous? Naturally, you are not afraid of hard work and testing is important, right?
Low-level unit testing is disastrous because the concept to a large extent relies on a false premise, and thus gives a false sense of security. Testing is important, but unit testing is most often not testing! Most unit tests are not real tests! You don’t have to take my word for it, as greater minds have already argued the point. James O Coplien is probably the most outspoken of those. You have probably read his post Why Most Unit Testing is Waste. But if you have not read his seque, I suggest that you do.
Most unit tests are not real tests!
All of the above is a known point of view, although it is still being debated.
It is less known what happens if you go to the other extreme, having the entire service as the SUT. In Martin Fowler’s terminology, having very sociable unit tests.
The obvious effect is the opposite of the low-level approach above — you don’t substitute internal dependencies with test doubles, so your test code is less fragile, it is more resilient to refactoring of your production code. And you have less test doubles to maintain — the Object Mother pattern will be very useful again.
The less obvious effect will emerge when you combine your testing effort with BDD thinking. Don’t think about what class or method to test next, but think about what behaviour to verify next.
With focus on behaviour, as defined by requirements, your tests can be seen as real tests which verify actual requirements defined by stakeholders.
The focus on behaviour means that the set of tests, if properly chosen and documented, can show stakeholders what parts of requirements are tested. Even better, if you do it right, you can show what you have chosen not to test and why.
This way you can create visible confidence for stakeholders.
While testers will see your service with 3000 low-level unit tests as completely untested, they will more likely see tests as described above as something they can work with. Even continuous cooperation is within reach. That's how you do continuous testing in the context of Continuous Delivery.
I believe the above shows that the idea of very sociable unit tests and large units has some merit, so I will put QED here.
Show me the code
Before we dig into the actual C# code, I want to briefly mention the magic that makes it all work — Dependency Injection (DI).
DI is not part of .NET Framework, but it is very much integrated into .NET Core. That’s why Andrew easily could include the ASP.NET Core middleware in his tests. The principle behind the following will work for .NET Framework and other code as well, but it is quite easy to do with ASP.NET Core, which I will demonstrate shortly.
The reason this works, is that the Microsofties have followed DI best practices this time. They have probably read Mark Seemann’s book.
What I particularly like is that Microsoft has a standard pattern with a Startup class. The ConfigureServices method is essentially what Mark Seemann calls the composition root, i.e. the place in which you declare the object graph to the DI container.
Without further ado, here is a simplified Startup class from a sample C# project. The full sample project source code is in a public GitHub repository.
AddSingleton statements register two facade interfaces representing the external dependencies. The AddControllers statement registers a group of dependencies; this is a commonly used pattern.The Startup class is referenced in our service entry point,
We want to add test doubles for our two external dependencies. With ASP.NET Core, this can be done with the following two statements,
The real magic in our test project is introduced with the WebApplicationFactory<> class. It is used to create a test server for our tests, which is important.
It is equally important that it allows us to base our tests on the production code composition root, only replacing a few external dependencies with test doubles.
Here is how that can be done (I prefer to put it in the test project entry point),
You get all this for free from Microsoft. It’s well documented, so there is no need for me to repeat the details here.
But I will stress the dogma rule that you must keep in mind for this to work,
The composition root (
ConfigureServices) is where you declare services (compose object graphs).
If you find that you can follow the dogma rule, then you are in a good position for shift-left on testing. In case you are not entirely sure what shift-left on testing is, you should read Lisa Crispin’s post Shifting left & right in our continuous world. In a nutshell, shift-left on testing means that you need to test early and continuously in order to do continuous delivery (CD). Simply running automated tests in a pipeline is not enough, it’s the continuous cooperation which does the trick.
My corollary to the dogma rule is,
Don’t do anything else in the composition root.
If you instantiate and initialize services in ConfigureServices you violate the rule. If you, directly or indirectly, use external dependencies in ConfigureServices you severely violate the rule.
The consequence of breaking the dogma rule can be that you cannot (easily) substitute production code services with test doubles, and thus you will have a hard time putting certain parts of your production code under test.
The consequence of severely breaking the dogma rule by using external dependencies in ConfigureServices is that you make it tricky to properly do isolated, in-memory testing at all.
If you have to break the dogma rule when you include a nuGet package you need to provide feedback correspondingly to the package author.
Naturally, you will have to be pragmatic. Sometimes, it is OK not to be able to substitute production code services with test doubles. Sometimes, it is necessary not to piggyback on the production code composition root, you may need to recreate your entire production composition root in your test projects. I have been there many times with brown-field projects. But the tests tend to be less valid, and there is a lot of initial work and subsequent maintenance involved, so don’t go there unless you have to!
If you find that you can follow the dogma rule, then you are in a good position for shift-left on testing.
Here are a couple of final comment on the code I have shown so far.
If you have read a few of my previous posts, you will know by now that I am a sucker for consistency. I do not have code in each test for creating and customizing WebApplicationFactory<> instances, I consistently use a single method for that, referencing a single test composition root for an entire test project. See the example project, the file AssemblyInitializer.cs, for details.
In my last post, I also argued that external dependencies must be defined in terms of values passed across external dependencies, rather than in terms of specific test doubles and mechanisms. This is why the test composition root in the example project looks differently from what I described above.
Conclusion
Larger units, fewer test doubles and a bit of BDD thinking goes a long way towards providing real confidence with real tests and continuous collaboration among developers and testers.
We have used the approach with all green-field projects in Saxo Bank’s OpenAPI for a while. Our experience is that we get all of the above with a low maintenance burden and little friction.
But is it all rosy, are there only pros and no cons?
There are hurdles of course.
First of all, when a developer writes a test, she will have to think about its context (Given in Gherkin) in terms of external dependencies. That requires more brain cycles than simply passing in 42 for an int parameter to a method which is tested in isolation.
This is a relevant concern. However, in most cases it turns out that the extra brain cycles are well spent, since it means that the developer will get a better understanding of the code. It is an important goal of testing to learn more about the code and how it works. And if the method in question will never, ever receive 42 in production, but will do special processing of this important value, there is a risk that the low-level test is testing dead code. I would rather detect and eliminate dead code as early as possible.
It is an important goal of testing to learn more about the code and how it works.
Another relevant concern is that not all code is well-written. This means that the approach is cumbersome, but not impossible. Using DI best practices across the code base helps. Even if only external dependencies are handled by a DI container you are still in a good starting position.
We have also used the approach with large brown-field projects. It required much initial work, because we often had to recreate the production composition root in test projects, but looking back it was still worth it.
A final concern is how to do TDD with this approach.
First of all, don’t do TDD, do BDD. Dan North was friendly enough to evolve TDD into BDD, so let’s not be ungrateful.
Secondly, rather than having an inside-out or outside-in approach when you do your steps, try to do both. For each iteration, implement the simplest possible vertical slice, including access to any external dependency. A minimal slice is what you test in each iteration. Alternatively, write whatever low-level tests you need with whatever approach you prefer, but evolve the tests into integration tests — or delete them in the end.
Rather than having an inside-out or outside-in approach when you do your steps, try to do both.
Oh, I almost forgot the two most important concerns— not everybody agrees with the bold statements I have included in this post, and some people prefer not to change habits.
Well, I suppose you cannot win them all.
I would like to thank you for reading this far. I hope you find my post useful.
