Test like an Egyptian
It seems to me that virtually every developer I talk to these days is sold on Test Driven Development (TDD) and unit tests are on the rise. Now it’s time to keep building on those skills. I have been working closely with our QA department at work trying to solidify the concepts behind a well balanced testing pyramid. The question I’ve been faced with explaining is what tools do you use for the tests at the different levels. Also, where do you run these tests?
At our shop we are running on the .Net infrastructure with our back-ends implemented with FubuMVC and are in the process of migrating to ASP.NET Core. We also maintain a custom CRM solution which is currently a WPF application for our call centers. We also run on a Service Oriented Architecture (SOA) also called Microservices with dozens of web applications and services.
My goal here is to highlight a list of the tools we use along with a few alternatives that would work just as well. I’m finding that many individuals in our organization are unfamiliar with what options are available, and I hope to keep them from re-inventing the wheel as they work to fill out tests for the full test pyramid.
When it comes to unit testing you’ve got a lot of options. Most of the test frameworks you can use will work, so just pick one and run with it. However, there are some things you can do to reduce your coupling to a test framework so it’s easy to switch to a different framework if the one you chose is not cutting it. You can do this by choosing an assertion library that is not the same as your testing framework.
You are going to have the best success in writing unit tests if you are follow SOLID design patterns. Or in general architecting your code to achieve loose coupling and high cohesion will set you up so writing tests is easy.
In C# you may want to look at using an IoC container to help you construct loosely coupled dependency trees. Mocking libraries are also useful as they allow you to fake difficult integrations or infrastructure outside your control. Use caution that you are not over mocking in your tests. I have seen too many tests where an engineer was just testing his mocks and not the actual code.
Again this is more of a discipline to follow, since testing frameworks like NUnit come with their own assertions.
Assertion Frameworks: (There are arguments to avoid some of these because they modify the base Object prototype to achieve what they do.)
This portion of the testing pyramid is tricky because it really encompasses a few different ideas. At this level you should be testing Components, Component Integration, and APIs. So, if a set of classes or modules are supposed to work together, then test that they work together as expected. You don’t want the same level of granularity as you do with your unit tests, but you want enough of these tests to have full confidence that those modules work well with each other. You will also want to test your APIs, for example if you are building an SPA Web Application, then run direct HTTP requests against your backend just to test the endpoints individually. Again you are just looking for enough confidence that the application is putting all the modules in the right place when running your application.
System / UI Testing
These are traditionally your most brittle and slowest running tests. They test the system from end to end. So, for example, can a user actually sign-in, run through a shopping cart, checkout, and view their pending and past orders? That would be important information to know before you deploy your code into a production environment.
I intentionally left off frameworks for Integration and System testing because there’s some overlap in the frameworks you can use to facilitate them. For some of these tests you can just use the standard Unit testing frameworks above. If you plan to do that though you should separate out these tests to run as a separate process. Since these tests tend to be slower, it’s advantageous to only run them when the unit tests have successfully completed. This will give you a quicker feedback loop.
There are testing frameworks though that are better suited for tests at this level. For example Storyteller, Canopy, or Cucumber are great for these kinds of tests. Each have their pros and cons, but one nice feature of these them is that the Behavior Driven Development (BDD) language used to compose the tests are easy to read for business folks.
For harnessing the UI in testing browser based applications there is really only one option which is Selenium. I would recommend that you avoid using their test recorder since recorded tests will be the most unreliable you can get. Instead make use of the Page Object Model and other considerations they advise. I would also recommend you check out a past presentation I gave about Selenium. There are some async testing concerns to take into account which I highlight in that presentation.
For testing WPF desktop applications take a look at White, and follow some of the same considerations as you would for Selenium.
Performance and Security Testing
The testing pyramid does not explicitly state where these tests live. They should live at all levels of the testing pyramid! At the unit test level you shouldn’t need to test every component for performance, but you would want to test a few of the critical or high throughput code paths. Benchmark testing is what you would be targeting here. For security testing you would want to test your serialization, authentication, data validation points, and static code analysis. There are tools for all of this, but these are outside the scope I’ll cover today.
Once you have tests written you will want them to be run as soon as a developer commits the code and pushes them up to a central repository. You should run your Unit tests first and fail as fast as possible if there is an issue. You want to have a tight feedback loop to stay as productive as possible. As Integration and System tests grow you may want to look at options to fan out your tests and run them in parallel.
But this is a post about testing so why talk about DevOps? DevOps is an industry buzz word, but at it’s core it’s a culture to be followed by your development organization. There is the phrase “Jack of all trades, master of none is oftentimes better than a master of none”. At it’s core the DevOps movement is to broaden the skill sets of your specialists so they understand the entire picture when developing software. Testing skill sets should not be isolated to your Software Development Engineers in Test (SDETs). Every software engineer can and should assist to build and maintain these automated tests.
In the end are you building a quality pyramid, a foundation or an hourglass? If you only have unit tests you have a foundation. If you have unit tests with a lot of brittle UI tests, then your working with an unbalanced hourglass.
At the end of the day the purpose of testing is to have full confidence that the software you produce is and will continue to be ready for production at any time. If you don’t have an automated test that provides that confidence then add the test that will give you that confidence.