How to use Model Based Testing to increase test automation coverage?
When we think about software testing the first thing that comes into our minds are manual validations for which we need to identify preconditions, steps and desired outputs so we are able to execute them one by one every time we need to run a test suite. This practice is effective and has been in use for several years, but it is very expensive and hard to scale because it depends on manual work that needs to be done by real people, so if we want to move faster we need to hire more QAs.
With the purpose of reducing our costs and accelerate the time to market, companies are implementing test automation so they can run their tests suites automatically on CI/CD pipelines, but the testers still have to design the tests cases, with the proper preconditions, steps and expected results before they can actually start creating the test scripts and begin executing and reporting them.
So even having automated testing in place we still need to deal with some manual tasks and sustainment, that’s why we need to think in a new abstraction layer which allow us to have less manual work and more consistent and robust test cases, something that can be translated in better coverage, less sustainment and reduced costs.
So what if we are able to stop designing test cases, to stop coding our scripts one by one, but having more coverage, more velocity and less flaky tests at the same time? We all know that fault free software is not achievable, but that does not mean that we can not get closer to it every day, and this is where model based testing enters in the scene, because it introduces this new abstraction layer that allows us to reach almost 100% of coverage with less effort.
Model Based Testing is a new way of designing our test cases, as we do everything designing a model instead of using the actual application. The model consists of a graph composed by what we call Vertexes and Edges, which are basically assertions or verifications and state transitions respectively, that simulates our system behavior. We can have as many Vertexes as we need, relating them with Edges, for creating a digital twin that models the application under test but in a very simple way.
You might be wondering what’s all of this about, don’t worry, I was in that place the first time I heard about this, so let’s start working on what we need so then we can move forward and introduce an example.
First of all, I would like to tell you that Model Based Testing allows us to start testing as soon as we have the business rules, with no need to have the actual application, so we can say that it is a mix between behavior driven development (you can have the model and execute the tests cases and then start coding the application under test) and Business Driven Development (Each graph is a business representation of the application under test).
So let’s suppose we are in a new project and the Product Owner send some definitions for a simple app that allows the user to press a button, but only it is a registered user, so the application flows are something like this:
At this point we have the definitions, but there is no app yet. As we can see the app allows us to login, recover our password, press the button and logout, so thinking about creating our test cases we can imagine a few just by looking to the diagram, like pressing the button right after a successful login, or login in, login out, login in again and pressing the button, etc. We can have too many paths from one point to another and many different ways of achieving the same.
Then, our next step is to model the application under test, for what we need to create a graph (the model) with all the vertexes and edges, so it should be something like this:
As you can see, each square is a Vertex and each arrow is an Edge, remember that a Vertex is a verification or an assertion, it is the place in our code where we validate that we are in a specific page, that we see an element or that an API call returns a specific value, and the Edges are actions and transitions, like pressing a button, closing a window, calling an API, etc.
In this case we modelled a very basic application, but good news is that for more complex systems we could create a model of models, where a model can represent a specific subpart of our application and it can connect with another model representing another subpart of the application.
Another important thing that we can see at this point is how the test design is totally independent of the actual code implementation, we do have a model representing our test cases but nobody wrote a single line of code. This is very important for testers as it allows them to involve people who don’t know how to program to the test design allowing them to just draw the models in some tool.
At this point we have our application modeled with all the actions and transitions that are available to the final user, we can generate the tests until we have the desired coverage by just moving around the Vertexs with all the combinations available to the user.
Now, let’s suppose that the application actually starts being developed and we have a real product to test, one that is the same we modeled when creating our graph so we have a product and a model, what we need to do now is to implement each Edge and Vertex on with our code, which actually execute the test steps, something similar to what we do with BDD when we match the feature files with the step definitions. For this we can use api calls, selenium, appium, whatever we want to use to code our graph transitions and statuses.
Once we implement the code associated with the model, we can use the model to execute all the possible combinations to perform a task, like in our example above, where we can perform the same activity like pressing the button by taking different paths. It’s important to highlight how our test suites are automatically generated from the model.
This has some advantages compared to traditional test automation models, such as having less flaky tests (flows are automatically generated) and more combinations, on the other hand we don’t have the control of the different flows, so if we find any error we should replicate it manually.
Of course, we can integrate this approach with a continuous integration server, but we need to be careful and take into account that executions might be too long so we would like to define different types of test suites, like smoke tests, functional and not functional. How can we achieve this? Well we can execute our models with parameters that allows us to specify how we want to execute the tests over the model, for example if we want to execute a smoke test we can tell our model to execute tests using the shortest path to specific nodes, if we want to execute more functional tests we can configure our model to execute tests until every node is visited at least once, or we could execute an overnight exhaustive test for 5 or 6 hours, something that will guarantee that new permutations will run every night. As we can see, the amounts of permutations can be huge, so it’s recommended to properly log the flows during the test executions to be able to reproduce the paths in case we found any error.
Model based testing is a whole new approach that changes the way we test our applications, and why not, the way we develop them, it’s up to you to see how much effort you want to invest on this approach and to analyze if it suits your needs.