Practical Model-Based Testing — Say “Hello MBT”
Imagine — you start your development sprint with ready-to-run automated acceptance test, with all the test-cases already generated before you develop any code. Does this sound like a dream? Today, I’m going to show you how GraphWalker, a Model-Based Testing toolset, makes Continuous-Testing a reality. And no, it doesn’t require Test-Driven Development processes.
In this post, you’ll learn how to build your first Mode-Based Testing automation, using Java and GraphWalker.
*It’s important to note that it is possible to use GraphWalker for testing other language codes by running it as a service
Let’s begin by providing an example of what we are going to build.
All the detailed explanations of each step will be given later in the post when we go through the development steps themselves.
Imagine that your team’s development workflow is as follows:
- The PM (Product Manager) brings some new requirements:
- A pedestrian traffic lights
- Begin with a Green light
- Toggle between Red and Green lights
2. You formalize the requirements into the following model:
3. The Test-Generator automatically generates a full coverage test-case suite:
4. At this stage you can execute those automated generated test-cases ….
“But wait!”, you would probably say, “Those changes are not implemented yet”.
5. Go ahead and write the implementation. You can run the generated tests during implementation and stop writing code when all the automated test-cases have passed.
In the example above, the Model created from the requirements is also your automated executable acceptance test, which also becomes your DoD (the feature’s Definition of Done).
You have executable acceptance test, right at the beginning of the sprint, and before you write a single line of code.
If this looks like a TDD (Test-Driven Development) to you, you are right. But MBT (Model-Based Testing) not only removes the dependency between test-cases and implementation, it actually creates a very strong dependency between the test cases and the business requirements.
Code [ → Driven By → ] Test [ → Driven By → ] Business Requirements
Business Requirements, Test and Code, are strongly connected (unlike offline test-cases that are disconnected from the requirements and code) meaning that with a proper SDLC process, the code must be updated with any change to the formalised requirements (the model), otherwise the continuous automated generated tests, by using MBT, will fail.
Such development process can also be seen as BDD (Business Driven Development and Behaviour Driven Development).
But you don’t have to change your development processes to TDD or BDD in order to use MBT — it can be used on an already implemented project.
How Model-Based Testing Can Help
These days when more and more applications are served from a cloud, the development organisations constantly work on improvement of their velocity so that they can enable multiple deployments a day. One of the largest impediments in achieving this goal is testing. Producing reliable and extensive test automation that has also high code coverage, can be resource and time consuming.
I believe that with the increase in recognition of the importance of Continuous -Testing (CT), more and more companies will use MBT as their test automation solution in their CI/CD pipelines.
By connecting an MBT framework with an automation test execution platform (e.g. Selenium), we can achieve automated execution of those generated test cases.
Code integration tests can be executed directly from a Model-Based Testing framework, like GraphWalker.
Time to Code
Throughout the rest of this post I will discuss all the steps necessary to perform in order to complete the Traffic-Light example I described above.
We start by setting up the development environment and begin with the Model-Based testing tool: GraphWalker
GraphWalker is an open-source MBT, driven and developed by Kristian Karl and others (I also used Kristian’s previous MBT tool, org.tigris.mbt, about 13 years ago).
GraphWalker takes the user-friendly-first approach, sacrificing some of the MBT more advanced capabilities. This approach is what makes MBT with GraphWalker a breeze.
- Workspace
Create a workspace where you will have the project folder, as well as a lib folder that holds GraphWalker’s jar files.
$ pwd
/Users/me/workspace$ mkdir lib$ cd lib/
2. GraphWalker
Download GraphWalker’s jars into lib.
$ wget https://github.com/GraphWalker/graphwalker-project/releases/download/4.2.0/graphwalker-studio-4.2.0.jar$ wget https://github.com/GraphWalker/graphwalker-project/releases/download/4.2.0/graphwalker-cli-4.2.0.jar
3. Java
GraphWalker needs JDK 1.8, preferably Oracle’s (especially if you want to build the jars from source): https://www.oracle.com/java/technologies/javase/javase-jdk8-downloads.html
An alternative Java could be AdoptOpenJDK at: https://adoptopenjdk.net/ (unless you want to build GraphWalker from source).
If Java 1.8 is not your system’s default, after installing JDK 1.8, you have to run the following command at the beginning of every GraphWalker session (or the proper command for your system to have a shell session with Java 1.8)
$ export JAVA_HOME=`/usr/libexec/java_home -v 1.8`4. Maven
We will be using Maven, as this is GraphWalker’s default implementation environment.
On Mac you can install Maven through Homebrew:
$ brew install mavenI recommend on having the ‘jq’ Json utility as well:
$ brew install jq5. Default Maven GraphWalker Project
Create the Maven HelloMbt project and cd into the HelloMbt project’s directory:
$ pwd
/Users/me/workspace$ export JAVA_HOME=`/usr/libexec/java_home -v 1.8`$ mvn archetype:generate -B \
-DarchetypeGroupId=org.graphwalker \
-DarchetypeArtifactId=graphwalker-maven-archetype \
-DgroupId=com.mbtroads -DartifactId=HelloMbt \
-DarchetypeVersion=LATEST$ cd HelloMbt
6. GraphWalker Studio
Change the default model that was generated by GraphWalker.
Launch the GraphWalker studio:
$ java -jar ../lib/graphwalker-studio-4.2.0.jarOpen your browser at: http://localhost:9090/studio.html
In GraphWalker-Studio click the ‘Open’ icon on the left, and choose the model file:
...workspace/HelloMbt/src/main/resources/com/mbtroads/SmallTest.json
For the simplicity of the “Hello MBT” example that we build today, we will use this generated model with small changes in order to adopt it to our example application.
In the studio open the ‘Properties’ panel (click the 3-dots-lines button on the left)
At the top of the Properties panel, under ‘Model’ → ‘Name’, rename the model to “TrafficLightModel”.
Next, we will rename all of the model’s generated elements: the edges and vertices.
A Vertex is the rectangle that represents a state in which the model can be during runtime.
An Edge is the arrow that connects 2 vertices in a certain direction, and which represents a transition from one state to another during runtime.
To rename a model’s element just click it and rename it in the Properties panel, under ‘Element’ → ‘Name’.
The result should look like this:
Make sure that the ‘Start element’ button at bottom of the Properties panel is ‘On’ for the ‘v_Start’ element and ‘Off’ for all the rest (this what makes v_Start becomes green).
Your model should have the following 3 vertices: v_Start, v_Red and v_Green, and the following 2 edge names: e_Toggle and e_TurnGreen, where e_Toggle is used in three different edges in the model.
We will leave the chosen test-generator ‘random(edge_coverage(100))’ (at the bottom of the Properties panel) for this post. The generator is a directive to GraphWalker for how to generate the tests and when to stop. This generator will cause GraphWalker to randomly choose the next edge (when it has choices) and to stop once it has visited all the edges. We will look at other generators in a future post.
Click the ‘Play’ button to test that it doesn’t have any errors
Click the ‘Save’ button that saves the model as test.json in your ‘Downloads’ folder.
Exit the studio and copy the test.json file into the ‘resources’ folder in the HelloMbt project. Delete the SmallTest.json file (this is the generated model that we don’t need anymore) and rename the test.json model file to: TrafficLightModel.json
7. Test Cases Generation
At this point we can already generate the test-cases for our requirements.
At the root of the HelloMbt project run:
$ java -jar ../lib/graphwalker-cli-4.2.0.jar offline -m src/main/resources/com/mbtroads/TrafficLightModel.json “random(edge_coverage(100))” | jq ‘.currentElementName’
The generated test suite
GraphWalker can generate and execute tests that span on several connecting models to enable the representation of a large application or suite of applications.
8. Model’s Interface Generation
Run the following line to generate the interface from the model:
$ mvn clean graphwalker:generate-sources* The first Maven run may take longer since Maven downloads all the required dependencies.
If you use Mac and get errors on .DS_Store, just remove those .DS_Store files.
The generated interface can be found at:
target/generated-sources/graphwalker/com/mbtroads/TrafficLightModel.java
Later we will see how our test implements this model’s interface.
9. Write the Target Application
Let’s add the TrafficLight application code which we plan to test with our MBT test.
Create the file: src/main/java/com/mbtroads/TrafficLight.java
With the following application code:
package com.mbtroads;public class TrafficLight { public static enum Color {
RED, GREEN
} private Color myColor; public TrafficLight() {
myColor = Color.GREEN;
} public Color getCurColor(){
return this.myColor;
} public void setColor(Color color){
this.myColor = color;
} public void toggleLight(){
if(this.myColor == Color.GREEN) {
this.myColor = Color.RED;
}
else {
this.myColor = Color.GREEN;
}
} public static void main(final String[] args) {
final TrafficLight target = new TrafficLight();
target.setColor(Color.RED);
System.out.println(target.getCurColor());
}}
10. Test Code
We are going to re-write the default test that was generated for us by GraphWalker. In a future post I plan to show how to generate the test skeleton from the model.
Replace the code in the generated test file: src/main/java/com/mbtroads/SomeSmallTest.java
with the following code:
package com.mbtroads;
import org.graphwalker.core.machine.ExecutionContext;public class SomeSmallTest extends ExecutionContext implements TrafficLightModel { TrafficLight trafficLight = null; @Override
public void e_Toggle() {
System.out.println(“Running: e_Toggle”);
trafficLight.toggleLight();
} @Override
public void e_TurnGreen() {
System.out.println(“Running: e_TurnGreen”);
trafficLight.setColor(TrafficLight.Color.GREEN);;
} @Override
public void v_Start() {
System.out.println(“Running: v_Start”);
trafficLight = new TrafficLight();
assert(trafficLight.getCurColor() == TrafficLight.Color.GREEN);
} @Override
public void v_Red() {
System.out.println(“Running: v_Red”);
assert(trafficLight.getCurColor() == TrafficLight.Color.RED);
} @Override
public void v_Green() {
System.out.println(“Running: v_Green”);
assert(trafficLight.getCurColor() == TrafficLight.Color.GREEN);
}}
During test execution, GraphWalker decides which transition to perform on the model’s graph, and in parallel, the same transition is being executed on the test by calling the corresponding test method.
Let’s examine the code in the SomeSmallTest.java test file.
Test methods that implement edges perform actions that make the target application transitions from one state to another:
@Override
public void e_Toggle() {
System.out.println(“Running: e_Toggle”);
trafficLight.toggleLight();
}Here the Traffic-Light application transition from one light to another.
Test methods that implement vertices check (using assertions) that the target application is at the expected state (same state as the model at this point of the test execution):
@Override
public void v_Green() {
System.out.println("Running: v_Green");
assert(trafficLight.getCurColor() == TrafficLight.Color.GREEN);
}Here the test checks that the Traffic-Light application has the Green light.
11. Test Execution
Since the test in this example is not a Junit test (and is under the ‘main’ code), we have to explicitly ask Maven to enable the assertions in the test code. In a future post we will see how to have the MBT tests executed through Junit.
Enable assertions:
$ export MAVEN_OPTS=”-ea”Remember to set the right Java environment for the current shell session:
$ export JAVA_HOME=`/usr/libexec/java_home -v 1.8`Execute the GraphWalker test:
$ mvn clean graphwalker:generate-sources compile exec:java -Dexec.mainClass=”com.mbtroads.Runner”
The tests should pass.
You can now play with the code to see how it works, for instance, in TrafficLight.java change the code in toggleLight() from:
else {
this.myColor = Color.GREEN;
}into this:
else {
this.myColor = Color.RED;
}This will cause the TrafficLight application to toggle from Red to Red instead of Green.
When you execute the test again you should get an assertion error originated in the v_Green() method (you have to enable assertions as I showed above):
@Override
public void v_Green() {
System.out.println(“Running: v_Green”);
assert(trafficLight.getCurColor() == TrafficLight.Color.GREEN);
}When the model transitions to v_Green, the test v_Green() method is executed and it expects the traffic light colour to be Green but it found it to be Red.
Summary
While this was a very simple example, I hope it showed the potential of Model-Based Testing on a large-scale cloud application where you may have frequent business requirements updates. With every requirement change you only need to update the model to immediately get a running full-coverage generated test suite.
I plan to cover some more advanced features of GraphWalker in future posts.
My experience with Model Based Testing
- Part-time contributor to NModel (MBT project by University of Washington and Microsoft Research) for about 2 years until the project was closed.
- Initiated and led the development of the first MBT solution for a cloud platform at a large international leading software company
- Used several additional MBT tools like org.tigris.mbt, ModelJUnit and recently GraphWalker and AltWalker
- Researched the area of Model-Based Security Testing using tools like Tamarin-Prover and others.
