Testing State Machines — The Model-Driven Approach
When developing complex reactive systems, state machines are often used to describe and implement the logic of a software component. Following the model-driven paradigm, state machines are first modeled on an abstract level and in a second step transformed into source code. This has several advantages, as on the abstract level a different representation of the state machine can be used, like a two-dimensional one with boxes and arrows, also known as a state diagram or a statechart. This allows for a better understanding of the modeled system, also for non-technical persons.
However, when it comes to testing and debugging, it depends on good tool support if the model-based approach is a win or a hurdle for the development team.
Status Quo
Even if the model-driven approach is followed, testing and debugging often happens on code level as the modeling tool does not support these activities adequately. As the state machine code is generated to be in sync with the model, bugs first need to be found in the code and afterward translated into the abstract model level to be fixed there. This translation is often a manual task. Some tools may provide a tracing between model and code parts to easier find the model part for a given defective code part. However, a bug is seldom located in a single line of code. In most cases, a larger part of the code, and hence of the model, needs to be adjusted. Tools that allow testing the model itself in an automated way can save a lot of time here.
Once the generated state machine code is incorporated in a software component, testing is often performed on the component or application level. To test the component it is somehow stimulated by an input, which implies stimulation of the inner state machine. The state machine reacts and communicates back with the component, for example by sending an event, which eventually invokes some application code whose side effects can be checked in the test method. This is not the same as directly testing the state machine. In case of a failed test it is not clear whether the state machine itself is faulty, or its integration with the application code. Here again, testing the state machine already on the model level helps to identify problems early in the development process and also allows the reuse of state machines for different applications without rewriting the tests.
Quo Vadis? Or how can we do better?
As explained above, testing state machines on the model level is more beneficial to the development process than testing on the code level. The obvious prerequisite for this is that the state machine model is executable. Testing the model is only possible if one can interact with it by sending events which then provoke a reaction of the state machine that can be checked.
Once the state machine model is executable, you can interact with it manually or write scripts to perform interaction patterns automatically. Manual interaction can be used for early testing, already while modeling. This is the same as if you were running your application while coding to check if your code works. The ability to execute the state machine model allows to already verify it before any application code is written. This allows to speed up your development cycles as changes in the state machine logic do not need to be reflected in the application code until the state machine has a stable state.
The next logical step is to automate the manual interaction with the model. Automated tests ensure that model changes do not break existing functionality. Ideally, these tests reflect the requirements on the state machine logic. When requirements are added or changed, they can be translated into test cases first which leads towards a test-driven development approach on model level.
Once we have a bunch of executable tests for our state machine, the next step is to evaluate the quality of the test set. This is typically done with the help of coverage metrics. There are several coverage metrics, like statement coverage or branch coverage, however, when it comes to measuring the coverage based on a state machine model, we want to know which states are activated, and which transitions are taken by our test cases. This helps to identify redundant test cases as well as writing new test cases that add some value in terms of test coverage.
Tool support for model-driven state machine testing
Model-driven state machine testing is not feasible without adequate tool support. The ability to execute and test a model requires the ability to interpret a state machine model just as if it were code that is interpreted by an interpreter. YAKINDU Statechart Tools comes with exactly that ability. The graphical state machine models created with YAKINDU Statechart Tools can be executed and tested already on the model level.
Executable state machines
YAKINDU Statechart Tools allow executing the modeled state machine in a simulator. The simulator allows the user to raise events or to set variable values. The reaction of the state machine is directly reflected in the model by highlighting the active state(s). The simulation view lists all variables and their values. Furthermore, the user can set breakpoints and create snapshots to restart the simulation at a specific point of execution. This allows to easily verify the modeled behavior already while modeling the state machine.
Test automation with statechart unit tests
YAKINDU Statechart Tools comes with a scripting language (SCTUnit) that allows writing tests comprehensively and compactly. Each test, written in the SCTUnit language, consists of a well-defined sequence of instructions. When running a test, these instructions are applied to a state machine under control. The expected reaction of the state machine is expressed by assertions. With the SCTUnit language you can:
- raise incoming events
- change values of variables
- assert that a state is active
- assert that a variable has a certain value
- assert that an operation has been called with certain input values
- assert how often an operation has been called
- define simple operation mocks
- virtually proceed time or run-to-completion cycles.
Here is an example of a test case written in SCTUnit:
Explaining the details of SCTUnit is out of the scope of this article. If you are interested, you can read the documentation or watch this video by Professor Tom Mens.
Test coverage on the fly
When a test set is executed, a model coverage is computed on the fly and visualized in the state machine model. This allows to easily identify missing tests. For example, from the coverage metric below, it is obvious that none of the tests has brought the state machine into the Save Energy state. Once we add a corresponding test and run the test set again, the Save Energy state gets green.
Summary
Testing state machines on the model level requires a tool that can execute the state machine model. If such a tool is available, it can help a lot to test and debug the state machine logic which in the long term saves you time and money. YAKINDU Statechart Tools fulfills this requirement. It allows simulating a state machine manually, as well as writing unit tests to automate test execution. The built-in test coverage metric gives a visual hint on the quality of the test set.
