Microservices: High quality code development — Going into practice.

By Daniel Perazza, Emilio Gerbino and Nicolás Quintana

intive

Published in

intive Developers

13 min readJun 8, 2023

Introduction:

The purpose of this paper is to present a practical approach of combining two different techniques: Semantic Testing, a concept that aims to focus on what to validate over how to implement the validations; and Mutation Testing, a technique intended to evaluate and improve the quality of unit tests, and in consequence, the overall quality of the code.

The theoretical aspects of these topics were covered in detail in a previous article: “Microservices: High Quality Code development — The theory”. This article will put the concepts into practice with a high level of detail.

A practical case

Let’s build a simple and small microservice. This will be the base to explain how to combine Unit Tests, mutation and semantic testing.

As described, “Calculator” is a simple microservice that allows performing math operations, and it will be used for practical purposes in the next sections.

It is implemented in javascript, express with node, using Jest, Striker and cucumber for testing. (Link to the code example repository can be found in the Annexes section)

Basic concepts

Unit Tests

For example, functions considered as units:

Unit test example:

Continuing with the examples, for the average function, the add function is mocked in order to focus only on the logic of the function under test:

Component Tests

The average function above could be considered as a component, knowing it is composed of other functions, such as add and divide. Here below, the component test is presented. It tests the main piece of code and its dependency as a unique piece, let’s say, as a component.

Another example. In this case, the endpoint REST /add is considered a component.

This uses other units of code, but it is tested as only one piece.

Integration tests

This test is working on the very same endpoint REST /add of the previous example, but here the difference lies in validating how the code is integrated with a server application.

E2E tests

In this case, the add operation is tested against a pre- production environment. This test is checking the functionality and also non-functional stuff for example, an api gateway, a firewall , a load balancer in front of the microservices, in this particular example, hosted at https://mycalculator-preprod.com/

The end-point value is just an arbitrary example, not a real one.

Mutation testing

To better illustrate the benefits of mutation testing, a new functionality will be added to the calculator.

This functionality is aimed to solve quadratic equations using the following formula:

Given an equation of the form:

The solution can be calculated as:

where:

a,b and c are real numeric constants.
𝑏² — 4. 𝑎. c is called the Discriminant of the equation.

Bear in mind that:

The solution to the equation presents two possible variations depending on the discriminant square root value.
The solution can only be found if a is not 0, otherwise, it would not be a quadratic equation, or in other word, since the resolution formula is a division by 2 times a, we can not divide by zero.
If the discriminant is a negative value, then the solutions are imaginary numbers.

With all this taken into account, the functionality can be expressed as the following function:

As stated before, a set of unit tests needs to be provided in order to evaluate the function and ensure that it can solve all particular cases, and return an error whenever the parameters are not valid.

The code coverage that can be extracted from this suite of tests can be found in the following image:

As expected, the coverage shows that the function is 100% tested, however it is actually only showing that it has executed all possible paths inside said function and that’s why it can not be used to measure quality.

In order to solve the question “how do we know that this function is fully and correctly tested?”, a mutation testing framework can be introduced into the stack.

In this instance, Stryker Mutator will be used, but also others can be used as well depending on many factors, like the languages they support, mutation operators, ease of use, etc.

Stryker Mutator is a well established mutation framework that supports multiple languages like Javascript, Typescript, C# and Scala and testing frameworks like Jest, Cucumber, Mocha and Karma, as well as providing high speeds and over 30 possible mutations. It provides very useful reports and is open source also.

To add Stryker Mutator to the mix, follow this steps:

Run the command npm install -g stryker-cli to install the stryker cli.
Run the command stryker init to run the initialization prompt
Stryker will ask for information about the project, like what package manager is used and what library or framework cli is used (angular-cli, create-react-app, none, etc). Follow the prompt to finish the setup. At the end, stryker will create a stryker.config.json file with your setup configuration.
Run the command stryker run.This may take several minutes depending on the project.

This last command will execute stryker against your tests. Stryker will notify each step of execution and will indicate if any failures occur.

For the quadraticEquation function created before, the console will show something like this:

Focusing only on the quadraticEquation.js file, Stryker reports that 37 mutations were made and 28 of them were killed by the test suite giving a mutation score of 75.68%. This means that 9 mutants, aka potential bugs, were introduced and the test suite was not able to catch.

Fortunately Stryker provides a more developer friendly report that can be found in the reports/mutation/mutation.html file, that when served looks like this:

Using this report, the developer can navigate to the function that is being analyzed, and find what mutants survived (as described in the following picture).

Clicking on each red dot will reveal the mutant.

Mutation Analysis

One way of interpreting the survival of a mutant is to understand it as the failure of the test suite to verify or assert the code where the mutant is present.

Taking the first mutant as an example, it can be point out that since Stryker introduced a mutation that essentially erased the text inside the Error, an assumption can be made that indicates that the suite of test didn’t verify that the error thrown had the original “Invalid Quadratic Equation — Cannot divide by zero” string.

Then, the next course of action would be to modify the test to take that mutation into account:

The following mutation is related to arithmetic operators. Stryker decided to change a multiplication for a division and this mutation survived, which is alarming since changing the formula in any way should cause a miscalculated value of the quadratic equation for the same parameters expressed in the tests.

If looked closely, this mutation is indicating that the formula behaves the same way if the multiplication is changed by a division, and the parameters do not change. This might be showing that the example parameters used in the tests do not reflect all possible cases.

Sure enough, some of the tests use the value 1 for the first parameter “a”, that causes the mutation to survive. That is because a number multiplied by 1 and divided by 1 returns the same value, meaning that if the value 1 is used for the parameter a, the tests will not differentiate between the formula with a multiplication or a division.

To solve this, we could either change the value used in the test for something less trivial, or add more test cases with different parameters.

This solution will clear some of the other mutants, and the same could be applied for the tests that cover the inside of the if (discriminant < 0) conditional, resulting in the following changes.

Finally, there is one more mutant that is surviving. This mutant changes a less than operator for a less or equal operator. This is indicating that our tests haven’t checked for the equality case, aka what should happen when the discriminant is equal to 0.

Once more, Stryker is helping to identify possible flaws in the tests, and in spite of having a code coverage of 100%, the suite has missed a crucial case.

To solve this final mutant, it is enough to add a test case where the determinant is 0.

Running Stryker once more reveals the following score

The quality of the tests for the quadraticEquation function has been improved and now the function is fully and 100% tested.

As an exercise, the reader is encouraged to check out the project source code where more examples of mutation testing can be conducted over other functions, like the “lineIntestects” function, included in the utils.js file. The source code link can be found in the “Annexes” section of this article.

Semantic testing

There are different tools that can be used for writing semantics tests. For this practical example, Cucumber is the chosen framework.

Tool & frameworks

Cucumber is defined by themselves as an open source tool that tests business-readable specifications against any code on any modern development stack. It is the world’s #1 tool for Behavior-Driven Development.

Chai is the library used for assertion. It makes testing much easier giving a lot of assertions that can run against your code.

Setup

For this practical case, these are the steps to follow:

npm install — — save — dev @cucumber/cucumber chai
In the package.json file add the command to run the cucumber tests

3. Add the cucumber.js file to configure cucumber. The highlighted lines tell Cucumber that the semantic tests are in the feature folder and end with .feature. Also that the .feature files will require the code in step-definitions/**/*.js to implement the tests

The semantic test

So, finally this is the resulting test as desired. A test easy to understand that verifies functionality or behavior regardless of the test implementation:

Behind scenes, its corresponding implementation:

After running the semantic test, the following report is produced:

The image below shows a report with more features and scenarios. The green pie charts at 100% say that all features and scenarios were run successfully, that is great. If the semantics concepts are well applied a report like this gives a very accurate status of the health of our product.

Besides in the scenarios Divide two integers and Get the average of array of integers, it is seen that some steps are shared between both of them, meaning that steps and implementations can be reused reducing effort and time. The fact that many steps are shared gets more evident in the semantic tests since natural language and a business abstraction are used.

In the Annexes section it is placed a link to the repository with the examples used throughout this document. More examples can be found there either to view or run.

Pros and Cons

Mutation Testing

As it was shown, the application of mutation testing enables developers to detect potential bugs and ensures that the tests are created with quality in mind.

It also allows developers to understand and find cases and ambiguities in the source code that were not checked before.

And finally, mutation testing also makes sure that by eliminating small mutants, bigger, more costly and risky bugs are also eliminated.

When we talk about downfalls, the two major factors that can affect the application of this method are:

Cost of producing and evaluating mutants: Depending on the number of mutants generated, and the time it takes to run the test suites, the whole process can become extremely time consuming. Also, the number of mutants generated depends on the choice of mutation operators so the more operator developers wish to use, the more mutants will be generated. A test suite that takes about 5 seconds to run, with a code base that generates 2500 mutants can take approx. 4 hours. This issue can be addressed by putting in place optimizations techniques that are not in the scope of this paper (and most frameworks will implement by default).
Cost of equivalent mutants: Equivalent mutants are changes made to the original code that differ only in the syntax but not in the semantics, meaning the mutant generated creates a change that does the same as the original code. These mutants cannot be detected and create false surviving mutants that can influence the final mutation score and also represent a problem for developers when interpreting the results and trying to eliminate mutants.

Semantic Testing

Let’s mention some pros of semantic testing: it abstracts testing to the business domain without bothering with the code for those tests, plus it enforces quality by leaving coding issues behind.

Another benefit is test understanding. Semantic testing, as a first step towards semantic monitoring, is a technique that focuses on finding a standard way of testing software. This means to focus on “what” the system should be doing in a clear, easy to understand way. This is the key concept to ensure the complete behavior of the whole digital capability.

The last advantage is reporting improvements. Because of the way semantic testing works, it makes the reports easier and richer.

However a disadvantage of this approach is that extra complexity is introduced. In the worst scenario, without a deep understanding of the underlying tools, tests may become un-maintainable and unscalable.

Final conclusion

At this point, a clear idea of what mutation testing and semantic testing are should be on the table. The key point is to understand that combining both techniques will secure the code’s quality, as a manner to guarantee that the developed microservice is not only compliant with the intended behavior, but also that it has been correctly built internally.

This is the key point of this article. Both techniques are powerful and can be used in a stand alone fashion, however, by combining them the resulting quality gets boosted up. This produces a true confidence and sense of security in the delivered microservice. Also, by mixing both approaches other benefits come as well, like enhanced monitoring with health check alarms from Semantic Monitoring, meaningful code coverage that prevents subtle logic flaws or weaknesses from Mutation Testing, creating a well-built, solid software component.

As final tips, remember that:

Code coverage metric is a MUST, it tells how the tests are reaching the code, but it’s not sufficient to make sure quality is attained. Tests MUST be good and comprehensive, a way for achieving this is mutation testing, plus doing the right assertions. This metric also tells how good the regression test suite is.
Implementing a good test strategy is essential for quality. The goal is avoiding test overlapping. For instance, a good plan could be to cover most of the code with component tests, cover corner cases with unit tests and cover the acceptance criteria of main functionalities with e2e. Checking integration points either with e2e tests or with component ones.
Use semantic tests when it is more suitable to focus on quality over its implementation. It could be used for e2e, component or integration tests. The coverage would be the same but the test will be better.
Extends the semantic concepts to monitoring in order to join quality and observability in the same picture.
Implement a Continuous Integration (CI) pipeline to execute all this tool and strategies in an automatic, scalable and systematic way.

For a concrete example of mixing these two concepts together, please refer to the source code link in the annexes.

Hope you have enjoyed reading this article.