[Tech] Testing Guidelines

Engineering guidelines part 2

Photo by Science in HD on Unsplash

Tools

Unit tests

• We use xUnit.net , Fluentassertion and moq for testing most code.
• Jasmine is used for unit testing TypeScript or JavaScript code

Performance Tests

• We use K6for test inside Kubernetes clusters.

End-To-End Tests

• We use Playright for automated test frontend websites.
• WireMock for mock services.

Unit tests and functional tests

Assembly naming

1. The unit tests for the Microsoft.Fruit assembly live in the Microsoft.Fruit.Tests assembly.
2. The functional tests for the Microsoft.Fruit assembly live in the Microsoft.Fruit.FunctionalTests assembly.

NOTE — In general there should be exactly one unit test assembly for each product runtime assembly. In general there should be one functional test assembly per repo. Exceptions can be made for both.

Unit test class naming

Test class names end with Test and live in the same namespace as the class being tested.

For example — the unit tests for the Microsoft.Fruit.Banana class would be in a Microsoft.Fruit.BananaTest

Unit test method naming

1. Unit test method names must be descriptive about what is being tested, under what conditions, and what the expectations are.
2. Pascal casing and underscores can be used to improve readability

The following test names are correct:

PublicApiArgumentsShouldHaveNotNullAnnotation
Public_api_arguments_should_have_not_null_annotation

The following test names are incorrect:

Test1
Constructor
FormatString
GetData

Unit test structure

The contents of every unit test should be split into three distinct stages, optionally separated by these comments:

// Arrange  
// Act  
// Assert

The crucial thing here is that the Act stage is exactly one statement. That one statement is nothing more than a call to the one method that you are trying to test.

For example, this is not ideal:

int result = myObj.CallSomeMethod(GetComplexParam1(), GetComplexParam2(), GetComplexParam3());

This style is not recommended because way too many things can go wrong in this one statement. All the GetComplexParamN() calls can throw for a variety of reasons unrelated to the test itself. It is thus unclear to someone running into a problem why the failure occurred.

The ideal pattern is to move the complex parameter building into the Arrange section:

// Arrange
P1 p1 = GetComplexParam1();
P2 p2 = GetComplexParam2();
P3 p3 = GetComplexParam3();
// Act
int result = myObj.CallSomeMethod(p1, p2, p3);
// Assert
Assert.AreEqual(1234, result);

Testing exception messages

In general testing the specific exception message in a unit test is important. This ensures that the exact desired exception is what is being tested rather than a different exception of the same type. In order to verify the exact exception it is important to verify the message.

To make writing unit tests easier it is recommended to compare the error message to the RESX resource. However, comparing against a string literal is also permitted.

var ex = Assert.Throws<InvalidOperationException>(
    () => fruitBasket.GetBananaById(1234));
Assert.Equal(
    Strings.FormatInvalidBananaID(1234),
    ex.Message);

Use xUnit.net’s plethora of built-in assertions

xUnit.net includes many kinds of assertions — please use the most appropriate one for your test. This will make the tests a lot more readable and also allow the test runner report the best possible errors (whether it’s local or the CI machine).

For example, these are bad:

Assert.Equal(true, someBool);
Assert.True("abc123" == someString);
Assert.True(list1.Length == list2.Length);
for (int i = 0; i < list1.Length; i++) 
{
    Assert.True(
        String.Equals
            list1[i],
            list2[i],
            StringComparison.OrdinalIgnoreCase));
}

These are good:

Assert.True(someBool);
Assert.Equal("abc123", someString);
// built-in collection assertions!
Assert.Equal(list1, list2, StringComparer.OrdinalIgnoreCase);

Parallel tests

By default all unit test assemblies should run in parallel mode, which is the default. Unit tests shouldn’t depend on any shared state, and so should generally be runnable in parallel. If the tests fail in parallel, the first thing to do is to figure out why; do not just disable parallel tests!

Summary

1. We have specific tools for each project type.
2. We have naming conventions to create Projects, Files, Methods, Classes.
3. AAA Pattern (Arrange, Act, Assert)
4. Testing exception messages
5. Use built-in assertions
6. Parallel tests

References

• https://github.com/dotnet/aspnetcore/wiki/Engineering-guidelines