All you need to know about Testing in Unity3D… Part - 1

So.. to set some context on what this series is all about… We will be looking into Level 1 and 2 of writing tests in Unity3D and look into some special tips to cover tricky scenarios… Will be touching upon Level 0 as well but mostly to point you in the right direction and not to go deep into it.
But what is this scale you might ask… Well it’s a scale ranging from 0 to 5, where your understanding of writing tests grows with each level.

• Level 0 is where you know how to write basic nunit (EditMode) tests;
  i.e you understand how testing works and are familiar with the test annotations/attributes. (As part of this article we will be covering annotations that facilitate Parameterized testing)
• Level 1 is where you understand Mocking, Stubbing, Spying, Fakes, Dummies, Argument capturing… essentially the tools of the trade, and have scratched the surface with testing Unity lifecycle methods (aka PlayMode Tests).
• Level 2 is when you can test Events, use Reflection, have an in depth understanding of how to set up PlayMode tests and Simulating User Inputs.
• Level 3 is all about setting up complete Automation Suites, writing custom frameworks or Setting up tests in CI. And is beyond the scope of this series.
• Level 4 …Just like the 4th Dimension I have no notion of it… XD
• Level 5 …So these are just place holders for people to explore beyond and come up with more stuff. (You can call me optimistic or just foolish for it… :P)

Disclaimer: These standards aren’t concrete and you might actually have pieces from different levels, and the levels aren’t meant to judge but to organize the different topics so that it’s easier for people who are new to consume given the depth of what we are trying to cover.

For references we will be using this Repo: https://github.com/Neelarghya/testing-in-unity

Level 0

To get an understanding of how to setup the Unity TestRunner, what to test, what’s TDD and the general testing (i.e. Level 0 + Mocking) you can look through this awesome article by Kuldeep Singh…

Practical Unit Testing in Unity3D

…Back? Nice.

Let’s understand the basic difference between EditMode and PlayMode tests.

• EditMode Tests also known as Editor tests are tests that only run in the Unity Editor and have access to the Editor code. When using the Test annotation they run on a separate process spun up by the underlying NUnit framework.
  But when by using the UnityTest annotation you can run your tests on the EditorApplication.update callback loop, In this case you can test you Editor extensions you can also control entering or exiting PlayMode.
• PlayMode Tests run as a standalone in a Player or inside the Editor. PlayMode tests allow you to run your application or part of it. It runs as a coroutines when marked with the UnityTest annotation. This is where you would test your game loop (Unity lifecycle methods).

In tests marked with UnityTest annotation you can skip frames of execution to wait for background processes to end.

Basic NUnit Annotations — Full list can be found here

1. OneTimeSetUp → Identifies methods to be called once prior to any child tests
2. SetUp → Indicates a method of a TestFixture called immediately before each test method
3. Test → Marks a method of a TestFixture that represents a test
4. Order → Specifies the order in which decorated test should be run within the containing fixture or suite. (Not recommended, since tests should inherently be independent)
5. Ignore → Indicates that a test shouldn’t be run for some reason
6. TearDown → Indicates a method of a TestFixture called immediately after each test method
7. OneTimeTearDown → Identifies methods to be called once after all child tests

Parameterized Testing

So what is parameterized testing..? It’s cheating the test coverage report! :P
So the idea is you write fewer tests that can cover more scenarios by accepting parameters which corresponds to the different scenarios. Let’s see how we can set them up… For reference you can start by syncing with the above mentioned project…

git clone git@github.com:Neelarghya/testing-in-unity.git
git checkout f2c0e44

Let’s start by setting up a basic test to check the Area() of a Rectangle.

[Test]
public void TestAreaOfRectangleWithLength2AndBreadth3()
{
    Rectangle rectangle = new Rectangle(length:2, breadth:3);
    Assert.AreEqual(6, rectangle.Area());
}

As we can see the test covers a scenario where the rectangle has a length of 2 units and a breadth of 3 units. So in order to test another scenario as true TDD enthusiasts we should write another test for a different scenario… Something like TestAreaOfRectangleWithLength7AndBreadth4()… or do we?
*Drum rolls* Introducing TestCase annotation!
(TestCase is a method level attribute/annotation, i.e. it’s attached to the test method.)
So let’s convert these tests to a single parameterized test…

[TestCase(2, 3, 6)]
[TestCase(7, 5, 35)]
public void TestAreaOfRectangleArea(float length, float breadth, 
                                float area)
{
    Rectangle rectangle = new Rectangle(length, breadth);
    Assert.AreEqual(area, rectangle.Area());
}

Here we can notice a few things, firstly the test has parameters (namely length, breadth, area) and these are filled by the values passed to the TestCase annotation, finally each case is considered a separate test so they will run, pass and fail independently. Similarly we can add more parameters and cases for our tests.

[Commit: f7287d7]

So now we have the ability to the cases individually… But what if we needed to check a lot more values maybe between a range of values? Would we have to write all the cases? Let’s take an example, assume we have a function called Rectangle.IsInside(int x, int y) which return a boolean, if the point (x, y) is inside the rectangle (let’s assume the rectangle is always at 0 and we will have constant length and breadth for the tests). So if we visualized the scenarios we would have something like this…

So each circle represents a test point and red represents a return of false and green true for the IsInside(). So here if we tried to write TestCases it would be numerous. A couple such cases would look like..

[TestCase(0, 0, true)]
[TestCase(3, 3, false)]
public void TestIfPointIsInsideRectangle(float x, float y, 
    bool isInside)
{
    const float length = 3;
    const float breadth = 5;

    Rectangle rectangle = new Rectangle(length, breadth);

    Assert.AreEqual(isInside, rectangle.IsInside(x, y));
}

[Commit: e7f287a]

Thus the introduction of Range annotation…
(Range is a parameter level attribute/annotation, i.e. it’s attached to the test parameters.)

[Test]
public void PointsShouldBeInsideRectangle([Range(-2, 2)] float x, 
                                      [Range(-1, 1)] float y)
{
    const float length = 3;
    const float breadth = 5;

    Rectangle rectangle = new Rectangle(length, breadth);

    Assert.IsTrue(rectangle.IsInside(x, y));
}

So Range attribute takes 3 numeric (int/long/float/double) parameters the starting point, the ending point and the step (optional for int i.e. takes 1 step by default) (In the above example the parameters are implicitly type casted from int to float).

So by default if there are multiple Range attributes they come together Combinatorially to from thee test cases…

[Commit: 2ce1b80]

But if you have been paying attention you might have noticed I omitted a scenario, the false case when the point is outside the rectangle. Well this is a special case for a reasons. The external points don’t necessarily fall under a Range they are more like unique pairs of x and y.
Thus the need for Values attribute… (Values is a parameter level attribute/annotation as well)

[Test]
public void PointsOnLeftOrRightShouldBeOutsideRectangle(
        [Values(-3,3)] float x, [Range(-2, 2)] float y)
{
    const float length = 3;
    const float breadth = 5;

    Rectangle rectangle = new Rectangle(length, breadth);

    Assert.IsFalse(rectangle.IsInside(x, y));
}

[Test]
public void PointsAboveOrBelowShouldBeOutsideRectangle(
        [Range(-3,3)] float x, [Values(-2, 2)] float y)
{
    const float length = 3;
    const float breadth = 5;

    Rectangle rectangle = new Rectangle(length, breadth);

    Assert.IsFalse(rectangle.IsInside(x, y));
}

Here we are combining Range and Values attributes to 2 lines of values per test. Values takes an array of values you want to set as the parameter similar to how TestCase did, just the annotation is at the parameter level so they can be combined. (You can also use Sequential attribute or Pairwise attribute to combine the values sequentially or pairwise respectively)

[Commit: 6e339cb]

Other than these we have Repeat attribute.
Repeat is a method level attribute and as the name suggests it repeats the the test n number of times. “But why would anyone want to repeat the same test” you might ask… Well you might have a random pin generator and want to check it’s always in a certain range and doesn’t “repeat”.. :P
(Note: Repeat doesn’t work on parameterized tests or test suites)

private int _lastPin;

[Test, Repeat(5)]
public void PinShouldBeOf4Digits()
{
    int pin = PinGenerator.NewFourDigitPin();
    bool is4Digits = pin >= 1000 && pin <= 9999;
    
    Debug.Log("Pin: " + pin);
    Assert.IsTrue(is4Digits);    
    Assert.AreNotEqual(_lastPin, pin);

    _lastPin = pin;
}

Logging to demonstrate the repeating tests…

[Commit: 683cfa2]

And the final inline attribute we will cover today is Random.
Random is another parameter level attribute which as the name suggest produces random values. It has 3 parameters the range (from and to) and the count (i.e. number of values to generate). So we will repeat the test for points inside the rectangle but this time won’t use all the values…

[Test]
public void RandomPointsShouldBeInsideRectangle(
    [Random(-2f, 2f, 2)] float x,
    [Random(-1f, 1f, 2)] float y)
{
    const float length = 3;
    const float breadth = 5;

    Rectangle rectangle = new Rectangle(length, breadth);

    Assert.IsTrue(rectangle.IsInside(x, y));
}

From personal experience Random attribute tests are a bit flaky and sometimes get ignored by unity’s TestRunner, I avoiding using it…

[Commit: 47051df]

But you might be wondering “Why are you using a pair of floats for points like a Noob? Use Vectors!”
Well… You see what we covered were inline attributes, and inline attributes are setup at compile time i.e. you can not assign dynamic values like objects or properties, thus no “new Vector3()”.
So in order to setup values at run time there are two separate (can be set up from properties or methods) attributes, TestCaseSource and ValueSource. The concepts are similar to their inline counter parts TestCase and Values, they parameterize tests at a method level and parameter level respectively.

Let’s try testing which Rectangle is bigger i.e. Rectangle.IsGreaterThan (Rectangle) the function return true if the first Rectangle has more area than the other. For this test we won’t parameterize the length or breadth, rather we will parameterize the Rectangles.

private static object[] _cases = {
    new object[]{ new Rectangle(1, 1), new Rectangle(1, 1), false},
    new object[]{ new Rectangle(2, 1), new Rectangle(3, 1), false},
    new object[]{ new Rectangle(2, 2), new Rectangle(1, 3), true},
    new object[]{ new Rectangle(5, 4), new Rectangle(6, 3), true}
};

[TestCaseSource(nameof(_cases))]
public void CheckWhichRectangleIsBigger(
    Rectangle first, Rectangle second, bool isGreater)
{
    Assert.AreEqual(isGreater, first.IsGreaterThan(second));
}

[Commit: 0125c85]

And the last on the list is ValueSource, as mentioned before it used to parameterize runtime object to the test parameters. For this example we will use the Rectangle.Perimeter().

private static Rectangle[] _rectangles =
{
    new Rectangle(2, 3),
    new Rectangle(4, 2),
    new Rectangle(1, 5),
    new Rectangle(7, 3)
};

[Test, Sequential]
public void TestPerimeterOfARectangle(
    [ValueSource(nameof(_rectangles))] Rectangle rectangle, 
    [Values(10, 12, 12, 20)] float perimeter)
{
    Assert.AreEqual(perimeter, rectangle.Perimeter());
}

Here we are setting the values of the rectangles from the ValueSource _rectangle. You might have also noticed that inspite of having 4 x 4 parameters we don’t have 16 test cases… this is because of the Sequential attribute which tells NUnit to combine the parameterized values Sequentially and not Combinatorially (which is done by default).

[Commit: 47151c4]

One thing to keep in mind though when doing parameterized testing, you shouldn’t replicate the logic behind what you are testing just to make sure you can parameterize the tests. Tests should inherently be logicless.
With that we complete Parameterized Testing and Level 0… Congrats!

We already have more than 50 tests… (The 4 ignored are the flaky Randoms)
In the next part we will be dive into Level 1, i.e. the tools of the trade and get out hands dirty with PlayMode tests.

So until then let’s write fewer tests to cover more cases…

1 | Part 2 >