Automated Performance Testing in XR — Part 1 — Introduction
Well-defined performance testing strategies are essential for XR applications. Automation streamlines repetitive testing cycles, ensuring the quality and robustness of XR experiences. Let’s get in to automated performance testing.
PART 1|️|️ PART 2➡
Unity is one of the most demanding cross-platform game engines for creating games and other collaborative AR/VR (XR) applications. For any sort of applications or games, there is an extensive competition in the market to provide seamless user experience. Best user experience always holds the user. Thus performance of an application plays a vital role as it correlates at first hand with the user experience and compatibility of any application.
I am starting a multipart series to cover automated performance testing, this is the first article I am going to introduce you with performance testing with unity based XR application, and will share the recent exploration of mine in the performance testing. Performance testing in unity can be done in multiple ways from in-built tools to third-party cloud tools. My focus in the article will be on in-built ones. Let me introduce a couple of in-built tools for performance testing — Unity Performance Testing Extension and Unity Test Runner.
Basic Setup
Before moving on to the actual testing part we should obviously need a unity code base to perform tests. In case you don’t have any unity code to try the performance testing, you can utilize the Unity Hub provided sample games by default in the application. Once the application/code is ready we are good enough to start the performance testing.
Installing Performance Testing Extension package
Open the application/code base in JetBrains Rider.
- Add com.unity.test-framework.performance under dependencies and testables portion of manifest.json file.
"dependencies": {
…
"com.unity.test-framework.performance": "3.0.0-pre.1"
},
"testables": [
"com.unity.test-framework.performance"
]On successful execution of above, the package deployment can be confirmed from the Unity package manager window.
- Create an assembly definition file naming “Tests” under the same folder the tests we are going to execute( create folder name “Tests” to include all scripts). “Unity.performanceTesting” should be referenced in the assembly definition file.
"name": "Tests",
"references": [
"Unity.PerformanceTesting"
],
"optionalUnityReferences": [
"TestAssemblies"
],
"includePlatforms": [
"Editor"
],
"excludePlatforms": [],
"allowUnsafeCode": false
}Installing Test Runner
Unity Test Runner is a built-in tool in Unity game engine that allows developers and testers to create, run, and manage automated tests for their Unity projects. It allows us to select which tests to run, view test results, and even debug tests. We can perform any kind of testing including unit tests, integration tests, and functional tests using Test Runner.
Unity Test Runner can be opened from below location in Unity.
Window -> General -> Test Runner
Now we are well equipped with the initial setup. The performance testing extension package provides default testing scripts to perform the performance test for any given application. On top of that we can customize the tests based on our requirement and project.
Performance Tests
Performance testing extension provides API methods to scale the measurement in our performance testing. To avail this api methods we need to include “using Unity.PerformanceTesting” in our test scripts.
The several API methods are
- Measure.Method
- Measure.Frames
- Measure.Scope(SampleGroupdDefinition sampleGroupDefinition)
- Measure.FrameTimes(SampleGroupdDefinition sampleGroupDefinition)
- Measure.ProfilerMarkers(SampleGroupDefinition[] sampleGroupDefinitions)
- Measure.Custom(SampleGroupDefinition sampleGroupDefinition, double value)
These API methods in turn use an object of struct type — SampleGroupDefinition which helps to define our measurements. SampleGroupDefinition can be parameterized with name, sampleUnit, aggregationType, percentile, increaseIsBetter and threshold.
Sample test script
The following sample code is implemented to measure the performance of loading and rendering of the scene named “SampleScene” and reports the result through measure API.
[UnityTest, Performance]
public IEnumerator Rendering_SampleScene() {
using(Measure.Scope(new SampleGroupDefinition("Setup.LoadScene"))) {
SceneManager.LoadScene("SampleScene");
}
yield return null;
yield return Measure.Frames().Run();
}Execution of Performance Testing and Test Results
With the Unity Test Runner window open, we should click the “Run all” button displayed in the play mode tab to execute all the tests. To execute a particular test we should click the “Run Selected” button on the selected test.
In the above screenshot you can see that all the tests got executed and the test result was passed ( green tick mark along the test ). Below the test result we can observe the values displayed for each test. Here we can see the test result for individual tests.
The consolidated test results can be viewed by navigating through the given path below
Window -> Analysis -> Performance Test Report
Comparing Performance Test Results
Considering the requirement of the project we can decide the threshold value for each measurement such as frame rate, CPU usage, GPU usage, load time, render, physics and thermal rate.
Record the results for comparison against the future test runs. Analyze the data to identify areas of improvement. Implement required optimisation to increase the performance of the application. Re- run the tests after each optimization or with defined time intervals to ensure the performance.
Automation Test Integration In CI
Performance tests above can be integrated in Continuous Integration Pipeline to run in automated way. Read here more details on CI setup for Unity.
Similarly, it can be setup for Jenkins and more CI tools
Finally
As performance testing lays the way to identify issues early at the development stage before it becomes much difficult and costly to fix. Also performance testing helps to ensure the scalability and to provide seamless user experience.
Happy Testing !!
PART 1 |️|️ PART 2➡
