Automated Performance Testing in XR — Part 2 — FPS
This part covers measuring and benchmarking FPS for XR application, and setting up automated tests for the same.
⬅ PART 1 || PART 2 |️|️ PART 3 ➡
In the realm of XR app development, optimizing performance is paramount to delivering a smooth and enjoyable experience. Unity, one of the leading game development engines, offers a plethora of tools and techniques to fine-tune the performance of your projects. Last article introduced automated performance testing.
In this article, we’ll delve into the intricacies of frame rate metrics calculation in Unity applications for XR and the crucial process of benchmarking these values.
Understanding Frame Rate
Frame rate, often abbreviated as FPS (Frames Per Second), is a fundamental metric in gaming performance evaluation. It directly affects the fluidity and responsiveness of your game.
In Unity app development, maintaining a consistent and high frame rate is paramount for several reasons:
- Player Experience: A higher frame rate results in smoother animations and transitions, providing players with a more immersive and enjoyable gaming experience. It reduces visual artifacts such as screen tearing and stuttering, enhancing overall gameplay responsiveness.
- Perception of Quality: Players often associate a higher frame rate with a higher-quality game. A stable and high frame rate contributes to the perception of a well-polished and professional product, positively influencing user satisfaction and retention.
- Gameplay Responsiveness: In fast-paced games, such as action or first-person shooters, a high frame rate is essential for maintaining gameplay responsiveness. It reduces input lag, ensuring that player actions are reflected on-screen with minimal delay, thereby enhancing gameplay precision and control.
- Optimization and Performance: Monitoring frame rate is a crucial aspect of performance optimization in Unity game development. By identifying areas of the game that cause frame rate drops or performance bottlenecks, developers can optimize resource usage, streamline code, and improve overall game performance.
Measuring FPS
Measuring FPS in Unity can be achieved using various methods, but one particularly effective approach involves utilizing Perfetto trace files. Perfetto, a performance tracing tool, provides detailed insights into system behaviour, including frame rate analysis.
Start perfetto
First step of this process is executing the Perfetto command. Use the below command to record the trace file of the target game in your command prompt. This command helps to capture traces on-device directly.
adb shell perfetto -o /data/misc/perfetto-traces/trace_file.perfetto-trace -t 20s \sched freq idle am wm gfx view binder_driver hal dalvik camera input res memoryThis command leverages the ADB tool to access the shell of an Android device, enabling direct execution of commands. Within the shell environment, the command initiates Perfetto. By specifying the output file path and name, Perfetto saves trace data to a designated location on the device, facilitating subsequent analysis. The “-t 20s” flag defines the duration of the tracing session, set here for 20 seconds.
Crucially, the command specifies categories or data sources for Perfetto to monitor during the tracing session. These categories encompass various aspects of system operation, including CPU scheduling, frequency scaling, idle states, and hardware events such as camera usage. This is the generalised command which can be utilised for calculating several performance metrics. The visual representation of the trace records will looks like below one.
Within your framework, create a shell script file and then saves the file to a directory on the local machine. Model it as exemplified below.
Test.sh
adb shell perfetto -o /data/misc/perfetto-traces/trace_file.perfetto-trace -t 20s \sched freq idle am wm gfx view binder_driver hal dalvik camera input res memory
adb pull /data/misc/perfetto-traces/trace_file.perfetto-trace /home/user/Performance/trace_files/Extract dropped frames and average frame duration
From the above saved trace file we have to extract the dropped frames and average frame duration values. Dropped frames can significantly impact performance, especially in real-time applications like video streaming, gaming, or animations. When a frame is dropped, it means that it was not rendered or displayed on time, resulting in a noticeable interruption or lag in the user experience. In gaming, dropped frames can affect gameplay smoothness and responsiveness, making it harder for players to react quickly.
To perform this analysis, we leverage Perfetto’s Trace Processor Python API, which facilitates efficient data extraction and processing from trace files.
Create an Utils folder and add a python script file in the framework to utilize Perfetto’s trace processing capabilities to extract and analyze FPS metrics. Model it as exemplified below.
FrameRateMetric.py
from perfetto.trace_processor import TraceProcessor, TraceProcessorConfig
def main():
tp = TraceProcessor(trace='trace_file.perfetto-trace')
fps_metrics = tp.metric(['android_frame_timeline_metric'])
print(fps_metrics)
tp.close()
if __name__ == "__main__":
main()The above python script imports TraceProcessor and TraceProcessorConfig from the Perfetto package. Within the main function, a TraceProcessor object named ‘tp’ is instantiated, specifying the Perfetto trace file ‘trace_file.perfetto-trace’ for analysis. The script then extracts FPS (Frames Per Second) metrics using the ‘android_frame_timeline_metric’. This metric provides insights into frame rendering times.
Let’s have a look at how ‘android_frame_timeline_metric’ data presented in Perfetto UI.
"android_frame_timeline_metric": {
"total_frames": 337,
"missed_app_frames": 0,
"dropped_frames": 0,
…
"frame_dur_max": 15550000,
"frame_dur_avg": 4678562,
"frame_dur_p50": 4375573,
"frame_dur_p90": 4501291,
"frame_dur_p95": 4645417,
"frame_dur_p99": 15495681,
…By parsing the trace file metrics, such as average frame duration and dropped frames, we can calculate the average FPS using the formula:
Average FPS = 1 / Avg frame duration
The average duration value will be represented in nano seconds and the final output value should be converted to seconds.
Next, create a main file that calculates final FPS making use of the above Python script.
public void calculateFPSvalue()
{
Process fpsMetrics = new Process();
ProcessStartInfo startInfoForFPSData = new ProcessStartInfo
{
FileName = "python3",
Arguments = "./Utils/FrameRateMetric.py",
WorkingDirectory = "/Home/user/Performance",
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput = true,
UseShellExecute = false,
CreateNoWindow = false,
};
fpsMetrics.StartInfo = startInfoForFPSData;
fpsMetrics.Start();
string output = fpsMetrics.StandardOutput.ReadToEnd();
string droppedFramesLine = SearchForText(output,"dropped_frames");
string frameAvgDurationLine = SearchForText(output,"frame_dur_avg");
string frameAvgDuration = frameAvgDurationLine.Substring(19);
string dropped_frames = droppedFramesLine.Substring(18);
double frameAvgDur = Math.Round(1/(Convert.ToDouble(frameAvgDuration)/1000000000));
fpsMetrics.WaitForExit();
addToTestReport.WriteLine("Average FPS - - " + frameAvgDur);
addToTestReport.WriteLine("Number of dropped frames - - " + dropped_frames);
fpsMetrics.Close();
}This C# method, “calculateFPSvalue()”, is designed to measure frames per second (FPS) performance metrics. It initiates a Python script named “FrameRateMetric.py” using the “python3” command and retrieves its output. The Python script likely extracts FPS-related data from a performance trace file.
Through a meticulous parsing mechanism, facilitated by a helper method dubbed “SearchForText”, it identifies and extracts specific lines containing pertinent information like the number of dropped frames and the average frame duration. To grep the number of dropped frames value and average frame duration, the initial string value is provided.
After obtaining the output, the method parses it to extract relevant information such as the average frame duration and the number of dropped frames. The average frame duration is calculated in seconds and rounded to provide a clearer representation of performance. Finally, the method reports the average FPS and the number of dropped frames to a test report file using SpecFlow to output the values.
Take a look at the following output:
-> Average FPS - - 86
-> Number of dropped frames - - 0The optimal FPS
The optimal FPS (Frames Per Second) value for an Android device can vary depending on the type of application or content being displayed. However, in general, a smooth and responsive user experience is typically achieved with FPS values ranging from 30 to 60 frames per second. For most applications, aiming for a stable FPS of 60 is often considered ideal, as it provides a visually smooth experience for users. However, certain types of applications, such as fast-paced games or VR experiences, may require even higher FPS values to ensure fluid and immersive interactions. Ultimately, the optimal FPS value for an Android device depends on factors such as the device’s hardware capabilities, the complexity of the application or content, and user expectations.
Benchmarking FPS
Benchmarking FPS involves measuring the rate at which frames are rendered by an application or system within a specific timeframe. To begin, select the testing environment. Once the application is running under typical conditions, initiate the benchmarking process to capture FPS data over a predetermined period. Here the test could be executed in different time durations, different iterations and in different number of devices with same environment help to achieve the optimal benchmarked value. Record data across all variants and save it for further analysis.
Analyze the recorded data, paying attention to trends and anomalies, and compare results across different scenarios or hardware setups. This analysis provides insights into performance and highlights areas for improvement. By iteratively optimizing code, adjusting graphics settings, or upgrading hardware components based on benchmarking results, developers can enhance the FPS rate and overall performance of the application or system. The benchmarking process should be repeated periodically to ensure that progress is consistently maintained.
For more information on analyzing trace files with Perfetto, refer to the official documentation here
Happy Testing !!!
