Automated Performance Testing in XR — Part 3 — CPU

This part covers measuring and benchmarking CPU utilization for XR application, and setting up automated tests for the same.

⬅ PART 2 || Current |️|️ Next ➡

Augmented Reality (AR) and Virtual Reality (VR) technologies, along with other Unity apps, demand detailed performance analysis and benchmarking to ensure smooth and immersive experiences for users.

Performance analysis encompasses a range of activities aimed at identifying bottlenecks and inefficiencies within the app’s code, assets, and overall design. This includes measuring frame rate (FPS), CPU and GPU usage, memory consumption, loading times, and other key performance indicators (KPIs). In last part we discussed measurement and impact of FPS.

Automated Performance Testing in XR — Part 2 — FPS

This article covers measurement of CPU utilization, and defined acceptance criteria for the same.

CPU Utilization

CPU utilization is critical aspect of application performance, as it reflects the degree to which the CPU is being utilized during gameplay. High CPU utilization can lead to performance bottlenecks and degraded gameplay experience. Here are some key reasons why CPU utilization is important:

• Game Logic Execution: The CPU is responsible for executing game logic, including physics calculations, AI behavior, input processing, and scripting. High CPU utilization can lead to delays in executing these tasks, resulting in laggy or unresponsive gameplay.
• Platform Compatibility: Different devices and platforms have varying CPU capabilities. Monitoring CPU utilization allows developers to optimize their games for a wide range of hardware specifications, ensuring compatibility and performance consistency across different devices.
• Frame Rate Stability: Consistent frame rates are essential for delivering smooth and immersive gaming experiences. High CPU utilization can cause frame rate fluctuations, resulting in stuttering or juddering visuals, which detract from the overall experience.
• Thermal Throttling: Continuous high CPU usage can trigger thermal throttling, where the device reduces CPU performance to prevent overheating. This can significantly impact game performance, leading to decreased frame rates and increased latency.

Measuring CPU Utilization

In this blog will delve into the calculation of CPU utilization using Perfetto trace files. Perfetto offers a comprehensive suite of tracing capabilities, allowing developers to capture detailed traces of system activity, including CPU, GPU, and memory usage, as well as I/O operations and power consumption.

The foremost thing to automate CPU utilisation value is to create a shell script file to execute the Perfetto command and to save the recorded traces.

Sample CPU Perfetto Trace

Refer this blog for Perfetto command and sample shell script file to execute the Perfetto record trace command under the topic -Start perfetto.

Prior to delving into automating CPU utilization, it’s essential to familiarize ourselves with the operational intricacies within the Perfetto UI.

An approach to retrieve data from a Perfetto record trace file involves utilizing SQL queries. Within the SQL framework, CPU utilization data is accessible through the sched_slice table. The subsequent SQL query provides a summary of various processes, their end states, and process duration.

select ts, dur, cpu, end_state, priority, process.name, thread.name
from sched_slice left join thread using(utid) left join process using(upid)

Upon executing the aforementioned query within the Perfetto interface, the tables presented in the screenshot below will be displayed.

Output from a sample SQL query for CPU event

The sched_slice table presents a comprehensive overview of various system processes and their attributes extracted from Perfetto record trace files. Each entry includes a timestamp (ts), duration (dur), CPU utilization (cpu), end state (end_state), priority, process name (process.name), and thread name (thread.name). For instance, the first row showcases logd.klogd, a system process, with a duration of 247,188 microseconds, utilizing CPU 2 while in state ‘S’ (Sleeping), and operating with a priority of 130. Similarly, subsequent rows detail the activity of different processes, providing valuable insights into their execution patterns and resource utilization within the system.

In Unity projects, CPU utilization can be calculated by analyzing the duration of CPU processes from Perfetto trace files. By querying the trace file data, we can determine the total CPU time and idle time, allowing us to compute CPU utilization using the formula:

CPU utilization (%) = (1 — (Idle time / Total time)) * 100

Automated Performance Tests for CPU Utilization

On automation once after creating the shell script to execute the Perfetto command and saving the recorded trace files in Test.sh file, create an Utils directory within the framework and incorporate a Python script file to leverage Perfetto’s trace processing features for extracting and examining CPU utilization metrics.

Incorporate the forthcoming Python script to compute the cumulative idle duration for the specified process.

idleTime.py
import argparse
 from perfetto.trace_processor import TraceProcessor, TraceProcessorConfig
 
 def main():
 tp = TraceProcessor(trace='trace_file.perfetto-trace')
 
 # Iterate through QueryResultIterator
 res_it = tp.query('select total(dur) as totalValue from sched_slice left join thread using(utid) left join process using(upid) where process.name = \'com.Perfmet.TrashCat\' and end_state = \'S\'')
 for row in res_it:
 print(row.totalValue)
 tp.close()
 
 if __name__ == "__main__":
 main()

This Python script utilizes the Perfetto trace processor library to analyze trace data from a specified file. It imports necessary modules including argparse for command-line argument parsing and TraceProcessor for interacting with the trace data. The main function establishes a TraceProcessor instance with the provided trace file and executes a SQL query to calculate the total duration of scheduler slices for a specific process in a particular state (‘S’ for sleeping). The results are iterated through and printed.

Incorporate the forthcoming Python script to compute the total duration for the specified process.

totalTime.py
import argparse
 from perfetto.trace_processor import TraceProcessor, TraceProcessorConfig

 def main():
 tp = TraceProcessor(trace='trace_file.perfetto-trace')
 
 # Iterate through QueryResultIterator
 res_it = tp.query('select total(dur) as totalValue from sched_slice left join thread using(utid) left join process using(upid) where process.name = \'com.Perfmet.TrashCat\'')
 for row in res_it:
 print(row.totalValue)
tp.close() 
 
 if __name__ == "__main__":
 main()

In the above Python script a SQL query is executed to calculate the total duration of scheduler slices for a specific process. Since we haven’t specified any particular state, the query encompasses all states of the target process, providing the total duration across all states.

With our prerequisites in place, we’re prepared to retrieve the ultimate CPU utilization value in this file.

public void calculateCPUUsage()
 {
   Process cpuMetrics = new Process();
   ProcessStartInfo startInfoForCPUData = new ProcessStartInfo
   {
     FileName = "python3",
     Arguments = "./Utils/idleTime.py",
     WorkingDirectory = "/Home/User/Performance",
     RedirectStandardOutput = true,
     RedirectStandardError = true,
     RedirectStandardInput = true,
     UseShellExecute = false,
     CreateNoWindow = false,
    };
 
   cpuMetrics.StartInfo = startInfoForCPUData;
   cpuMetrics.Start();
 
   string idleStateDuration = cpuMetrics.StandardOutput.ReadToEnd().Replace("\n", String.Empty);
   cpuMetrics.WaitForExit();
   double totalTime = GetTotalTimeDurationForCPUMetrics();
   double idleState = Convert.ToDouble(idleStateDuration.Substring(0,idleStateDuration.IndexOf(".")));
   double CPU_utilization = (1 - idleState / totalTime) * 100;
   addToTestReport.WriteLine("Total CPU usage - - " + Math.Round(CPU_utilization)+"%");
 
   cpuMetrics.Close();
 }

This code snippet presents a method calculateCPUUsage(), intended to determine CPU utilization. It utilizes Python scripts to collect CPU metrics. Initially, it initializes a process to execute a Python script named "idleTime.py", which captures the duration of the CPU's idle state. Concurrently, another process is initiated to execute a Python script named "totalTime.py" to acquire the total time duration for CPU metrics. After retrieving these metrics, it calculates the CPU utilization by subtracting the idle time from the total time and expressing it as a percentage. This calculated CPU utilization value is then reported to a test report file using SpecFlow to output the values.

Take a look at the following output:

-> Total CPU usage — — 26%

The optimal CPU utilization

The optimal CPU utilization range for any Android device can vary depending on factors such as the device’s hardware specifications, the intensity of tasks being performed, and the efficiency of the software running on the device. However, as a general guideline, CPU utilization typically operates optimally when it consistently maintains a range between 20% to 80%. This range ensures that the CPU is adequately utilized to handle tasks efficiently without being overly stressed, which could lead to performance degradation or increased power consumption. It’s important for developers and system administrators to monitor CPU utilization regularly to ensure optimal device performance and battery life.

Benchmarking CPU usage

Benchmarking CPU usage in Unity projects involves evaluating the performance of the CPU within the context of game development and execution. This process entails running specific tests or simulations within Unity to measure how efficiently the CPU handles various tasks, such as physics calculations, rendering, AI processing, and game logic. Developers typically design benchmark scenarios that reflect typical gameplay situations or stress tests to assess CPU performance accurately.

Start by choosing the testing environment. Once the application operates under standard conditions, commence the benchmarking procedure to collect CPU utilisation data over a predefined timeframe. The test may vary in duration, iterations, and the number of devices, all within the same environment, to achieve the optimal benchmarked value. Capture data across all variations and store it for subsequent analysis.

To gain further insights into analyzing trace files using Perfetto, consult the official documentation here.

Happy Testing !!!

⬅ PART 2 || Current |️|️ Next ➡