Using AndroidX Benchmarking

What does ‘Benchmarking’ mean?

To Benchmark means evaluate (something) by comparison with a standard. In software development it means, to profile and improve code execution(generally in terms of time taken and resources used). Any code intended for a system has to run in the confines of the available system resources. Minimizing the use of these resources and still giving the right output. That is true optimization!

Responsiveness is one of the key factors in Mobile Application development and has become a new criterion for optimization.
It pertains to “how fast the app loads?” or “how smooth the animations are?”. This has generally been directly related to the available memory and processing power, hence the constant upgrades in flagship models. However an app developer has to support and target lower end models. The app shouldn’t just be running in them, but be responsive as any higher end model. This is where optimizing code comes in. A crucial step after feature development.

In this post, we take a look at the concept of Benchmarking, try out the Jetpack Benchmark library and see how it can help in improving the performance of an android app.

What is Code Benchmarking?

Simply put, it is a way to measure the performance of a code and make improvements for optimal results. Benchmarking is measuring code performance in a standard environment in order to see improvements on further code optimizations.

Let’s take an example, say function takes an integer list as input and gives a sorted list as output. The time taken by the function depends on the algorithm used to sort the list,… theoretically. In a real world scenario, the time of execution also depends on external factors such as the environment the code is run on, memory allocations in the system etc.
So, if you were to measure the performance of the function on your android device you would have to run it multiple times and take the average.

The Jetpack Benchmark Library

The Jetpack Benchmark library allows you to quickly benchmark your Kotlin-based or Java-based code from within Android Studio. The library handles warmup, measures your code performance and allocation counts, and outputs benchmarking results to both the Android Studio console and a JSON file with more detail.

One Off benchmarking

To quickly perform one-off benchmarking, do the following:

• Add the library to your module’s build.gradle file:
  project_root/module_dir/build.gradle

dependencies {
    androidTestImplementation "androidx.benchmark:benchmark-junit4:1.0.0"
    androidTestImplementation "junit:junit:4.12"
    androidTestImplementation "androidx.test.ext:junit:1.0.0"
}

• To disable debugging in the test manifest, update your <application> element to force-disable debugging temporarily as follows:
  project_root/module_dir/src/androidTest/AndroidManifest.xml

<!-- Important: disable debuggable for accurate performance results -->
<application
    android:debuggable="false"
    tools:ignore="HardcodedDebugMode"
    tools:replace="android:debuggable"/>

• To add your benchmark, add an instance of BenchmarkRule in a test file in the androidTest directory. For more information on writing benchmarks, see Write a benchmark.

@RunWith(AndroidJUnit4::class)
class MyBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    @Test
    fun benchmarkSomeWork() = benchmarkRule.measureRepeated {
        doSomeWork()
    }
}

Here, measureRepeated code block takes care of making repeated calls to doSomeWork() in a standard environment to benchmark.

You should reset any caching/initialisations to make sure each doSomeWork() invocation runs the compete code you wish to benchmark. You should use runWithTimingDisabled to avoid including the reset code in benchmarking.

@RunWith(AndroidJUnit4::class)
class MyBenchmark {
    @get:Rule
    val benchmarkRule = BenchmarkRule()

    @Test
    fun benchmarkSomeWork() = benchmarkRule.measureRepeated {
        runWithTimingDisabled {
            resetDoSomeWork()
        }
        doSomeWork()
    }
}

Full project setup

To set up for regular benchmarking rather than one-off benchmarking, you isolate benchmarks into their own module. This ensures that their configuration, such as setting debuggable to false, is separate from regular tests.

The code you wish to benchmark has to be in separate library module that can be included as dependency in the benchmark module.
To create a benchmak module :

Enable AS template

1. In Android Studio 3.5, click Help > Edit Custom Properties.
2. Add the following line to the file that opens:
   npw.benchmark.template.module=true
3. Save and close the file.
4. Restart Android Studio.

Create the module
The module contains

1. The benchmark plugin
2. Custom runner androidx.benchmark.junit4.AndroidBenchmarkRunner
3. Pre built proguard rules
4. The benchmark library

Run the benchmark

In Android Studio, run your benchmark as you would any @Test. On Android Studio 3.4 and higher, you can see output sent to the console. To run the benchmark, in the module, navigate to benchmark/src/androidTest and press Control+Shift+F10 (Command+Shift+R on Mac).
On running the benchmark(which has to be on a device), the following screen come up for the length of the test

And the following result is output :

$ adb shell am instrument -w -r    -e debug false -e class 'com.healthifyme.benchmark.ExampleBenchmark#benchmarkLog' com.healthifyme.benchmark.test/androidx.benchmark.junit4.AndroidBenchmarkRunner
Started running tests
benchmark:   27,575 ns   0 allocs   ExampleBenchmark.benchmarkLog
Tests ran to completion.

You can target to reduce the time or memory allocation on further optimization of the code.

Configuration errors

The library detects the following conditions to ensure your project and environment are set up for release-accurate performance:

• Debuggable is set to false.
• A physical device, not an emulator, is being used.
• Clocks are locked if the device is rooted.
• Sufficient battery level on device.

You can suppress some errors as warnings by using instrumentation argument for example, to suppress Debuggable error :

android {
    defaultConfig {
        testInstrumentationRunnerArgument 'androidx.benchmark.suppressErrors', 'DEBUGGABLE'
    }
}

Note: Suppressing errors allows the benchmark to run in an incorrectly configured state, but the output of the benchmark will be intentionally corrupted by prepending test names with the error. That is, running a debuggable benchmark with the above suppression will prepend test names with DEBUGGABLE_.

To wrap it up!

Benchmarking of code isn’t the first step to optimizing. It helps only in measuring code performance in a close to standard environment. The code still needs to be profiled before benchmarking. That would help in finding the expensive operations that are worth optimizing. As an added bonus, benchmarking will also help clearly measuring the improvements added to the code.
Happy Benchmarking!!