How to automate SDK testing and sleep through the night — a sensor simulation story

I’ve been developing an SDK for indoor positioning based on sensors used in almost every smartphone.

SDK development is just like app development, but comes with a different set of challenges:

• SDKs tend to be more atomized, often having up to five different versions in production.
• Maintaining backward compatibility is paramount.
• Testing becomes more complex for manual Quality Assurance (QA) teams, as there is no user interface (UI) to interact with.

Each release must be as robust and stable as possible, because every crash or bug impacts the client’s apps. Moreover, any version upgrade requires manual action from the client side.

Where We Started

From the get-go, the company has used a Demo app with our SDK to showcase our technology to potential clients. Each new SDK release includes a Demo app release with a standard test plan:

• Manual testing of new features
• Upgrade test from the previous version of the app (SDK).
• Backward compatibility manual testing is performed by developers while integrating new SDK features.
• Evaluation on ~15 devices for both Android and iOS

Challenges We Faced

By comparing the challenges and the test plan, we were able to see that it’s far from ideal and not comprehensive enough. We then brainstormed the best way to improve the reliability and stability of our product and came up with the following:

• Perform tests on a wide variety of devices to preventively find problematic devices (with missing sensors for example).
• Perform tests on different versions of the SDK to cover backward compatibility, regressions, degradations, etc.
• Write cross-platform UI tests for at least the main feature of our SDK, which is positioning.
• Perform tests on the final, obfuscated binary which is going to be distributed to our clients and cannot be achieved anyhow with unit or internal E2E tests.

So we decided to develop an app that would satisfy all our needs. However, there were a few small flaws in that plan. The main issue is that we use phone sensors to define user position. Also, we test the SDK from outside, so we don’t have access to our internal classes, only public API.

Problem-solving :)

We faced two main challenges on this path:

• Pre-record real user positioning sessions and “play” them on a device farm to test their integrity on a wide variety of devices.
• Replace the internal sensor manager that provides sensor data to the SDK with our custom instance from outside.

I’m not going to talk about recording sensor data in this article, so let’s focus on substitution.

Sensors Disclaimer

Android SDK has a system SensorManager, which can be accessed by context.getSystemService(Context.*SENSOR_SERVICE*) as? SensorManager

If you want to start receiving updates from any sensor — you have to call

public boolean registerListener(
 SensorEventListener listener,
 Sensor sensor,
 int samplingPeriodUs,
 int maxReportLatencyUs
)

Afterward, you will start receiving sensor updates within a given period public void onSensorChanged(SensorEvent event); of your listener you passed to the manager.

Also important class SensorEvent

public class SensorEvent {
    public int accuracy;
    public Sensor sensor;
    public long timestamp;
    public final float[] values = new float[0];

• sensor — enum with type of sensor
• values — float array filled with sensor values(x,y,z, etc.). The meaning of each can be found in the documentation.

Substitution

SensorEvent has a package-private constructor, so you can’t create instances easily, but all fields are mutable, so we can:

• Receive at least one sensor update for each type we need. It’s not necessary, but we want to be sure that all sensors are ready and functioning.
• Modify them and provide it to our system as if it is real sensor data.

object AccelerometerModifier {
    fun modify(event: SensorEvent, with: ModifiedAccelerometerData, timestamp: Long): SensorEvent {
        event.values[0] = with.x.toFloat() * -android.hardware.SensorManager.GRAVITY_EARTH
        event.values[1] = with.y.toFloat() * -android.hardware.SensorManager.GRAVITY_EARTH
        event.values[2] = with.z.toFloat() * -android.hardware.SensorManager.GRAVITY_EARTH
        event.timestamp = timestamp

        return event
    }
}

Replacing Internal Class

It is possible to create a custom Gradle plugin that will manipulate classes during the build process.

To achieve this, we created our own AsmClassVisitorFactoryand provided a custom ClassVisitor. This ClassVisitor goes over each class and is able to apply desired changes. It is capable of replacing methods, classes, types, etc. Then we just passed AsmClassVisitorFactory to Variant.transformClassesWith method in the plugin.

In ClassVisitor we needed to replace the constructor (method) and type, so that whenever someone injects or creates an instance of our class — it gets the class from outside.

override fun visitMethod(
        access: Int,
        name: String?,
        descriptor: String?,
        signature: String?,
        exceptions: Array<out String>?
    ): MethodVisitor {
        return object :
            MethodVisitor(api, super.visitMethod(access, name, descriptor, signature, exceptions)) {
            override fun visitMethodInsn(
                opcode: Int,
                owner: String?,
                name: String?,
                descriptor: String?,
                isInterface: Boolean
            ) {
                if (opcode == Opcodes.INVOKESPECIAL &&
                    owner == "OriginalClassPath" &&
                    name == "<init>"
                ) {
                    mv.visitMethodInsn(
                        opcode,
                        "CustomClassPath",
                        name,
                        descriptor,
                        isInterface
                    )
                } else {
                    super.visitMethodInsn(opcode, owner, name, descriptor, isInterface)
                }
            }

            override fun visitTypeInsn(opcode: Int, type: String?) {
                if (type == "OriginalClassPath") {
                    super.visitTypeInsn(opcode, "CustomClassPath")
                } else {
                    super.visitTypeInsn(opcode, type)
                }
            }
        }
    }

Where We Ended Up

Regression test app

Simple UI, 3 screens:

• Actions — to launch SDK API calls/apps functions
• Info — to display the current app state
• Events — JSON text with all events that happened in the app, mostly callbacks from SDK with consumed time

The QA team selected the top 50 most popular devices from both platforms using our metrics on BrowserStack. Then, we wrote several E2E tests for the most popular scenario, where a user logs in, loads a building, uploads a pre-recorded collection of sensors, and starts the positioning process. At the end, we verify that the user is at the expected point after the expected amount of time.

Now, for every new version of the SDK, these tests run on all the versions that clients are using, including the latest version just before its deployment. This provides the assurance that the entire flow from start to finish is functional, nothing is broken, and everyone can just chill 😎.