6 ways to improve QA testing of an “invisible” SDK
How Streamroot tests our iOS SDK
As the title suggests, this article details 6 suggestions to make QA testing more effective when you’re dealing with an SDK that has no visible effect on the user experience. I’ll be exploring these suggestions through examples from a tool we recently implemented at Streamroot for testing our iOS SDK.
When I arrived at Streamroot a few months ago I was asked to help revitalize the app that we use to QA our iOS SDK. The team had already created an automated testing suite for QA on desktop (which we presented at Demuxed in October), but our mobile testing app still only had basic functionality.
The Streamroot SDK enables hybrid CDN and peer-to-peer delivery for video streaming. Acting as a proxy, Streamroot determines the best source for the video segments and directs the video player where to get the next segments it needs. This technology is difficult to QA because it requires several devices to run videos simultaneously for an effective test and, even then, the effects of the technology are almost entirely invisible from the user perspective and hard to qualify from a single session. As we need to make sure that our technology works across all players, devices, and network conditions, it’s critical to do effective and exhaustive QA tests with each release. Testing the SDK allows us to bring solutions to the challenge we face while building the Streamroot SDK.
Previously, we were using a fairly simple app that only allowed you to set configurations for the SDK and launch video streams. Using this app, a typical QA session involved gathering several devices, updating to the latest SDK on each device, setting your configuration, and finally letting each device run the stream. Each device showed some high-level stats on SDK performance, but in order to see precise details about the behind-the-scenes functionality, you would need to set up tools like Charles Proxy to intercept and view network calls. All in all this process was labor-intensive, required repeated boilerplate work, and took a lot of time. My goal was to design a tool to make this whole process quick, easy, and effective by cleaning up the UX, sharing test configuration between devices, creating visibility into the SDK, making it quicker to launch configurations, and allowing for more automated tasks.
1 — Tool design
Design is often overlooked when developing internal tools. While end users may never interact with the tool, it is nevertheless important to design a clean UX for your fellow engineers as your design choices can have a large impact on your colleagues’ workflow.
The first thing that I did with our QA app was to implement a new design. The intention here was to improve both aesthetics and ease of use. Our QA engineers should be able to navigate the tool easily and have no trouble executing any necessary operations. This clean design is intended to decrease the amount of time and effort that goes into manually setting up and maintaining tests.
The new app is laid out in a more intuitive way. Buttons for creating new configurations are clearly marked and are distinct from those for launching already saved configurations. Long tapping on a saved configuration brings up a context menu with quick options to edit, delete, and duplicate the configuration. When in a stream, you have the option to see base statistics on how the SDK is performing. These stats are presented using a customized graph which quickly gives an idea of the current performance.
2 — Sharing test configurations between devices
The next challenge was that a lot of QA time was spent inputting configurations on various devices. Often this involved inputting the same configurations over and over again, and a single mistype could cause a false negative test. The solution here was to connect to an internal database that stores our frequently used configurations and stream URLs. With this solution, each device has access to an up-to-date set of valid test configurations and our QA engineers need only to select the correct configuration to launch a given test.
For our purposes, we separated the idea of test configurations and video streams, as you should be able to use any configuration with any video stream. Our QA app allows you to search for the stream URL to use with a desired configuration. The stream URLs are stored in the database along with their format and type (multi-bitrate or single bitrate and live or VoD). Each URL also has tags that describe various features about the stream or the source (CDN, multi-audio, subtitles, etc.). This setup is intended to allow you to search for streams by feature rather than by content as in a typical video stream database.
3 — Remote device logs
How do you test something you can’t see? This is one of the major challenges for QA testing the Streamroot SDK and many other technologies that are invisible from the end user perspective. As I mentioned above, we were previously using tools like Charles Proxy to get an idea of what was going on behind the scenes. While Charles Proxy is great, it is time-consuming to set up on each device, it only gives insight into network activity, and it doesn’t support automation.
Our SDK logs every event that happens internally, which is great for testing and debugging. However, these logs are, of course, only enabled when running in debug which means they’re not available to our QA engineers who use release mode. Besides that, in order to read device logs, you would need to run the app from Xcode and you could only access the logs of one device at a time.
The solution here was to make a new build configuration identical to the release configuration except for one custom compiler flag. This flag allows us to expose logs from the SDK only when the app is built for testing and not in the version of the SDK that we distribute to customers.
From here, we use a companion system to send all device logs to a centralized computer where we can organize and analyze the test outputs. We use a simple server to coordinate communication between desktops and devices, using HTTP requests and WebSockets, so any local computer can subscribe to receive live logs from any connected device.
On desktop, we have two available clients for receiving these logs. The web client is shown above and is written in React.js. For the command line lovers, we also have a CLI written in bash and JavaScript that has the same functionality as the web app.
4 — Deep linking
So far we still have the issue of having to enter the same configurations on each device you’re testing with. Sure, using our shared configurations database makes this pretty quick, but it’s still a manual process that needs to be done on every single device. For this, I took inspiration from our web testing tool that allows you to create links that generate test pages. These links contain all the necessary information to select a player and set the SDK configuration. For iOS apps, we have access to deep linking using a URL scheme. This allows us to construct links like the one shown below:
When the app receives a deep link, it decodes the parameters provided and launches directly into a test with the given configuration.
Deep links are great for launching tests quickly because as long as every device has the same link, they can all open the exact same configuration. This means that you don’t have to select the same configuration on each device and you don’t have to worry about accidental typos since all the configuration information is contained in the URL.
One problem still remains: you have to send this link to each device and then manually tap on it! This may seem like a small amount of work, but it can quickly add up for multi-device tests. If only there were some connection between a centralized desktop and several QA devices…
5 — Remote test launching
As I mentioned in #3, we have an intermediary server for sending logs from several devices to one desktop. Well this server is also useful for sending messages in the other direction, which means we can generate deep links on the centralized desktop and send them to all connected devices simultaneously.
This makes it entirely painless to launch a specified test in an instant on several devices. Simply write your configuration in the web app or CLI mentioned above, generate a deep link, select the desired devices, and send your link. All the devices will immediately spring to life and begin running your test.
The important caveat here is that the QA application needs to have an active connection to the intermediary server. The app therefore needs to initiate this connection and it must be open and in the foreground or else the connection will be closed.
6 — Non-traditional automation
A big part of making QA painless is automating as much of it as you can. In our case, automated tests provide the additional benefit of giving us access to large device farms so we can run the same tests concurrently on several devices to simulate real production traffic.
A traditional use of automation would be to navigate through the app and perform different actions and analyze the UI to ensure that it functions as anticipated. That style of automation doesn’t exactly work for our SDK because you can’t directly interface with the SDK and there’s no way to tell if it’s working from the UI. So we had to get creative.
Rather than testing a series of actions to produce an expected UI, we test different configurations and analyze the output logs for expected patterns. The configurations are directly injected using the deep links discussed above, so it’s quick and easy to get each test up and running. We also have a couple testing scenarios that mimic simple user behavior such as watching the stream without interruption, or frequently pausing and seeking in the video. So finally, a single test is made up of the SDK configuration, the testing scenario, and the devices running concurrently. Using these three points of configuration, we are able to simulate a variety of real user activity situations and look for patterns in the output logs to determine whether or not the SDK is working as expected.
Wrapping it up
In summary, our QA app utilizes a few key features to make testing our iOS SDK fast, painless, and effective. Our testing tool design makes it quick to navigate and run different tests on a single device. A centralized, shared database of SDK configurations allows us to reduce repeated, manual work on multiple devices. Sending device logs to a remote computer gives our testers visibility to key internal processes. Encoding configurations as deep links further reduces the manual labor required to set up tests. These deep links can be sent from a remote computer to several devices at once which enables easy concurrent testing. Finally, we can combine SDK configurations and automated testing scenarios with analysis of output logs to adapt automation for use with our SDK.
Testing needs to be quick and easy so that it can be done often and thoroughly, decreasing bugs and increasing confidence in your code. If you have any suggestions, comments, or other interesting testing experiences, share your thoughts below! And if you’re interested in working on these kinds of engineering challenges, check out our careers page for open positions!
