Firebase TestLab — A dev tool you must explore!
If you’re part of any tech team you would’ve surely for once heard someone say — ”The bug is not reproducible”. But what happens when this bug starts affecting your app experience and your crash-free users begin to drop?
On our Android e-commerce app, we were facing an internal runtime crash that was not at all reproducible at our end. It started out as few instances and increased in number gradually. This issue was open for at least 7 months but when the sale period came close and the audience on the app increased, the number of crashes increased exponentially.
All hell broke loose when there was a significant impact on the crash-free user percentage — we almost exceeded 1.5% at the crash rate! 😫
The logs on Firebase Crashlytics didn't specify where the crash was occurring (no class/line details). All we knew was that this was happening on our product listing page(got this from analytics) - on Samsung devices with the Android 12 version.
Hours spent trying to reproduce the issue along with QA, and sending out small fixes for this issue(try till you succeed) went all in vain!
Frustrated and close to giving up on this issue, I reached out to people from other teams in case they could reproduce this issue — but all went in vain! While going through the Firebase console to access Crashlytics, I came across Firebase Test Lab and thought of giving it a try and to my luck, TestLab was a guardian angel in helping me and my team quickly fix this long-standing issue.
In the next section, I’ll walk you through the oh-so-simple steps of running tests on TestLab.
How to run a test?
To run a test on Android, Click on the “Run a test” button and select the “Robo“ option (which performs a monkey test on your selected device). For iOS testing, we would need to use XCTest.You can guide the Robo test to perform specific actions by recording a Robo script in Android Studio and uploading it along with your app bundle.
TestLab also provides Instrumentation and Game Loop tests.
Upload your APK / AAB for Android and click on confirm, if you have a script for testing you can upload that as well, click on continue.
Click on “Customize“ to select the virtual device you want to conduct the test. Test Lab provides a range of 50+ virtual devices to run the test on Android alone
In our case we knew the crash was on Samsung with Android 12 — so I filtered it based on the info.
Click on “Run test”
Click on device details on the test — it will take you to the page with all the info about the crash.
Bonus point:✅: it had a crawl graph, screenshots, logs as well as videos of the test performed, which helped us a lot to triage and fix the issue.
After going through all the crawl graphs and the videos, we understood the issue was occurring on a banner on the listing pages. The issue occurred because of the adjustViewBounds used to display the banner image. If you use android:adjustViewBounds="true" and wrap_content as width and height parameters for an image whose source is a URL from the API response, Glide will draw a full-size image irrespective of the ImageView's dimension which will ultimately cause this error.
After we fixed the issue, we ran around 10 tests to confirm the same!! — very cautious & why not!? 😉
After releasing the new build with the fix, our crash-free percentage increased to 99.97% 🥳🕺🏻💃🏻
Firebase TestLab ran tests on real devices and gave us actionable results which helped us get rid of our pain area. To conclude this post, I would say that Firebase TestLab is a friend every native app developer must have ✌🏼
For information related to the Test lab, please refer to the official page!!
