A Guide to Mobile Analytics 🔧 Part I: Beta Distribution, Bug and Crash Reporting
This series of articles is a regularly updated guide into the world of mobile analytics. I grouped tools by their main functionality, presenting pros and cons of each. We start with tools for beta distribution and bug and crash reporting needs.
Realising the first version or an date of your app always requires testing it beforehand. The first user impression matters. If something didn’t work out as expected the user might be gone forever. Testing is a core concept that helps building trusted relationships with your users.
I believe that the adoption of a tool is a good indicator of how reliable it is. You do not want to end up spending time implementing a tool that barely has users, is likely to be buggy, and possibly disappears soon.
In terms of adoption, there are three outstanding solutions for bug, crash reporting and beta distribution needs: TestFlight, Fabric (Crashlytics) and Hockey App.
I have defined ten criteria besides adoption that can help you choose the right tool. A more detailed explanation of each of the criteria is below.
Platform Support
The number one reason to favour one of these tools over another is the platform or platforms your app is built for. If your app does not run on iPhones or iPads, you can ignore TestFlight. If your app has both iOS and Android versions it makes sense to combine TestFlight with Fabric or Hockey App. The last two are a necessity for Android and are complementary to TestFlight’s features. However, it is a common good practice to have “one source of truth” for different platforms. Either Fabric or Hockey App can perfectly fit the role of this “source”.
Ease of Setup
The next important criteria is the time your team needs to spend implementing a certain tool. In this regard Hockey App is the most time consuming. If gathering feedback and managing lists of testers is not a necessity I would rather go with Fabric or TestFlight.
Review Time
For Android there is no review required — your app can be instantly tested. Even for deploying in the Google Marketplace there is no review procedure.
For Windows Phone the reviews takes up to 5 working days. There are certain types of betas that can be available faster but in this case your beta and your app are different things and you would have to manually transfer data from one to another for an official release.
For iOS the review time differs between the type of testing you want to organise. If it is an internal test among your colleagues or family members you can add up to 100 testers through you Apple Developer account and your builds will be instantly available to these people.
For external testing the TestFlight review time for a new app and each release is about 24 hours. However there is a trick how you can save your app from 24 hour breaks. The next time you submit changes to the same version of the app, do it without launching a new one.
Maximum Number of Testers
For Android and Windows Phone — you can invite an unlimited number of testers using either Fabric or Hockey App.
For iOS the maximum number of testers is restricted by Apple and applies to all tools including TestFlight.
Ease of Inviting Testers
Bulk inviting testers to your app can be a minor feature if your team is small (let’s say up to 20 people) and you don’t have external testers. In this case Fabric can serve you well as this tool allows inviting one tester at a time by manually adding his email address. With TestFlight and HockeyApp you can invite a list of testers by uploading a CSV file.
Segmenting Testers
Segmenting testers is a feature that is available in each tool. The only difference is user friendliness of its implementation. It was hard to find tester grouping functionality in Fabric, for a long time I thought they just don’t have it.
Multi Version Testing
Multi version testing can significantly speed up the process of building and launching an app. It is a vital feature for apps in highly competitive markets. Both Fabric and HockeyApp provide multi version testing. A few months ago, TestFlight also added this functionality in beta mode.
Tester Onboarding
TestFlight tester onboarding is harsh. You send an invite, the user follows the link from the email and gets referred to the AppStore to download TestFlight. The user flow is very confusing. From experience, only around 5 to 20 percent of invited testers complete the process.
Fabric and HockeyApp provide a way better tester onboarding experience. Testers get an invitation via email, follow the link and press a download button after entering their name. The beta app is now on their device.
Gathering Feedback
The way feedback functionality works on TestFlight:
Yeah, that is it. Testers can send an email! Not very advanced feedback tooling. But Fabric doesn’t have any at all. From this perspective HockeyApp is the only proper way to go. Testers can even create video records of things that are confusing or work wrong to them. For big organisations where different departments or stakeholders are involved into the app testing process HockeyApp is the only viable solution on the market.
Performance Reports
TestFlight has very basic crash reporting — it simply shows the number of crashes per user. The absence of detailed crash reporting is one of the main reasons why TestFlight is commonly used in tandem with some other tool. Both Fabric and HockeyApp provide a broad range of bug, crash, engagement and event reports. Fabric recently even added a separate revenue report.
Unique Features
I believe Fabric’s popularity is driven by the broad range of third-party integrations. It is way easier to get data out of this tool. Being able to easily share results via GitHub, Slack, Jira makes performance visible and accessible to everyone in the team or even within the organisation.
Unique to HockeyApp is to me its active support community via Slack or email. When you need real people to help, you can get it fast with this tool.
After giving a talk on this topic I was asked about my favourite mobile analytics stack.
I can’t name a one size fits all solution. For example I definitely prefer Fabric’s user experience over HockeyApp. But the former does not allow you to invite hundreds or thousands of testers at a time or group them easily by certain criteria.
You need to identify your problems and define your goals to choose a stack that fits your product. Be careful choosing, ho-ho-ho… as migrating to another tool is honestly a pain and a big waste of time for your team.
Reach out to us at Blackbelt Labs if you need help organising mobile user testing and setting up an analytics pipeline.
