How to measure the impact of your product features in the mobile app ecosystem
This article is meant mostly for product managers who are working in the mobile app ecosystem and for someone willing to learn more about the complexity of analytics in mobile apps.
A little context, I faced this problem while executing a complete design overhaul according to the material design of the 91mobiles mobile app and got baffled by the lack of literature on this matter. Hence I am putting it out there for someone who might be in a similar situation.
“How to measure the impact of your product features in the mobile app ecosystem”.
On the onset, this seems like a trivial problem. Mobile has been around for over a decade and someone somewhere would surely have designed a full proof solution to this but as it turns out it is not the case. A quick google search on “measure the impact of your app features” throws up relevant blogs from several 3rd party integration going on and on about all to familiar terms of Acquisition, Cohort, CAC and asking you to use their SDK to help you measure them. But using a 3rd party SDK increases your app size by around 200–300 Kb, would set you back by 1000$ a month and above all the data will start pouring in from your next app update which is a deal breaker if you are already in development phase for the app feature you are planning to roll out.
What makes the problem unique to the mobile app ecosystem?
1. An app is a shipped software — Unlike the web platform, where any change that you make are reflected immediately for all users, it takes time for the app to be available to all users and even then only around 70–80% users actually update their app. So unlike on the web, where you could directly compare numbers for the week after the release with the numbers a week prior to the release, it is just not feasible to do the same on the app.
2. The difference in user behavior due to different install date — The user behavior varies considerably according to the time elapsed since they installed the app. The behavior of a user who just installed the app has more chances to actually use the app relative to a person who installed the app 2/4/6 Weeks ago.
3. More Engaged users are more likely to update — Updating a particular app requires effort from the user end, so it is safe to assume that users who are updating the app are more loyal and engaged users, hence comparing their data with users who did not update would result in a wrong conclusion.
The traditional solutions which work for the web and how it fails for mobile.
Possible Solution 1 — I will roll out an update and measure data for users who update versus those who don’t.
Classical Mistake. I know I did it :). This solution would throw up false positives, as any user who is actually updating your app would be an engaged user and more likely to have better metrics.
Possible Solution 2 — I will roll out an update and wait for 30–45 days for the app user base to stabilize and then measure.
One word “Seasonality”, The behavior of users varies on a month on month basis. So the difference you are measuring could just be seasonal changes.
Possible Solution 3 — I will roll out an update and compare numbers for the previous week to the next
Traditional web approach, though on the web you might see a positive or a negative step in a day of rolling out the feature, It takes time on the app and you will have to wait for 2–3 weeks, which again turns this into Solution 2.
Possible Solution 4 — I will run A/B test where half of the users who update/new users see A and the other half see B.
Theoretically correct, will work well for small features such as changing a particular page or changing a particular workflow but if you are going for a big change, this requires significant tech bandwidth which is not readily available at most tech companies. This would also require to keep the previous code and assets in your app which would increase the app size and would require higher testing efforts to test both versions. Additionally would require you to set up an internal A/B testing architecture or integrate a 3rd party SDK. Certain metrics such as retention are harder to track by this method.
The Solution
“Do a 50% rollout on Play Store and compare data of only new users who receive previous app version (A) to new users who receive the newer app version (B)”
Play Store allows to do a 50% rollout and the new users installing your app are also included in the 50% rollout. Google Analytics allows creating a segment by date of first session and app version. The option for the same is present under New Segment > Date of the first session.
The ingenuity of the solution is that it relies on readily available tools, Play Store, Google Analytics, does not require additional tech bandwidth, caters to all the core problems mentioned of the app ecosystem and does not require a 3rd party integration.
Below is a screenshot of the report generated using the above method.
Conclusion
So there you have it, a solution to measure the impact of your feature in the mobile app ecosystem. If you have devised any other quick method which addresses the core problems while using little or no additional tech effort, Do leave a comment or drop me a message, would love to hear your solution and add it to the list for everyone to see.
