App Re-Engagement: competitive research & A/B test results

App developers spend most of time and resources trying to acquire new users that have never used their app before. Although, keeping the existing users in the app can be more cost effective and it definitely worths optimizing.

Push Notifications are one of the most impactful channels. Sending it is almost free, and we do not need any permission by law like in the case of email. It can be well targeted, sent in the right time and it leads the user seamlessly to the app or the specific page we want. On top of that, we can easily measure the impact.

On the other hand, sending too many messages can annoy the user and can be harmful. Last year a survey by Localytics tried to measure how sensitive US users are to the number of messages sent by one app in a week. While people are getting used to it, the proportion of users discouraged by push notifications is surprisingly high.

Number of weekly push notifications that make the user stop using the app.

Competitive Research

To find out whether push notifications worth bothering for your mobile app, you can check what the approach of the leaders of the market is. For health & fitness apps, I decided to observe the leading workout and meditation apps in the category. As the push notification roadmap usually depends on events the user performs, I decided to cover only one, yet the most important use-case. User opens the app, answers onboarding questions, and signs-up if required. After that, I did not take any further action like starting a workout or checking the payment screen and just closed the app and never opened it again. Then I was observing what notifications the app sends to get me back. To clarify, the experiment took place in Q3 2018 on two devices to double-check (Android phone and iPad), and the notifications were always allowed.

Simplified push notification strategy of leading H&F apps.

As we can see in the table above, only half of the apps sent at least one push notification to the non-engaged user. In the case of Freeletics app, the only notification seems to be time-independent. So basically every user receives it no matter what his history with the app is or how likely he is to churn or pay. Most coherent notification roadmap was seen in the Runtastic Results app. Their push notifications looked very personalized as they always include the user’s name. Most of their messages were telling the user to start the training plan or explaining how easy it is to improve his lifestyle. On the figure below you can see what the notifications were like.

Example of re-engagement push notifications sent by Runtastic Results app.

I have even captured one gender-specific message. You can see the difference in communication with male and female audience in the following figure.

Example of a gender-specific push notification.

It is not clear why most of the apps did not take advantage of push notifications. But as most of these companies definitely have the budget and headcount to experiment, I assume that they have tested notifications earlier, but the results were not satisfactory. This method of observing competition is quite time-consuming, and you need some extra devices to test on. However, it can give you some idea of how many push notifications you should send to your users and when. You will also see many creative messages that can inspire you when building your own push notification strategy.

Testing and evaluation

In order to send messages that are relevant and have the potential to improve user’s engagement, the developer should define the structure of his notifications. As mentioned earlier, I have done some research on the competitor’s strategies. Making decisions based on competition’s best practices is not very reliable and therefore an A/B test could be very beneficial for many developers. We created a simple push notification roadmap consisting of four messages. You can see the messages in the following figure. The message took the user’s name into account as well as gender. For male users, there was an image of Jackie (male coach at Fitify), and for female users, there was Adele (Fitify’s female trainer).

Tested notification roadmap of the Fitify App.

Using Firebase A/B testing feature, we defined the control group who did not receive any messages. The experimental group received all 4 messages (unless they finished their first training). Only users who signed up were considered to avoid the bias from users who did not even gone through the onboarding.

The most important information for us was the percentage of users who started the first workout and the 4–7 days retention. Both of these metrics were significantly improved. The proportion of users who began their first workout grew about 5% (median), and the retention grew by approximately 6%. For both of these metrics, we are more than 99% confident they were improved.

We can hopefully improve the retention even more by testing frequency, times, advanced targeting, or the creative part of the message. I may share the results as well so make sure you hit follow on my Medium or Twitter