The Most Important Push Notification Metric

Yes, Push Notifications are a viable retention tactic. And app developers have prudently started to take all the right measures to optimize flows to get users to opt in to push notifications. But once that happens, are we as app developers doing right by the user to ensure that we are using this sacred push notification channel optimally? It doesn’t sound like it — in fact, we might be abusing the channel, and not realizing the consequences of the abuse.

I’ve seen this pattern way too often now:

1. Build Web app
2. Send Emails
3. Build iOS App
4. For every email we send, send a Push Notification

In most cases, this pattern is just unacceptable and it can overwhelm your users to the point of no return.

For example, this is Pinterest’s Push Notification Settings page:

This is Pinterest’s email settings page:

Emails and Push Notifications are fundamentally different

Just to spell this out…

Although 50-65%of all emails now get opened on mobile devices, email is still an asynchronous form of communication. Emails can be ignored.

Pushes are invasive and your phone is impossible to ignore (I’d love for everyone to heed Fred Wilson’s advice). There is no “unsubscribe all” option for pushes like there is in the footer of most bulk emails. In fact, it is much easier for users are to just delete your app rather than to find your settings page and turn off push notifications (or to turn off notifications for your app globally from Settings.app).

Sure, sending more pushes could mean you see an increase in retention from some of your users. But can you prove that you are not losing more users by sending (more) pushes?

Churn — the most important thing about Push Notifications you’re probably not measuring

It’s important that your app is instrumented correctly to measure and track all the right things before diving in.

1. First, track repeat usage behavior — Intercom has a great writeup on Measuring Retention (looks like Google Analytics is testing cohort retention capabilities too)
2. Log when a push token is created for a user
3. Log when push tokens are no longer valid (make sure you use “Soft” Deletion — more on Stackoverflow)[1]
4. Attribute Pushes — sending parameters with your pushes so that when a user swipes through, your app can make sense of which specific push the user swiped on (this warrants a whole other post)

At the very least, you should have a device tokens table that looks something like this:

• token,
• user_id,
• created_at,
• deleted_at

Table 1 — An example device token table

Now here’s where tracking #2 & #3 from above come in handy. You should be able to write queries that tell you when cohorts of push tokens are “churning”:

Table 2 — Push Token Attrition

Push tokens can most likely be invalid if a user has deleted your app or if the user has turned off push notifications for your app[1]. Either way, you are unable to send pushes to that user any longer.

While retention is the metric we most commonly look at — the glass half full metric — it is just as important that we look to see if the pushes we’re sending are causing users to churn out. In the example in table 2, 10 tokens were no longer valid after day 1. A total of 20 tokens were no longer valid after day 3, which means 20% of the users who had signed up and opted for push notifications are no longer reachable. What do you think that is doing to your retention?

My $0.02

1. “Transactional” Pushes Only: Yes, Gilt sends pushes at 9am. And so does Lumosity. That doesn’t make it ok for you to do it if a user has just blanket consented to accepting push notifications from your app. For example most commerce companies, send out mass marketing emails (“daily sales”) and transactional emails (“your item has shipped”). Once the user has opted in, you could automatically opt your user in for the equivalent of just the transactional emails, but make sure to either prompt the user user to opt-in for daily marketing pushes or explicitly point out that the user has been opted in for daily marketing pushes.
2. Volume: If the user does opt-in for a high volume of pushes, ease them in — don’t flood them from day 1, and most importantly if you find that a user is not swiping through on your pushes after a while, consider turning the volume back down or shutting them off altogether.
3. AB Test your pushes: You should be able to test timing and content of your pushes. Test what works for your users before rolling it out on a broad scale.
4. Crawl, Walk, Run: (This may sound a little counterintuitive to 3, but) If you’re a smallish startup trying to figure it all out, don’t go build the most exhaustive personalized push notification system in the world. Remember that your users have made it through some very complicated funnels to get to this point — found your app, tapped on the “get” on the AppStore, typed in their password or TouchId, installed it, launched it, signed up, opted-in and they like what they see. Keep them engaged but you don’t have to go overboard. Start with the basics, figure out how to optimize what you’ve started with before you decide that what you have built is no longer sufficient. When in doubt, read this PG article: Do Things That Don’t Scale.

The question you should be asking yourself as a developer sending pushes is, how would I feel if I got this push and what if it were delivered to me as a text message instead of a push?

@ai

[1] — this can be conflated by factors such as an OS update, or if a user resets his or her device

Thanks @danielpearson & @drinkzima for reading drafts of this.