How sending a Firebase push notification took down our app for 10 minutes
At Candide, we’ve been very careful not to become “yet another annoying push notification factory” app. As a result, almost all of our push notifications are for things like “digs”, comments on your posts, replies to your comments. We don’t generally send re-engagement push notifications.
We do periodically send push notifications to any users who are still using old versions of our app when we have released new features. We did this when we released our automatic Plant ID feature, for example.
One day, we sent a push notification via the Firebase Cloud Messaging UI to all our users on old app versions to advise them to update to the latest version and we ended up taking down our server. Here’s how…
Firstly, it’s worth noting that although our userbase is roughly a 50/50 split between Android and iOS, Android users tend to update to the latest version more slowly than iOS users. The charts below show this.
You can see that the Android releases not only have a slightly slower upward slope but also longer tails with spikes of usage lasting well into later releases. This meant that the number of users being targeted for our “upgrade your app” push notification was very large on Android.
Secondly, and most importantly, the way the React Native Firebase package seems to handle background push notifications was, let’s say, unexpected.
When an Android app using React Native Firebase receives a push notification when the app is closed, it starts the app in full, which means that all the data that the app would normally load (which was four tabs worth of content from our GraphQL server) was loaded. Interestingly, this does not happen on iOS. This meant that our GraphQL server and backing microservices were asked to handle around 100,000 requests pretty much all at once. Requests backed up and cleared after around 10 minutes.
What we learned from this is that a) React Native Firebase/React Native does unexpected things and b) don’t do that again.
We have since updated React Navigation to the latest version which I suspect might prevent all the data loading on a push notification, but ultimately we have since just decided not to send such a huge batch of notifications at once until we know. When we have figured out how to mitigate this (beyond not sending lots of notifications at once) I will write up our findings here.
