Mobile: Feature Flags, Remote Configuration, and A/B tests at Crunchyroll

The Android team at Crunchyroll had their share of legacy solutions for remote configuration over the years. In this blog post we’ll go over our key findings when working with feature flags and remote configuration to allow for a faster release cycle and a nicer app overall.

Background

Initially, the Crunchyroll Android app had no feature flag infrastructure in place, which made it impossible to keep shipping regular updates while working on something bigger in the background. Some features take a couple of months of development and testing, which means we just didn’t ship the smaller improvements or fixes while this feature was being developed. However we knew that if we had the feature hidden behind a flag, we’d be able to continue shipping other updates.

We also didn’t have any remote configuration or A/B testing options, which made it impossible to experiment. We had to get all the decisions “right” from the start as it could only be changed in the next release.

We had to do something about both problems, preferably with a single solution.

Overview

We have an in-house Config Delta service responsible for providing the latest config. We integrate with it by supplying a JSON schema of the config for our app. The JSON schema is uploaded to an S3 bucket on every commit to the main git branch, which then gets picked up by the Config Delta service.

Key takeaway #1: don’t nest the config more than 2 levels.

We decided to not nest the config more than 2 levels to keep it simple. Here’s a good example of a feature flag defined in the JSON schema:

Here you can see the 2 levels: commenting is the parent and is_enabled is one of the configuration points.

Key takeaway #2: load config delta, merge it with default

We also package a default version of the config into the app since Config Delta service only responds with, well, delta of the config. It only returns the modified fields, which differ from defaults.

Key takeaway #3: validate the loaded config before merging

The app is also responsible for validating the delta it received from the backend before merging it into defaults. If the config delta doesn’t pass validation, it is not merged and only the defaults are used.

App start time

We wanted to keep the app start time low so we were hesitant of loading the remote configuration on every app start. But, since this API call is executed in parallel to some other calls we make (like fetching updated info about the user), we didn’t see any significant increase in the app start time.

Key takeaway #4: consider the impact on app start time

One optimization we’ve made was kind of a low-hanging fruit — remote configuration is still loaded every time the app starts, but we’ve adjusted the logic when the splash screen is by-passed.

The Splash screen waits for the remote configuration API call if:

• This is the first app start
  It is important to know the freshest A/B test details even when the app is launched for the first time. The Onboarding flow is a prime specimen for experimentation.
• This is the first app start after app update
  With the new version of the app, there might be updates to the config schema.

There’s a great blog post from Firebase Remote Config team on different loading strategies.

Consolidating local and remote feature flags

We used local feature flags to be able to continue shipping new versions of the app to users while a major feature was being developed. We wanted to consolidate local feature flags with remote configuration while keeping the safety of using local-only values.

Key takeaway #5: remote_configurable is king

In the end, we adopted the idea of defining a new remote_configurable field for each feature flag. This field should be false until the development of this feature is complete and ready for rollout.

If someone accidentally enables a feature in production but remote_configurable is false, then the feature will not actually be enabled on our users’ devices. This idea was introduced to us by this SoundCloud blog post.

This way, we can reuse the same feature flags when developing the feature and rolling it out to users.

Conclusion

Feature flags and remote configuration are never as simple as we would like them to be. Having a clear goal with strict guidelines makes it just that much easier. Stay tuned for our blog post about Mobile Release Trains!