Introducing Crunchyroll Config Delta

Experimentation as a first class citizen

Authored By Dionis Spinei & The Experimentation Platform Team

On Experimentation

In modern application design, experimentation is critical towards optimizing user experience and enables product teams to evaluate features against key performance indicators. Additional controlled feature rollout helps us move quickly and safely, as well as deploy changes more frequently. Applications across web, mobile, and living room have different content consumption models requiring means for per-platform experimentation for unique user flows. At Crunchyroll we support more than a dozen different devices, leading to the need for a unified, easy to integrate solution across platforms on how to approach experimentation and feature rollout. Product owners need mechanisms to evaluate and analyze the experiment consistently across platforms.

To address these needs we have developed the Config Delta platform with some key design principles, which are outlined here, that we feel would be useful to a broader audience.

Traditional implementations of an experiment often boil down to having a feature toggle that activates a certain code path. The value of the toggle is calculated based on the parameters of the experiment: traffic allocation and eligibility criteria. We historically used feature toggles extensively in Crunchyroll, and they often become a significant maintainability burden as they make the code flow more obscure. Especially in the case of experimentation, the feature flags are ad-hoc and cumbersome. Consider the example:

let upsellButton = {
  text: 'button',
  color: 'white'
};

if (inExperiment(E10_UPSELL_BUTTON)) {
  upsellButton.text = 'experimental button'
  upsellButton.color = 'blue'
}

renderButton(upsellButton)

But there’s no actual business logic in the conditional itself. The upsell button configuration could be resolved to the final values at configuration resolution time. Thus, we set out to reduce the implementation code to something like:

renderButton(config.get('upsellButton'))

It should be clear what is more maintainable and requires no followup after the experimentation is complete.

Why Config Delta?

Our approach led us to create a microservice with the following design goals:

• No application conditionals for experimentation or feature toggles
• No application knowledge about ongoing experimentation
• Support for web, native, backend, mobile or living room applications
• Ensure configuration consistency to avoid getting applications into invalid configuration states
• Seamless failover without impacting user experience in case of communication or consistency failure.

To achieve this we kept all the experiment configuration resolution and user allocation on the microservice’s side. The resolution process produces a JSON payload that contains user-specific dynamic partial configuration that is merged into the application’s default configuration. This is called the Config Delta.

The application consuming Config Delta has a default working configuration that defines its base user experience. During initialization, it will retrieve the delta config, a partial configuration that defines the delta between the base user experience and the experience we want to present to this specific user based on his eligibility status for current active experiments. The Config Delta contains only configuration values and leaks no information about whether the user is in a special flow or why.

JSON Schema

Given the above mechanism to deliver user experience variations to applications what happens when one of those pesky users of the experiment configuration tool makes a mistake? How do they even know what values are expected for a given configuration? The configuration is called upsell_button.color, but it could be in any of a number of different formats.

To define and enforce the configuration format we use JSON Schema. This is a convenient standard that allows us to validate that the experiment configuration will be handled correctly by the client application. The JSON Schema is defined in the client codebase and uploaded to a common S3 bucket at build time, also keeping track of the application’s build version. The application then sends the build version to Config Delta calls, so that it can be validated against the appropriate schema. The application also ships with a schema validator library and revalidates the response on its side, to avoid issues in case of Config Delta failure.

The second enormous benefit of using JSON Schema to define the configuration structure is react-jsonschema-form. This library provided us an easy mechanism to automatically generate experiment configuration forms for our admin interface.

For example, given the following client schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "additionalProperties": false,
  "properties": {
    "STORE_BUTTON_CUSTOMIZATION": {
      "properties": {
        "background_color": {
          "default": "#FFFFFF",
          "enum": [
            "#F47521",
            "#EF4323",
            "#213944",
            "#2ABDBB",
            "#FAB818",
            "#73253E",
            "#EDEDED",
            "#59595B",
            "#000000",
            "#FFFFFF",
            "#363136"
          ],
          "type": "string"
        },
        "label": {
          "default": "generic.titlecase.store",
          "type": "string"
        },
        "text_color": {
          "default": "#363136",
          "enum": [
            "#F47521",
            "#EF4323",
            "#213944",
            "#2ABDBB",
            "#FAB818",
            "#73253E",
            "#EDEDED",
            "#59595B",
            "#000000",
            "#FFFFFF",
            "#363136"
          ],
          "type": "string"
        },
        "text_hover_color": {
          "default": "#FAB818",
          "enum": [
            "#F47521",
            "#EF4323",
            "#213944",
            "#2ABDBB",
            "#FAB818",
            "#73253E",
            "#EDEDED",
            "#59595B",
            "#000000",
            "#FFFFFF",
            "#363136"
          ],
          "type": "string"
        }
      },
      "title": "Store button customization",
      "type": "object"
    }
  }
}

react-jsonschema-form allows us to generate the following form:

Relying on JSON Schema allows us to isolate and limit the dependency surface between Config Delta and the client application while allowing safety and ease of integration. Tying experimentation to code configuration values against configurable component based application architecture enables generalized control of enabling, disabling and configuring application features beyond what was originally identified as a target for experimentation, providing additional flexibility.

We have deployed this microservice and have some early usage. High performance is a primary concern for us in this particular instance, as the delta config retrieval directly affects the initialization time of our client applications. We managed to keep our response times below 5ms over 10k requests per minute by using Golang and DynamoDB and being mindful of minimizing necessary database operations in the request hot path. We project the request volume to grow to 100k requests per minute maintaining our current performance.

Stay tuned for our next installment: Anatomy of an experiment

If you’re interested in working on projects like these on Crunchyroll please reach out.