How Fidelity measures content

2 innovative metrics and a powerful learning engine are helping Fidelity measure whether our content is inspiring better financial futures

Fidelity has a long history of providing authoritative, credible content on financial topics. But how do we know whether our content is improving our users’ financial standing?

For years, Content Strategists adeptly used diagnostic metrics (page views, scroll reach, bounce rates, etc.) to optimize our content. We analyzed these metrics, developed a hypothesis, and then validated that hypothesis through A/B testing.

And while we saw improvements, until recently, we didn’t have a framework to measure what financial-related actions our users were taking after reading our content.

From engagement to outcomes: Evolving our measurement framework

So, in partnership with Fidelity’s Customer Knowledge and Strategic Insights group, our Content Management Tribe fast-tracked an experiment.

We wanted to create a measurement system that would analyze the user’s interactions with Fidelity after reading a piece of content, beyond just diagnostic engagement metrics. By doing so, we hoped to better understand whether our content was helping people take appropriate actions to improve their financial standing.

We believed we could tie content consumption to intended outcomes by leveraging the Jobs to Be Done (JTBD) framework. We assign a primary job to each piece of content. Then, after a user engages with that piece of content, we monitor their interactions with Fidelity throughout a fixed time period. Depending on the job, we measure within the same session, within 7 days, or within 30 days.

By supporting this system with strong analytical backend platforms, we believed we could empower content creators with a dynamic, nuanced view of whether our content was driving desired outcomes to improve our customers’ financial standing.

We call this new methodology and framework the Content Learning Tool.

Creating new metrics to measure success

The Content Learning Tool can currently track approximately 70 different jobs that fall into 3 broad categories:

• Solution-driving actions include transactional and financial-related actions, including opening an account, executing a trade, or hiring Fidelity as a financial advisor.
• Engagement-driving actions are leading indicators and include signing an options agreement, completing research, or completing a Fidelity.com profile.
• Service-related actions are the customer-service related tasks such as updating beneficiaries or a profile.

To track the completion of these jobs, we developed an innovative new data point and also leveraged existing data:

1. Engagement-to-action (ETA) ratio

What would be the holy grail of measurement? Combining users’ online content interactions with Fidelity’s business key performance indicators (KPIs). Enter the engagement-to-action ratio, a new metric created for the Content Learning Tool.

The ETA ratio is a fancy phrase for a simple statistic: Of the total number of users who engaged with a piece of content, how many of them went on to complete the job assigned to the piece of content that would improve their financial standing?

The ETA ratio allows us to compare similar pieces of content that are all designed to execute the same JTBD. By judging them against a common standard, we can see which piece is most effective in helping users reach an intended goal. Higher ETA ratios indicate pieces that helped answer user questions and prompted them to take action.

ETA ratios are one of the primary KPIs of the Content Management Tribe. We’ve been chasing this holy grail for a while, but with the Content Learning Tool, we’re finally able to define, measure, and track this value over time. By delving deeper and validating hypotheses through A/B testing, we can continue to improve our content for users to make sure it is helping them improve their financial standing.

2. New Money Per Page View (NMPPV)

Currently, our Tribe creates content on about 100 different topics. In a capacity-constrained environment, time and resources need to be intentionally directed at the highest-impact work.

NMPPV is an existing metric. It is the attributed, gross amount of new money a user brings in over a period of 30 days after engaging with a piece of content. Because users may engage with multiple pieces of content before acting, we use a time decay methodology: More recent interactions receive a higher proportion of the new money attribution.

NMPPV helps give a quantifiable view into the specific financial topics that users are interested in and are driving business outcomes. This establishes clearer priorities for our content creators, so that we know we are directing our attention toward the right topics.

A real-life use case: Automatic withdrawals for RMDs

Here’s an example of how Content Strategists have used the Content Learning Tool to measure outcomes and inspire better financial futures for our users.

When you turn 72*, the IRS mandates that you take required minimum distributions, or RMDs, each year from traditional IRAs or employer-sponsored retirement accounts. Failure to do so will result in costly penalties.

This is a source of confusion for users. Therefore, we advise them and provide a service to set up automatic withdrawals. It gives users peace of mind, and, most importantly, stops the government from unnecessarily eating into their hard-earned savings.

Despite the enormous benefits of automatic withdrawals, many of our users still have not enabled this feature. So to drive adoption, we’ve written articles that help folks understand its key benefits. Prior to the Content Learning Tool, we had no way of knowing if users indeed went on to set up automatic withdrawals after reading the article — but now we can see exactly how many users have completed this job after consuming the content.

We’re now able to draft multiple articles on this topic, experimenting with different angles to be as clear and simple as possible. The Content Learning Tool allows us to compare the ETA ratio in one view, so we can see how effective each article is in persuading users to set up automatic withdrawals to avoid unnecessary tax bite and gain peace of mind.

What’s next

The MVP of our Content Learning Tool launched in November 2018. Since then, and through multiple iterations and improvements, we’ve seen the Content Management Tribe adopt the Content Learning Tool as a critical resource to inform content decisions.

Today, we measure all articles and many other pre-login pages on Fidelity.com that provide guidance to those looking to Fidelity for help. Our new filters offer the ability to dissect results by persona, age, asset level, and entry source. This helps provide deeper insights into content effectiveness and distribution opportunities.

Looking ahead, we are working on the ability to measure education and thought leadership in our mobile apps.

By introducing an improved framework and methodology, the Content Learning Tool has helped Content Strategists definitively answer the question of whether our content is “working.”

*The change in the RMD age requirement from 70½ to 72 only applies to individuals who turn 70½ on or after January 1, 2020.

#FidelityAssociate