The Coverage-Magnitude-Cost framework

An easy-to-use framework for prioritizing relevance features

By Nikhil Dandekar, Engineering Manager, Search at Foursquare

In my last post, I introduced the term relevance feature. To recap, relevance features are user-facing, data-science driven features like search engines (Google, Bing, Foursquare), home screen recommendations (Netflix, Amazon), news feeds (Facebook, LinkedIn) etc.

Let’s say you have already built and shipped such a relevance feature. Let’s take the Foursquare local search engine as an example. Your feature has been out for some time with real people using it.

Now there will exist a bunch of possible ways you can improve your feature. Here are some possibilities:

1. You can improve the existing models and sub-features that are a part of your feature. E.g. you can build a new version of your core ranking/recommendations model using more data or better signals or better algorithms etc.
2. You can add new sub-features. E.g. you can add an automatic spell correction system or you can implement fuzzy text matching in search.
3. There will also be existing relevance bugs that you will discover and need to fix. E.g. your search engine might be doing poorly when the search is a restaurant-name. Or your social news feed might be consistently demoting new stories in favor of popular ones.

The question facing the team then becomes: How do you prioritize among the various improvements you can do to your feature?

Here is a simple framework that you can use for this task.

Coverage-Magnitude-Cost

Coverage, Magnitude, and Cost are 3 high-level factors that can help you make your prioritization decisions.

Coverage

Coverage is the number of page impressions (i.e. user page-views) your change would impact. For a search engine, that’s the same as number of searches.

Here are some examples of coverage values:

• If you make a significant change to your core search engine model, you might change ranking for almost all of your search impressions. In this case your coverage is 100%.
• If you implement a decent spell-correction algorithm, the coverage might be much lower, say ~10%. That is around 10% of your search volume will be spell-corrected.

It’s often hard to come up with a coverage estimate before implementing the change, but this is where your familiarity with your data and the skill to do quick data analysis comes in handy. Given impression logs of past user searches or page-views, and a rough idea of the type of impressions you would impact, you should be able to come up with a decent coverage estimate quickly.

Magnitude

Magnitude quantifies how big your average change will be, given the page layout. For a search engine, here are some examples for magnitude values:

1. If you improve the top 2 results significantly, users will immediately notice it. It’s a high-magnitude change.
2. If you make a slight improvement to the top 2 results, some users might notice it, but some users wont. It’s a medium-magnitude change.
3. If you improve something below the fold, only those few users who actually scroll down on the page will see it. It’s a low-magnitude change.

Depending on the exact problem that you are trying to solve, you should be able to come up with a meaningful definition for magnitude.

Cost

This is the Design/Product/Engineering time for implementation. Software cost estimation is a known hard problem. There are a lot of resources and books on cost estimation that you can learn from.

Cost is often estimated using engineer-days, i.e. the number of days a single engineer would take to implement & then maintain the proposed solution.

When determining cost, you need to make sure to estimate the cost for a solution with the right level of complexity. For relevance features, it’s easy to build solutions that are too complex. It’s also easy to build solutions that aren’t complex enough, i.e. they are hacky and non-general. Some examples:

1. For a search engine, detecting navigational queries can be easily done by looking at click-through rates on individual results. If you build a machine learned model with 15 features to classify navigational queries, in most cases, you are building a solution that is too complex for the problem.
2. On the other hand, if you implement spelling corrections by simply hard-coding a small list of known spelling mistakes, your solution wont be general enough and wont extend to the large set of spelling mistakes that happen in the long tail of your search stream.

Generally, the more complex the model, the costlier it is to maintain. It helps to make everything as simple as possible, but no simpler.

Your final cost estimate should thus include implementation costs and maintenance costs for the solution with the right level of complexity.

High coverage and high magnitude tasks are the more important ones. Also, lower the cost, the quicker you can get it done.

You can thus measure the priority score of a task as

Priority Score = Coverage * Magnitude / Cost

The high-coverage, high-magnitude, low-cost tasks, if any, are your low-hanging-fruit and the ones that you should tackle before everything else. Before every sprint or release cycle, you should look at tasks in a decreasing order of priority score.

Given your actual situation, there will be lot of other factors to consider. For example:

• You will need to prioritize tasks that match the skills and interests of your team over the ones which don’t.
• You might want to focus on those improvements that differentiate you from your competitors.
• You might want to focus your product towards a new market or demographic. Then you will need to prioritize tasks that make the feature better for that target demographic, even if it’s a low coverage task across your current user-base.

Your final prioritization decision will include all of these and more.

The Coverage-Magnitude-Cost framework is a structured way of thinking about relevance features. It factors in quantified measures of user-impact (coverage and magnitude) along with the complexity and the cost of implementation. It applies to most relevance features and quite a few general engineering tasks as well. Try it out for your features!