North Star or sign post metrics: which should one optimize?

Carl Anderson
Nov 25, 2018 · 9 min read

North Star versus Sign Posts

Examples of North Star versus Sign Post metrics for various business models. For instance, in hotels it would be easy to game the sign post metric and drive bookings through heavy discounting. To drive revenue, however, is much harder, and requires driving bookings but also customer satisfaction, price, customer loyalty, and more.

Implications of North Star Metrics as A/B test KPIs

Handling commitments longer than desired A/B test period

Testing for difference in cancellation rate between treatment and control in a test that is shorter than some of the commitments.

Implications of Sign Post Metrics

i) Perverse incentives

Purchase, a north star metric, has clear and honest success and fail signals. Add to cart, a sign post metric, has an asymmetry: an increase does not necessarily it is good for bottom line.

ii) A Possible Solution

In a visitor site where the visitor team can only directly control the number (and quality) of visitors, they can be held accountable, not just to a sign post metric of visitor volume but to a downstream and north star metric such as conversion, if you control for the A/B tests by comparing the red (AA) versus blue (BA) path.

iii) Optimize for wrong thing

iv) Myopia

Carl Anderson

Written by

Director of Data Science, WW (formerly Weight Watchers), NYC. Author of "Creating a Data-Driven Organization" (2015, O'Reilly). Web: carlanderson.ai

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade