Global metrics, Local metrics
Experiment analysis always takes longer than you anticipate.
“Wait, why did that metric move?? And how did it move independently from this other one? I guess we’ll look more into that. Oy.”
But amidst all this nuance, there’s one beautiful and definitive divide you can rely on: Global v. Local metrics.
Every feature will have Local metrics (did users click the new button, play with the new widget, etc) and every company will have Global Metrics (time spent in app, number of days users return, etc).
Local Metrics do three valuable, specific things:
1. Make sure the feature is on (e.g. no major bugs)
2. Teach you about how people use the product (for ongoing user insight)
3. Light the path toward the next iteration (if any)
Global Metrics tell you:
1. If this feature is actually valuable
👉 You should watch and learn from Local metrics, but features should live and die by the Global metrics
Local metrics are easy to move—give someone a bigger button and they’ll push it more. You can game it, you can twist it.
Beware: if you’re org is aligned to ship locally high-performing features, your savvy PMs will build the brightest, blinkiest features they can devise.
Global metrics, if designed correctly, will be really hard to move. Even small gains are rare, coveted and prized.
A definitive 0.5% increase in a solid global metic is cause to rejoice! 🍾🍾🍾
I see a lot of PM candidates get hung up on this: “My hypothetical feature is a success if people hit the button!” they say, “50% increase in button clicks! Ship!!!”
But what if all your button clicks are up but your retention metics are down? What do you do then?
There’s still lots to be learned from High Local, Low Global tests. But in most cases you want to do that learning by rooting through the results after you’ve turned off the experiment.
It can feel really uncomfortable… as a designer, PM or developer you put a lot of effort and love into building that feature, you’re inherently emotionally invested in it doing well.
But as a PM, sometimes you gotta cut down your particular tree and keep your eye on growing the whole forrest.
Got other ways of discussing metrics at your org? I’d love to hear! Respond below and let me know—I’m always expanding my understanding of this area.
