Decoding Good Metrics
Navigating Proxies with Sensitivity and Specificity
In the fast-paced, data-rich environment of today’s businesses, metrics serve as more than mere numbers. They guide strategy, reflect the health of our initiatives, and often, mark our success or failure. However, as many experienced leaders acknowledge, we can’t always measure precisely what we intend to. This leads us to rely on proxy metrics — indicators that may not be perfect but closely reflect the desired outcomes. This brings us to a crucial question: Are we looking at the right numbers? And are they good enough to guide us?
To answer these questions, let’s talk about two basic ideas: sensitivity and specificity. At the crossroads of these two, you’ll find a metric that not only alerts you of a signal (an event or outcome) but also ensures that this signal is not a misleading noise.
Sensitivity and Specificity in Action
Consider Skooldio, an online learning platform, as an example. Their primary goal is to ensure a high level of customer satisfaction for their learners. But how do you measure satisfaction? Skooldio needs a metric or set of metrics that can help them approximate user satisfaction.
Sensitivity is about catching real positive changes. It gauges how well a metric can correctly identify a positive change or signal when it truly exists. For Skooldio, this means recognizing a satisfied user.
Specificity, on the other hand, is about avoiding mistakes. It assesses how well a metric can correctly reject a false signal or show no change when there really is none. For Skooldio, specificity ensures that the metric does not mistake a dissatisfied user for a satisfied user.
Now, consider Skooldio’s options:
Metric 1: Net Promoter Score (NPS)
For Skooldio, a natural starting point might be the Net Promoter Score, a staple in many businesses’ metric toolkits, gauging the likelihood of customers recommending a service like Skooldio to their peers. In terms of specificity, NPS performs well: a particularly low score typically signals dissatisfaction. However, its sensitivity can sometimes be a point of contention. Even if the service quality or satisfaction level improves, it might not always translate to a higher NPS. Some users might feel the service has improved but not enough to shift them into a higher NPS bracket. Plus, not everyone responds to NPS surveys, and some might be influenced by recent experiences, which can be misleading.
Metric 2: Course Completion Rates
This metric exhibits a high sensitivity — if users are satisfied, they will finish their courses, right? Mostly, yes. However, its specificity is somewhat moderate. Some learners might complete courses but remain unsatisfied, perhaps due to the content quality or presentation style.
Metric 3: Enrollment in Advanced Courses
Taking a deeper look into user loyalty, Skooldio might consider the number of users enrolling in subsequent, more advanced courses after their initial completion. It’s a strong specificity metric. Users enrolling in higher-level courses often signal a deeper satisfaction and trust in Skooldio’s offerings. However, the sensitivity is moderate. After all, not all satisfied users might have the desire or need to take a more advanced course. So, while this number is excellent at showing trust, it might not catch all the happy users.
Metric 4: Customer Complaints
Direct feedback, even when negative, can be a goldmine of insights. Complaints are a clear sign of dissatisfaction, showcasing high specificity. However, the sensitivity of this metric is lower. Not everyone speaks up—some might think it’s too much trouble or that it won’t change anything. So, while complaints clearly show dissatisfaction, they might not show everyone who’s dissatisfied.
In Conclusion
Skooldio’s metrics journey reflects the broader challenges faced by businesses. It’s not just about choosing metrics; It’s about truly understanding them, grasping the nuances of sensitivity and specificity, and ensuring even indirect proxies move us nearer to the truth. In a world full of data, this careful selection is what separates meaningful insights from mere noise, allowing businesses to truly align with their customers.
