Measuring Feature Success
Three things to measure to learn if your feature is truly succeeding!
Building a product is part art and part science. Where you lie on that scale from science to art depends on the product manager, the organisation, the market and more. However, one thing is certain, its important to measure whatever it is you are building. Validating your hypothesis is the only way to learn from your users and grow!
As a product manager, I measure the success of any feature I ship using three — yes, just 3 — simple yardsticks. Its very easy to give in to the temptation of measuring along many more dimensions especially given how cheap it is to instrument your product, but it pays dividends to keep it simple! Occam’s razor states that among competing hypotheses, the one with the fewest assumptions should be selected ….or in other words, Keep it simple silly!
So, what are these 3 buckets?
A feature is built to be used (duh!) and therefore measuring how users are engaging with your feature is pretty important. Measurements here typically include metrics such as how many times users are using a feature, how often they are using it, when are they using it and so on.
Engagement is well understood so I won’t dive more into it but a word of caution: Its often very easy to fall prey to vanity metrics and by extension its pretty easy to pass off a feature as successful solely based on engagement numbers. The ultimate success of a feature depends on more than just engagement, it also depends on relevance which is measured by the other two buckets!
Users are the sole reason for your existence & they must be your sole obsession. Its important that you understand who they are and tailor your product to meet their needs. Measuring along the user dimension helps you to validate the fact that the user you built the feature for — the target persona — actually matches the ones using the product.
The key here is to start from your target persona and break it up into smaller individual chunks that can be measured in a meaningful way to validate your hypothesis. For example, you are a building a data saver feature for your product. The target user persona is potentially someone on a metered connection in an emerging market where connectivity is poor and expensive. Measuring this involves the following individual chunks: geography, device profile, internet speed etc. In practice however, you might be surprised by the things you learn by doing just this. Maybe, in addition to emerging markets you notice a whole chunk of users coming from parts of the world with high data speeds. That could potentially mean that your users are more conscious about data usage than you originally anticipated or that they really love tinkering their preferences across the board. Each of these will spark off newer questions and more measurements.
Answering the unknowns surfaced by measurements like these will go a long way in bolstering your understanding of who really uses your product, their circumstances and will help you truly be in sync with your users.
It seems pretty obvious to state but the best products are those where every feature blends in with the rest to offer a cohesive, seamless experience. New features can either be additive — they enable a whole new experience — or multiplicative — they enhance the existing experience significantly. Every feature that you bolster a product with has to earn its place with users.To truly understand why users are using a certain feature,it is imperative to measure the context in which in they use it. To me personally, this is the most the important dimension to consider!
Let’s consider the ability to mark a conversation as unread in a chat app. Its a pretty basic feature and is multiplicative in nature. It allows users to quickly scan any new messages in real time and flag it for their attention at a later time. It makes them less antsy about going to a group chat with 100 messages when they don’t have the time. The key here is to measure how this feature affects everything else. If a user can mark a chat unread, they are more likely to click through on a notification for new messages, dive in and out of chats to catch up and come back to your app even in situations when they can’t immediately respond. So by measuring these seemingly related behaviors, we begin to understand the how and why behind the feature.
Measuring the ripple effects of your feature really helps you understand the broader context of how it fits in with the overall experience. It helps you tie together understanding of who your users are and how they are engaging with it by tempering that with the context in which they actually use it.
Ship, Measure, Learn, Repeat!
Measurement and successes aren’t ends unto themselves! As you chug along your journey of dreaming up the latest and greatest and measuring along these three dimensions — Users, Engagement & Context — you are bound to discover new problems to solve and shinier things to build. After all, the fun is in making!
True makers are those who engage in measured insanity!