How to measure the success of your product in Emerging Markets
Dhanveer Matharu commented on a previous post with a good question — “how would you measure the success of a product local to one country, city or neighborhood — and then plan to scale / re-launch it elsewhere?”
The short answer — build a product that users love. If the following factors hold, you are ready to scale beyond your initial market
- User activity is stable — The product is running mostly organically on its own or even growing
- Business is at steady state — You are past the investment phase. Your new customer acquisition and operational costs are affordable. The day to day work from your team is routine (working crazy hours only happens once in a while)
- Product / market fit — You know what works and doesn’t work and the next steps you need to scale
The detailed answer is to set targets for when you will scale. The basic metrics are usually some combination of
- Active / repeat customers over some time period (e.g. 30 days)
- Activity of those customers (e.g. # of transactions)
- Revenue (or the equivalent)
Your targets should be ambitious but realistic. If you set expectations too high, your team probably won’t deliver which will hurt morale. For each target market, determine the realistic possible customer base for your service. Start with the total size of the market and make assumptions using comparable metrics from other markets or similar products. For example, most mobile money products report 20–30% active users. The most successful one, M-Pesa in Kenya, reports 50% active users. Thus, if you launch a new mobile money service, and you hit 20–30% active users, you are probably ready to scale.
Your sample size should be a good proxy. Your initial launch needs to reach enough customers for you to be able to draw conclusions. As a general rule, if you reach 10% or more of the total customer base then you can start to trust the metrics you have as reliable.
Another approach is to baseline yourself against competition. Are you are 1st, 2nd, 3rd in your industry or product segment? Is your organization among the leading names? If you are still a small player in your initial target market, then you aren’t ready to scale.
Good target metrics are ones that you control and directly measure. For example, when we launched a product to provide health information to users in Uganda over SMS, we tracked the daily number of SMS messages that we received. Users could ask questions about symptoms and treatments for health conditions. The mission of the service was to improve health outcomes. However, this is difficult to track. Many different factors beyond our service could influence a particular person’s health choices. We ran a research study to understand whether we were affecting health outcomes (the results were inconclusive) but even if the results had been positive it would have been unrealistic to expect to use that as a deciding factor to scale. In this case, the number of SMS messages was a more direct metric.
Make sure you understand the story behind the numbers. Are the numbers real? Dig into the data. From the SMS example above, we checked samples of the messages we received to find out what portion of them were accidental, spam or repeat messages sent from the same user. Try to collect individual success stories of customers using your business. These can be intrinsically rewarding and also help to tell your story to the press, investors and potential employees.
Be careful about metrics that are tied to incentives. Do you have a business where you reward your customers, staff or others based on the metrics? People may try to game the system to minimize their effort and maximize their reward. Look for signs of abuse in the top performers. For example, for one product we compensated vendor staff for customer signups. When we looked at some of the top performers, many of their signups were one time users. The top performers were creating fake customer signups just to hit their targets. This is human nature. You probably can’t eliminate it altogether but you can try and minimize it. Try to find the right balance of fixed and variable compensation. Incentives should motivate people to do quality work instead of cutting corners.
Don’t use vanity metrics for scaling decisions. Vanity metrics make your business sound good to the press, investors or employees but may be unrelated to the fundamentals of the business. A common example is measuring total registered customers. It sounds great to say that you have reached 1M customers. However, it’s not reliable if most of your customers are one time only.
Ensure that scaling makes sense for your overall company strategy. What is the end goal of your organization? Will scaling up the business help / hurt your effort? Scaling up might drive more revenue and usage but also increase cost and complexity. Depending on the organization, it could also bring cost savings from economies of scale. Sometimes you might get better results from investing more in existing markets than scaling to new ones.
Pilot in 2 different markets. If you fail in your first market, don’t give up. Determine whether you failed because you picked the wrong market. If so, try another one. Conversely, finding success in one market, doesn’t mean that the same business will work everywhere. You can scale after your formula works in 2 distinct markets. Do not pilot in multiple markets simultaneously. You may save time or get a first mover advantage but these wins are not worth the cost of dividing your attention and stretching your resources. If you must have multiple markets then at least try and make them close together. I worked on simultaneous pilots in Kenya and Philippines and the travel times and time zones were brutal!
What other advice do you have for measuring the success of a product in Emerging Markets?