Measuring Impact Better

Valuing what real people value OR What RCTs get wrong

Development programs in Southeast Asia have often taken the form of handouts, with organizations providing goods or services for free to communities in need. However, this approach can disempower recipients and create a culture of dependency when business owners are handed pre-designed solutions. It’s immoral or impossible to run Randomized Control Trials (RCTs), and Impact measurement becomes simple counts of users completing training. We can all cheer for the headline numbers of users completing training, but lose visibility after the program ends, and are often back in the same area starting over in the next year. These programs are well-meaning, but often short-lived, and impact measurement is difficult!

Luckily, we don’t need to create a new mechanism from scratch, because we already have a great model: the free-market economy. This market contains semi-rational actors that buy and sell goods and services while facing limitations from their constrained resources. Everyone must make difficult choices to maximize their utility because you can’t have it all. We’re all spending our resources on things that we find valuable and that make an impact on our lives in ways that we find uniquely valuable to us! So when we seek to measure impact better, we can use these same mechanisms to design better products and outcomes that lead to sustained economic development in the region!

Market-Based Impact Measurement

When we say a market-based solution, we’re simply asking users to deploy limited resources in a way that they believe provides the greatest benefit to them. When something is free, we short-circuit this mechanism.

As Dan Ariely says in his book, Predictably Irrational, the problem with free is that it disconnects the relationship between the benefit and the cost. When something is freely given away, people do not attach the same value to it as they would if they had to pay for it. This can lead to a lack of appreciation for the service, and can also create a culture of dependency. Ariely suggests that people pay for something, even if only a small amount, as they are more likely to value it and take it seriously. This is why market-based solutions, even if subsidized with grants, can be more effective than providing solutions for free.

In practice, these resources can be money, or it could be alternative currencies like time, effort, or data. Anything that represents a “cost” to the user can be utilize this market-based mechanism for impact measurement and avoid the problems of free that Ariely mentions.

Said more directly: if the user finds value in the solution, they will “purchase” it. If they don’t, then they’ll pass. A “purchase” represents impact delivered.

So when we measure impact in this way, we only need two ingredients:

• Constrained Resource: Money, Time, Data, etc to avoid the problem of free
• Market: A place to freely choose how to spend that resource. This is the product we’re delivering to users that they can choose to use or not.

Not only does this give us a simple, yet powerful tool to measure impact, but we give people the honor of choice and agency in their decisions rather than predetermining what’s best for them. When people pay for a solution, even if subsidized, they have a greater sense of ownership and investment in its success.

Case Study at ONOW

ONOW has been leading the way in the shift from freely-delivered, grant-funded projects to these market-based solutions that provide business incubation support for businesses in Myanmar and Thailand. We noticed that it became far too easy to celebrate the headline numbers of Participants Completing Coaching/Training. This measured initial content delivered and gave us a nice jolt of feel-good-adrenaline, but failed to really assess long-term impact for our target audience and their changing needs over the next months and years ahead. So we began to explore how we could use these market-based solutions to improve our solutions to business owners.

Most of our target audience represent users with extremely low incomes and micro-businesses with limited ability to pay for solutions. So instead of using money as our currency, we’re using time. In a free-to-use product with no start or end, we can instead proxy the impact generated by how many people choose to come back multiple times, and choose to enter data that helps them track their business outcomes and goals. The metrics below represent a small sample of our impact measurement criteria, but they have been incredibly influential in motivating our product design and delivery for users. Our hope in sharing these is that it can help others to learn and incorporate these practices for yourself!

Impact Measurement Metrics:

• Count of Repeat Users: Our latest product is designed to engage users multiple times rather than a one-and-done content delivery. If users choose to come back, we can deduce that they are finding value in the service. Since the product requires that a user “spend” their valuable time, repeat users represent someone finding sufficient value that justifies the spending of that resource. Over time, this can continue to be tracked if the service were converted to a paid and/or subsidized resource that requires users to spend their money as well.
• User Activation Rate: [Count of Repeat Users] / [Count of Total Users]. We seek to maximize this number as a measure of the efficiency of impact delivery per user. A good target number is 40%.
• Percentage of Shares/Referrals to other people: Not only does this reduce our marketing cost, but studies show that it’s far more effective to find users that are passionate about your product and wanting to share with others than trying to do that work yourself.
• User Churn Rate: By measuring the lapses of previously engaged users, we can begin to identify when, where, and how users are losing value from the product. Did they simply outgrow the need for it? Or could we change our offering to better meet their needs? This is an extremely useful feedback mechanism that forces us to reassess our product and improve it.

Conclusion

One of the key benefits of using market-based solutions is that they allow us to understand the true value of a solution. When people pay for a solution, they are making a conscious choice to invest in it, which indicates that they believe it has value. In contrast, when solutions are provided for free, it can be difficult to determine whether they are truly making a difference in people’s lives or simply being used because they are free.

Impact measurement is a well-intentioned, but often ill-defined metric that focuses on RCTs as the gold-standard for impact delivered. However, this approach still assumes a single definition for impact, and fails to account for the multiple dimensions and complexities for what real people find valuable in the world. Some business owners may desire increased profits. Some increased business stability. And others may want to learn how to increase their product offering. A one-size-fits-all impact measurement fails to capture these varying metrics.

When we shift our focus from a pre-defined definition of impact, to a model that looks at constrained choices of users that are seeking to maximize their utility with limited resources, we begin to create and deliver more effective and impactful programs that truly support and encourage economic and social development of the region.