Building the basis for impact measurement in frontier markets
Impact measurement is no easy task anywhere in the world, and it can be particularly challenging in developing contexts. At Alter, we use a lean approach to monitoring and evaluation system — one that emphasizes rapid learning and efficiency — in response to the challenges we have faced in measuring impact in frontier markets.
The rise of entrepreneurial support organizations has occurred in parallel with an increased pressure to measure their effectiveness. And that is a good thing. It pushes them to constantly reevaluate their value proposition and theory of change: are they turning high potential ventures into champions? If not, what needs to be changed or tuned?
It also holds them accountable to stakeholders. Funders need to know the social return of their investment in order to better allocate their money. Entrepreneurs should know the value of the services they receive. How can they know if a mentor they just met will provide effective advice if there are no performance metrics available? As we discuss below, measurement is challenging, but critical to our success at Alter.
CHALLENGES IN IMPACT MEASUREMENT
The attribution problem. This is arguably the central problem with measurement. To what extent can employment generation be attributed to Alter’s support services? To answer this, we would need to compare the situation with Alter’s engagement and the situation in which Alter is absent (the counterfactual). The problem, of course, is we cannot observe both concurrently.
Figure 1. Graphic display of the net impact of an intervention
Performing a rigorous impact evaluation — such as a randomized controlled trial — would require Alter to randomize its support to a pool of entrepreneurs where each one has an equal probability of selection. That is unfeasible and, to an extent, unfair. In the geographies in which we operate, the pipeline of high growth potential ventures is too small to afford denying one of them access to our services for the sake of measurement.
A startup dilemma. As a young organization, we need to prove our effectiveness to our stakeholders — particularly to our funders and entrepreneurs. But this is also precisely the moment when getting data is more difficult, either because it is too soon for outcomes to materialize in the data or because, as we refine our services, we may also need to change the indicators we monitor.
Customized service provision. As Ozair described in a previous post, we believe that services should be adapted to the entrepreneur’s needs. Because we support a diverse group of ventures, this implies a heterogeneous service mix, with varied and sometimes uncertain outputs and outcomes. Alter may recruit talent from the diaspora to work in one of the ventures and at the same time bring a mentor to provide advice to the CEO. How much does each service contribute to the company’s outcomes and how does this compare to the effect of connecting a venture with an investor?
Data and trust in frontier markets. Despite the improvements in data collection and storage that have come with technological progress, there remain gaps in data availability in frontier markets. Measuring is costly and data-driven decision making is not common practice. Metrics that are not actionable — that is, tied to specific actions undertaken by the company –will not generate value to the entrepreneur, making them reluctant to spend resources on measurement.
Moreover, trust can be a major obstacle in contexts where law enforcement is weak or where there is a history of value capture by foreign entities. Even if entrepreneurs have the data, they are sometimes unwilling to share it. They feel that doing so could make them vulnerable to competitors and the government.
For instance, Myanmar companies, which arguably have the best data and reporting systems among the countries Alter works in, are still hesitant to completely share their metrics, especially markers of business success. To address this, we ask our companies to report ranges instead of single numbers. We work hard to develop and maintain a trusted relationship with all the entrepreneurs we work with, where they feel comfortable sharing these details about their business.
THE ALTER APPROACH TO IMPACT
These challenges have not stopped us from attempting to measure our effectiveness. We are aware that this is an ongoing process that needs to be adjusted to circumstances. Here are a few guidelines that help us with these challenges
1. Start with a solid, evidence-based theory of change. Metrics should be deduced from there. We begin by focusing on the problem we want to solve and envision the change that we want to see in society (the impact). Alter has put its attention on the lack of opportunities for people to thrive in frontier markets, particularly, insufficient jobs and lack of basic services. By supporting high potential entrepreneurs, we expect to make frontier markets more dynamic, so that entrepreneurs can create better products and services, and provide better employment opportunities. Our outcomes, therefore, are jobs provided, customers served and vertical linkages created.
To get there, we need to take several steps. A good theory of change establishes causal links between those steps, backed by evidence of comparable interventions. Before we reach our intended impact, that evidence can make us confident that we are on our way. And even if measuring the impact is not feasible, the evidence backs up the claims that the impact is likely. Sound theories of change can fill some gaps in the data, especially at early stages.
2. Leverage the power of qualitative data. Entrepreneurial support organizations should constantly talk to entrepreneurs, employees and end users to understand the value that their product or service is generating (or not), and why. The why is important, as qualitative data tends to be more nuanced and insightful than only quantitative data.
We can infer a venture is adding value if the number of clients is constantly increasing. But to understand what about the product or service clients value the most, we should probably ask them. As an example, companies in Myanmar have consistently been receptive to Alter’s diaspora meetups since this presents an opportunity to find exceptional full-time talent. Quantifying the impact of a single diaspora meetup is challenging, but we know they are valuable when companies have always expressed their feedback to organize such events.
3. Build consensus around metrics that matter. Stakeholder involvement is key to identifying and understanding the most important outcomes. Meaningful metrics are those that contribute to the decision-making of stakeholders. While Kreyol Essence in Haiti cares about the number of farmers in its supply chain, Impact Terra cares about farmers as consumers. The two companies both care about farmer livelihoods, but impact them in different ways.
We must also account for the fact that the ventures can be highly heterogeneous in terms of geography, industry or stage of development. Indicators to measure their progress should not all be the same. As I’m writing this post, Orlando is working with ventures in Cuba to set specific targets and milestones against which performance can be monitored and success be evaluated. Having specific and agreed upon metrics ensures that both Alter and entrepreneurs’ efforts are aligned and can be held accountable.
4. Develop a culture of measurement. Our success depends on our ventures’ success. Without consistent data , it will be hard to get any conclusions about the value we create. But measurement is also valuable for the entrepreneurs themselves. As suggested by ANDE, training on proper calculation of metrics and completion of surveys is a valuable part of the support given to ventures. By doing so, entrepreneurs will also be more prepared to prove their own value to potential partners.
Furthermore, we need to be as transparent as possible with regards to how we use the data and who can access it. Even so, building trust will probably take time. To accelerate the process, we consistently focus on listening to the feedback that entrepreneurs give us regarding the services we provide. It is only with that baseline of trust that we can ask them to share their data.
With increasing ICT penetration in frontier markets and new ways of measuring impact being developed and tested (for instance, lean data, SROI, BACO, and efficient impact frontier, among others), the future looks brighter for impact measurement. Alter is committed to that goal. Looking forward, we have two long-term goals: first, we will lead the efforts to standardize metrics and collection methods across the entrepreneurial support organizations, so that their impact can be comparable and resources can be better allocated. Secondly, we will remain focused on metrics that matter to our entrepreneurs — those that allow them to take corrective actions when necessary, and those that announce that major goals have been met and that it’s time to celebrate.
