Lean Data Learnings
Constituent Voices
At Omidyar Network, we start from a fundamental belief: People are inherently good and capable, but they often lack opportunity. We believe if we invest in people, through opportunity, they will create positive returns for themselves, their families, and the world at large.
But too often the voices of those at the far end of our interventions — the people we hope to empower — are not heard by the actors driving capital, policy, and resources for their benefit. Conversations center instead around entrepreneurs, capital markets, cost-benefit, or other top-down considerations.
We believe it’s essential to listen directly to the perspectives of the people we are working to serve. This series will share insights from those who engage with our own portfolio companies and individuals more broadly. The goal is to help ground the activities of investors, philanthropists, and social change actors in the views of the actual people whom we all aim to empower, and to generate dialogue that can uncover changing trends to drive more effective outcomes.
In this second issue, Lean Data Learnings, we present the findings from our global survey of 11,500 customers and constituents of 36 of our investees to understand how the people they serve feel about the products and services we are funding.
Click here for Issue 1: Trust and Privacy
Lean Data Learnings
Omidyar Network commissioned Acumen’s Lean Data team to survey customers across 36 investees to understand how well those companies are serving their customers. The research finds that customers assign an average Net Promoter Score of 42 to our participating portfolio companies, albeit with wide variation. Seventy-four percent of those customers say quality of life has improved because of these companies. This research has significantly deepened both Omidyar Network’s and our investees’ understanding of how the customers they serve view them.
How often, after completing a purchase online, do you see a survey pop up: “On a scale of 1–10, how likely are you to recommend us to a friend?” The goal of this question is to generate a Net Promoter Score (NPS), which is a measure of customer satisfaction. Developed in the early 2000s, it has become ubiquitous for for-profit companies. However, startups and social sector organizations do not always have the capacity to engage in this kind of dialogue with their users.¹
Over the last year, Omidyar Network has partnered with Acumen Lean Data to drive better outcomes for our portfolio through consumer insights. In the second half of 2017, we completed our first ever “Lean Data Sprint” where we surveyed over 11,500 customers of 36 of our investees across 18 countries for their opinions about the products and services being provided. For many of our investees, this was the first time they had ever systematically surveyed their customers. For Omidyar Network, this was the first time we asked standard questions and obtained comparable answers to: i) how the people we serve feel about the products/services we are funding, ii) how much the product or service has improved their lives (if at all), and iii) from which income bracket these customers come.
While the research validated some of what company management and Omidyar Network’s investment managers already knew, it also brought new insights for both teams. Below we share the most meaningful insights from quantitative data and qualitative feedback.
Quantitative data snapshots
The Lean Data analysis has added a quantitative representation of customer experience to our understanding of how well our portfolio companies and organizations are serving their target beneficiaries. The data shown below includes metrics of Net Promoter Scores, effect on quality of life, and inclusivity. We note, of course, that there are clearly limitations with such customer data, such as the “snapshot in time” of a survey, and the fact that answers are likely relative to expectations — if a customer had low expectations from the outset, it’s easier for the company to outperform on customer surveys, or vice versa. Nonetheless, given the rapid approach, we and the participating portfolio companies have found this data to be a resource-efficient addition to the data currently available for portfolio management. Given the standard questions applied across a diversified slice of the Omidyar Network portfolio, we also find value in using this analysis within our learning strategy more broadly.
Net Promoter Score (NPS)
The average Net Promoter Score for the surveyed companies was 42, and there was wide variation: scores ranged from -18 to 90, with more than a third of the sample scoring above 50. Users of this metric tend to consider scores between 0 and 50 as “fair to good,” and scores above 50 as “excellent.” For comparison, Apple, Amazon and Netflix have Net Promoter Scores of 72, 69, and 68, respectively. Those giving high scores often cited integrity and transparency as the rationale. The drivers of low scores tended to be unresolved complaints and insufficient or ineffective communication.
Quality of Life Improvement
On average, 74% of respondents say quality of life improved; 39% say “very much” As a complementary data point to NPS, we asked the simple question: Has your quality of life improved because of [company]? On average across the portfolio of companies participating in the sprint, three-quarters of customers reported positive impact generally, with 39% of those customers indicating that companies had “very much improved” their lives. Figure 2 shows the data for each company with a cumulative representation of those who indicated quality of life was “slightly” or “very much” improved. In qualitative comments, the changes cited ranged from better spending habits for a personal finance product, to an improved relationship between parent and child for an education provider.
Income Distribution
On average, 50% of customers live on less than $6/day (2011 PPP)². Part of our ambition as an impact investor is to deliver better, more affordable products and services to low-income populations who often pay a “poverty premium” for living and trading in informal economies. With the Lean Data Sprint, we were able to collect data across the portfolio to provide a snapshot of the income levels of our investees’ customers.
Figure 3 shows the income distribution of customers for each company participating in this part of the survey. Naturally, there is a variety of profiles — some companies at the top of the chart are reaching predominantly low-income customers, while others at the bottom of the chart are predominantly serving higher-income customers. Those with a steeper-sloped profile are reaching mixed income brackets, while flatter profiles are more consistently targeting one bracket.
It is important to note that we often find that impact businesses serve a diversified income-level constituency, and some of these profiles reflect just this. In fact, we have been building a research base to verify that multi-income models are effective. A recent report, Reaching Deep in Low-Income Markets, finds that serving populations at somewhat higher-income levels does not seem to prevent organizations from also reaching much lower-income levels. In fact, the prevalence of these cross-income models may indicate that this characteristic is key to financial sustainability. With the income data collected through this survey, we can begin to test these hypotheses over time.
Inclusivity
Figure 4 shows some of the country-level inclusion data, comparing two countries where we have a large enough sample to retain anonymity — South Africa and India. The solid line shows the national income distribution, and the dashed lines each represent one company’s customer income distribution, but with more granularity on income brackets. The fact that most of the dashed lines representing companies’ customer income are below the country lines means that the income distributions of our participating investees’ customers are skewed toward higher-income groups relative to the national distribution. While we expected to find most companies serving mid-income alongside lower-income customers (as per the research referenced to the left), it’s been helpful to see the degree of that skew across companies and countries, and identify what stands out for further investigation.
Examples of Qualitative Insights by Sector
Pairing our quantitative findings with qualitative insights brings more richness to the picture, particularly by sector. By asking about customer experience in an open-ended way, we can begin to build a picture of what is important and most noticeable from the customer perspective. Below, we share some of the initial feedback that customers provided, with the caveat that what we share here is just a snapshot of feedback for flavor rather than conclusive findings. The company-specific qualitative feedback is very rich, and we have distilled only a selection of sector-level consistent feedback to represent anonymously the type of content that came up in the survey.
We believe listening to constituents is critical in delivering positive outcomes for the beneficiaries we are working to serve.
These insights are just a taste of what was shared: actionable and specific insights from customers to help guide company management and Omidyar Network’s investment teams to best support the companies in better serving those customers over time.
The Value of Real Feedback
It is often a challenge for impact investors to develop efficient data practices to support impact measurement within their portfolios. We’ve found that Lean Data enables the collection of consumer feedback in a resource-light way. The practice has given us valuable insights into how to best support our portfolio companies’ financial and social outcome performance, and we’re pleased to learn that other funders are also using the tool. For instance, the UK’s Department for International Development has applied this methodology to understand customer views on a poultry feed production plant in Ghana, providing critical feedback for themselves and for the plant.
The Lean Data Sprint is just one method we’ve employed recently to better understand the impact of our portfolio companies on individuals around the world. This survey of over 11,500 people sits alongside in-depth, single company Lean Data analyses we commissioned last year, as well as other data collected for portfolio management, monitoring and evaluation. We also conduct non-company specific research on constituent views through direct surveys, interviews, and deeper ethnographic-style research. Research such as the above mentioned Reaching Deep in Low-Income Markets also informs our hypotheses about what might achieve positive impact.
Just as consumer testing is a key element in product development, we believe listening to constituents is critical in delivering positive outcomes for the beneficiaries we are all working to serve, and this is one tool we are using to help amplify their voices in our work.
Acknowledgements
We would like to thank Kasia Stochniol and Tom Adams for their dedication, diligence, and creativity in leading Lean Data projects for Omidyar Network; and members of the Lean Data team (Prashant Maheshwary, Sonia Kuguru, Jessica Martin, Ashley Speyer, Ushnisha Ghosh) for their help.
Most of all, we would like to thank Omidyar Network investees who were open to experimenting with a new tool in order to serve their users better.
[1] We use the terms “customer”, “consumer”, “constituent,” “user,” and “beneficiary” interchangeably throughout this issue. All terms refer to the population that our portfolio of for-profit and non-profit organizations serve through their work.
[2] International poverty levels are measured using Purchasing Power Parity (PPP). PPP is an economic theory that compares different countries’ currencies through a market “basket of goods” approach. According to this concept two currencies are at par when a market basket of goods (taking into account the exchange rate) is priced the same in both countries.
[3] The Poverty Probability Index® is an easy-to-use survey tool that uses asset and household indicators — like household size, or what the roof is made of — to estimate the likelihood that a respondent is poor or low-income.
