How Y-Combinator Grad New Story Quantifies Impact Measurement as a Startup

By the winter of 2016, the founding team behind New Story Charity had accomplished a lot. In their first two years of operation, they had built a crowdfunding platform to construct homes for families in some of the world’s most vulnerable communities. They had figured out a model where — for about $6,000 — they could work with partner organizations to construct safe and clean houses. They had successfully funded the construction of 151 homes for an entire community in Leveque, Haiti, providing shelter to over 755 people who had been living in tents since the 2010 earthquake. They had graduated from Y Combinator, the Silicon Valley tech incubator, where they had learned to become laser-focused on growth. And they had built an organizational culture that valued transparency and trust. A key part of this model is working with partner organizations in the field and providing every donor with video-proof of the families they funded moving into their new home.

In early 2016, New Story had strong anecdotal evidence of the impact of housing from beneficiaries like Nancy, who had started a small charcoal business outside of her home to help her earn extra income. But, as New Story’s Global Impact Manager, Emma Lalley put it: “We had a palpable and potent sense of the impact our houses were having on people’s lives, but we had not yet systematized evidence collection in a quantitative way.” Although they wanted to collect more rigorous impact data, their team was still small and resource-constrained. Trying to assess impact using methods like randomized control trials or formal surveys seemed like it would be overly ambitious and unfeasible for a young organization whose limited resources were focused on home quality, funding and building their brand.

Then Alexandria Lafci, one of New Story’s co-founders and their Chief Operating Officer, researched Lean Data, an approach to impact evaluation that Acumen was piloting with social enterprises, and her mindset began to shift. When she realized that it was possible to collect data in low-cost, customer-centered ways using mobile phones, she began to think that New Story might be able to gather impact data earlier than anticipated. This spring, she hired Emma to help her architect New Story’s approach to impact evaluation.

THE LEAN DATA MINDSET SHIFT

Emma quickly signed up for Acumen Academy’s Lean Data course. “We used the course to develop a framework to better understand our value proposition, indicators, and the types of data we need to collect,” she said. When she completed the first assignment of the course — the Customer Promise worksheet — it helped her formalize her framework for impact. “By mapping out our key assumptions or ‘promises’ we were able to dive deeper into the impact we believe a home creates. Our impact data is broken down into three key indicators: health, education and economic opportunity,” she explained. “The course helped us understand how to map the indicators and explore what metrics can be used to construct mobile surveys that the team could implement to collect impact data on the ground in Haiti.”

“Lean data shifts the conversation from monitoring and evaluation to listening and learning.”

“The course shifts the conversation from monitoring and evaluation to listening and learning and iterating. When we took the course, we realized quickly that lean data augments our efforts to put beneficiaries first.” she said. Emma came to realize that impact evaluation is not rocket science and lean data methodology allowed her to prioritize data collection in a way that was feasible and unobtrusive. “This is a big part of why we work with partner organizations on the ground. Because they are part of the community, they understand the community. Lean data allowed us to amplify beneficiary voices in a data-driven way” she said.

USING LEAN DATA ON THE GROUND IN HAITI

One of the key things that the Lean Data course emphasizes is asking customers about how a product or service has been meaningful to their lives. When it’s easy to assign a value to products in the private sector based on price, social entrepreneurs must create new frameworks to assess value. Acumen’s Impact team had learned that instead of assuming the ways in which a product or service is beneficial and imposing those answer choices upon customers, you can gain richer insights if you first ask them about why the new product matters to them. In other words, why is it meaningful to their lives? Accordingly, when Emma landed on the ground in Haiti, she used this emphasis on “meaningfulness” as a starting point and asked people to simply “tell me about your home.”

“This question became a great opener for us,” she said, “It led us to get tours of family’s homes. We were able to see the outdoor kitchens people had created with their leftover tent materials and see other additions families had already made to their homes. One simple question worded correctly allowed us to create a dialogue.” Using this question, she quickly came to understand two of the major ways that a new house could be meaningful to new owners — it could provide a sense of safety and security and it could give them a sense of ownership. So she designed three questions to get at these dimensions:

• Have you purchased anything for your house?
• Have you made any physical changes to your home?
• Do you feel safe walking in the community at night?

She found that these three questions led people to tell her stories about how they had grown gardens in front of their homes or created beautiful walkways made of concrete and seashells that evinced a real sense of ownership, pride, and community. By testing some of the questions she had drafted in the Lean Data course with real customers, she was able to use their responses and real conversations to refine the types of questions and answer choices that she programmed on the mobile survey.

COMBATTING THE CLIPBOARD CULTURE

After updating her questions, the final piece of the Lean Data approach was to roll out the survey using mobile tools. But she was nervous about how New Story’s partner organizations — the ones who would actually carry out implementation of the baseline and impact surveys — would respond to having to use new mobile tools.

Because they work with partner organizations that have been operating in communities for decades, New Story wants to design programs or processes that will not add significant weight to their existing scope of work. “Our partners are extraordinary. The last thing I wanted to do was layer in a cumbersome tool our team had to master,” said Emma. However, when she asked her partner organizations if they’d be willing test a mobile survey to ease data collection efforts she was pleasantly surprised when they said, “Yes.”

“When we designed the survey and selected our mobile platform, our goal was to do so as leanly and unobtrusively as possible,” she said, “it really helped the conversations to get started.” Mobile phones are so prevalent in Haitian communities that no one was perturbed by seeing people typing on their phones. In contrast, Emma quickly learned, Haitians had developed a particular distrust of clipboards. They had been approached so many times by various researchers, aid organizations and nonprofits in the aftermath of the earthquake and had been asked a standardized set of questions about personal details that they were reluctant to have conversations with people who came bearing clipboards. When people were collecting data on a mobile phone, however, in a way that felt respectful of their time and in sync with the norms of the community, they seemed much more comfortable opening up. “There were multiple times while surveying on the ground that people from our partner organization said, ‘These mobile phones are a game-changer. This makes things so much easier,’” Emma said.

MOVING THE LEAN DATA APPROACH FORWARD

There is still much work to do to move New Story Charity’s work on impact evaluation forward. Emma will be iterating on survey questions and implementing the lean data approach in El Salvador and Bolivia where New Story has plans to construct over 300 homes in three communities.

More local enumerators in Haiti will be trained so that New Story can get a regular pulse on the impact they’re having as they continue to build. Emma wants to learn more on how to accurately calculate sample size and inject more rigor into her data collection process. But, in the last five months, there has been rapid headway in assessing the impact that New Story is having on the families they are aiming to help.

“The balance between pragmatism and academic rigor is where Lean Data offers the sector tremendous value.”

“Gathering comprehensive impact data may seem insurmountable with a traditional monitoring and evaluation mindset, especially if you are an early stage company,” Emma reflects, “But by taking the [Acumen Academy] course and learning about Lean Data methods, we learned that data collection can be feasible and actionable. It can also be a humanized process instead of just reading standardized questions from a script. The balance between pragmatism and academic rigor is where Lean Data offers the sector tremendous value.”