Collect Data that Can Drive Action

How do I collect just enough of the right kind of data to power scale?

Given the ubiquity of ever cheaper data collection methods, there is a tendency for organizations to collect, and funders to require, more and more data. However, data is most powerful when it is able to support decisions or actions that drive scale. Too much data or the wrong kind of data can stifle a leader’s abilities to make decisions — or worse, lead to poor decisions because the data collected drives the wrong kinds of actions as you scale.

“You are usually only really addressing the top one to three pain points. Take care of the hearts and lungs, and you can live with an achy knee. There are always trade-offs.” — VisionSpring and EYElliance Co-Founder Jordan Kassalow

IN ACTION: Harambee, always asking “how will this data serve us?”

Harambee’s leadership starts every Monday with a weekly stand-up call during which senior leadership reviews a dashboard of the organization’s key performance indicators (KPIs). The call is an opportunity to have a conversation about where the organization is today, where it is heading, and what priorities are on the radar. In order to ensure these calls drive the right kind of action, Harambee knows it must be reviewing the right KPIs that keep it laser-focused on the youth it is serving. When deciding which KPIs are important to collect, Harambee considers two things — the first is the number of KPIs it is collecting and the second is the alignment of these KPIs with its mission. For Harambee, the magic number of KPIs to collect is under 10. Having a small number of KPIs allows it to focus on these data points, assign ownership, track the KPIs regularly through an organization-wide dashboard, and use those KPIs to drive further inquiry and ultimately important strategic decisions. It has the discipline required to move from collecting data that is “nice to know” to collecting data that “will drive action.” Harambee asks itself three questions when it chooses to collect new data:

1. Why are we tracking that?
2. What behavior will it drive internally?
3. Does it keep the young person (i.e., Harambee’s client) at the center?

With these three questions, Harambee is able to prioritize and clarify what is useful to collect versus what is more likely to sit on a shelf.

Top Tips in collecting data to drive action

1. Identify data points that will help manage unique tensions of scale.
   While each organization has unique data points based on its own theory of change, it is also important to track the critical drivers of scaling efficiency: how quality of impact and cost per unit change as the work scales. Emily Bancroft, President of VillageReach, often speaks about the tension between impact, financial sustainability, and scale, and how organizations must often make trade-offs within this “three-legged stool.” She says that, with enough money, an organization can increase its reach and achieve high quality impact — but it is challenging to maintain that level of financial input over time. Or, it could focus on tightening costs while broadening reach and allow for some dips in impact outcomes — a decision that sometimes makes sense for the short-term while scaling. For the purposes of data, then, it is clear that organizations must closely monitor the balance between costs, reach, and quality of impact — and put in place trip wires (i.e., data that can serve as early warning signals) to determine when the desired balance is off track.
2. Recognize that data that drives scale is both quantitative and qualitative.
   There is a natural tendency to provide more weight to quantitative data gathered through the scaling process. But interviewees stressed repeatedly that qualitative data was just as important and helped them ensure that the context of quantitative data was well understood and that client experiences were centered in an equity-driven path to scale. For those reasons, it is essential to balance both quantitative and qualitative data collection and analysis. Our interviewees used focus groups and interviews to develop and test hypotheses, leveraged stories to provide more nuanced context, and ensured that qualitative data was helping to uncover biases, trends, and experiences that add depth to analysis. Qualitative data approaches are well studied and offer rigorous approaches to data use for scale; find a helpful list and description of these methods on page 11 of Rockefeller Philanthropy Associate’s Assessing Systems Change: A Funders’ Workshop Report, or in MarketShare Associates and USAID’s Methods and Tools for Measuring Systemic Change.
3. Advocate for the data that you have shown makes sense for your organization.
   “Many organizations approach data in a fractured, reactive way — often driven by data that is needed to answer a question for a funder. Instead, organizations should think proactively about the data that they need to drive scaling decisions and the types of data that they can collect in scalable ways,” says Bob Filbin, Co-Founder & Chief Data Scientist of Crisis Text Line. Filbin’s comment reflects VisionSpring’s experience as well. In VisionSpring’s early years, it worked to quickly respond to its donors’ every data request. But as VisionSpring scaled and developed a deeper understanding of its impact, it tried to compel its donors to accept the metrics that were critical to VisionSpring for managing its business. As VisionSpring Co-Founder Kassalow explains, “We tried to say, ‘this is data that is important for our business. We assume you don’t want us to create data that’s only for you?’” Kassalow shared that once its donors saw that it was asking the right questions and collecting the right data, the vast majority of them agreed not to add on top of that.
4. Limit the number of key performance indicators (KPIs) and prioritize those that will drive scaling decisions.
   Bart Houlahan, Co-Founder of B Lab, acknowledges, “Data is incredibly powerful to a point. It can also paralyze an organization.” Houlahan encourages enterprises to “get good enough and go.” And Asif Akram, CTO of Living Goods, emphasizes the need to understand what data is of highest priority, encouraging enterprises to ask themselves, “Are we a data surveyor, or are we providing a core service?” For Living Goods that meant collecting only data that related to its core service — providing health care to pregnant women and children under the age of five — not collecting additional health or demographic data that the government or funders were interested to know about the population Living Goods is reaching.

Read next: Collect Data with Scale in Mind, Build Data Infrastructure to Bolster Scale or return to see all articles in Data for Scale.

This article was written by Erin Worsham, Kimberly Langsam, and Ellen Martin, and released in June 2020.