As You Drive to Systems Change: Shift your conceptions of data

Data to track, measure, and identify systems change¹ often requires a different approach than that to collect data from more discrete programs. So what frameworks and methods can social ventures use to collect and communicate data as they steer toward systems change? The organizations we interviewed shared advice on two important questions:

WHAT should social enterprises be measuring in their systems change efforts?

HOW should social enterprises go about measuring that data?

What should we measure?

Bart Houlahan, Co-Founder of B Lab, spoke about his organization’s theory of change: “We’re trying to build a movement of people using business as a force for good to create a more inclusive economy. The approach is simple: we certify leaders as certified B Corporations (B Corps), shine a light on those leaders, and then we encourage others to follow.” Although the approach may be “simple” in Houlahan’s words, the vision is grand. Houlahan and his team at B Lab are trying to fundamentally change the system of capitalism, shifting the focus from maximizing returns for shareholders to serving all stakeholders. B Lab has defined the systems change it seeks to achieve, but what data should it collect to assess progress toward those goals?
In reviewing B Lab’s key systems change metrics, we found that they largely aligned with a framework we find useful for informing and mapping such metrics: FSG’s Systems Change Conditions. These six conditions of systems change (see definitions below) can serve as a prompt for organizations to identify the critical levers that will lead to systemic change and therefore the data points that must be tracked and measured.

Graphic from FSG’s Water of Systems Change

FSG’s Systems Change Conditions — Definitions
Excerpted from The Water of Systems Change

Policies: Government, institutional and organizational rules, regulations, and priorities that guide the entity’s own and others’ actions.

Practices: Espoused activities of institutions, coalitions, networks, and other entities targeted to improving social and environmental progress. Also, within the entity, the procedures, guidelines, or informal shared habits that comprise their work.

Resource Flows: How money, people, knowledge, information, and other assets such as infrastructure are allocated and distributed.

Relationships & Connections: Quality of connections and communication occurring among actors in the system, especially among those with differing histories and viewpoints.

Power Dynamics: The distribution of decision-making power, authority, and both formal and informal influence among individuals and organizations.

Mental Models: Habits of thought — deeply held beliefs and assumptions and taken-for-granted ways of operating that influence how we think, what we do, and how we talk.

Example: B Lab

FSG’s systems change conditions map well to many of B Lab’s systems change metrics. See below the systems change conditions, along with B Lab’s related activities and metrics.

1. Policies
   B Lab Activities: B Lab has helped create a new corporate legal form, the “benefit corporation,” and collaborates with businesses, the capital markets, and policy makers to drive adoption around the world.
   Sample Data Points:
   • # states and countries in which “benefit corporation” legal form has been adopted.
   •# policy changes that create incentives related to tax and procurement preferences.
2. Practices
   B Lab Activities: Through its B Impact Assessment (BIA), B Lab provides tools and tracks data to measure companies’ impact on workers, community, environment, and customers.
   Sample Data Points:
   • Business-level changes in BIA metrics, e.g., through its “Inclusive Economy Challenge”; 173 companies have reported substantive changes to their inclusion-related practices, working to narrow racial and gender gaps.
3. Resources Flows
   B Lab Activities: B Lab tracks the flow of both financial and human capital to understand how B Corps and benefit corporations attract these resources, compared to traditional businesses.
   Sample Data Points:
   • Amount of funding raised by B Corps and benefit corporations.
   • Revenue growth rates, e.g., 3x faster growth rate than traditional businesses
   • Measures of employee attraction and retention, e.g., B Corps have more than double the employee engagement rate.
4. Relationships & Connections
   B Lab Activities: Through the certification network, convenings, and recognition of industry leaders, B Lab empowers and inspires companies to form coalitions and pursue collective action around areas of common interest.
   Sample Data Points:
   • Actions taken, e.g., 500+ B Corps committing to net zero greenhouse gas emissions by 2030 and coalitions formed around racial equity, women’s empowerment, and salary disparity.
5. Power Dynamics
   B Lab Activities: B Lab’s mission and activities are fundamentally focused on changing the power dynamics in business — shifting from shareholder primacy to a balance of the interests of all stakeholders, including employees, underrepresented populations, and other community members.
   Sample Data Points:
   • # of traditionally underrepresented populations in ownership, board, and management roles.
   • Data on CEO pay ratios, e.g., B Lab data shows pay ratios of 6:1 versus 271:1 for traditional companies.
   • # of companies adopting the benefit corporation legal form and therefore committing to balancing interests of all stakeholders.
6. Mental Models
   B Lab Activities: B Lab recognizes that the most meaningful systemic changes come from shifting cultural narratives and underlying mental models, and so it partners with and empowers actors at key leverage points in the narrative on capitalism — universities training next generation consumers and business leaders, media, and key corporate influencers.
   Sample Data Points:
   • # universities engaged in teaching about B Corps to train the next generation of leaders, employers, and consumers.
   • Use of the term “stakeholder capitalism” in media.
   • Earned media for B Corps.
   • Engagement of influencers — including multinational corporations — which are pursuing B Corp certification.

Example: EYElliance

In another example, Jordan Kassalow, Co-Founder of VisionSpring and EYElliance, spoke about how the approach to data he had used at VisionSpring would not work at the systems change-focused EYElliance. “The idea behind EYElliance was to take a multi-sector stakeholder kind of approach, try to elevate the issue area onto the global development agenda, get governments and private sector players to come along and partner with the NGOs, and create a larger group of organizations who were caring about this issue area. That’s what the EYElliance is. It really is looking at how you solve the problem in its entirety.” In order to measure that work, EYElliance selected the most critical levers needed to shift the system (notably, also aligned with FSG’s framework) and identified data points around those:

1. Relationships & Connections
   EYElliance Activities: Building relationships, engaging the public and private sector.
   Sample Data Points:
   • # of proven solutions integrated into government systems.
   • # of investable vision activities that can attract new sources of capital.
2. Resource Flows
   EYElliance Activities: Advocating for funds to be appropriated to eyeglasses in key donor country foreign assistance budgets.
   Sample Data Points:
   • Amount of new funds raised. (Example: after four years of EYElliance advocacy, the US Government State and Foreign Appropriations Bills included a line item for eyeglasses, allocating $2.5 million in 2019 and $3.5 million in 2020.)
3. Mental Models
   EYElliance Activities: Changing the mental model around vision (i.e., vision is not just about sight but also critically about economic development, road safety, and education) and raising the profile in the development community.
   Sample Data Points:
   • # new partners and sectors engaged. (Example: International Labour Organization thinking about vision as an input.)

Figuring our what to measure is just the start, though. The next big question is…

How should we measure it?

By its nature, data for systems change is hard to measure systematically, and the outcomes are rarely controlled by one organization alone. Interviewees shared how this reality requires a shift in the way they approach data, and provided advice for others pursuing systems change. Advice from the field includes:

Shift expectations for data and evidence.

Given the nature of systems change and the challenges that it poses for data collection, interviewees spoke about having to shift their — and their funders’ and partners’ — expectations around what data and evidence is reasonable and meaningful when it comes to systems change. Some factors to consider:

• Contribution over attribution. No single organization can be in control of all the factors contributing to systems change outcomes. Therefore, organizations stressed needing to value contribution, not attribution, and be transparent with funders about what is in and out of an organization’s sole control. Organizations also stressed the importance of selecting partners thoughtfully and agreeing to data collection and reporting goals upfront. See here for more on data collection with partners.
• Qualitative measures. “Not everything that counts can be counted, and not everything that can be counted counts.” This quote, of disputed authorship,38 applies well to systems change — as not all aspects can be assessed with quantitative data, nor should that be the ideal. Embracing high quality qualitative data (e.g., anecdotes, observations, etc.) is equally important and might often be better able to capture key milestones. There are formal qualitative methods for evaluation that organizations can pursue (e.g., outcome harvesting, comparative case studies, and developmental evaluation [see a list and links on page 11 of Rockefeller Philanthropy Advisors’ “Assessing Systems Change: A Funders’ Workshop Report”]), but many approaches may be less formal, requiring less intense resources.
• Slower data cadence. We usually think about data as something that is collected regularly — e.g., enrollment numbers weekly, monthly, and quarterly — and for which trends can be easily tracked over time. But measuring systems change takes longer and is less regular: new relationships that may take months or years to formalize, policies that have changed over the course of time spent advocating and educating, etc. To track these changes, organizations must identify shorter-term progress indicators and be prepared to stick around for the long term.

Be bold in selecting long-term measures that may not be in your control.

The path to systems change unfolds over long timelines and in a complex web of stakeholders where no single organization controls all the inputs to that change. It can be scary for an organization to set goals that are bold and truly drive to the systems change needed, but interviewees stressed that setting bold goals with partners is the only way that systems change will be achieved.

According to Sara Standish, VP for Strategy, Learning, and Impact for Health Leads, “As we have shifted to be more focused on systems change, we’ve had to name population level changes we want to see. These changes are challenging to achieve but we must be courageous to pick those hard things, understand our role in driving progress, and pair that with a results-based learning approach. It is scary because we know it’s not dependent on us alone, but it forces us to think carefully and intentionally about who we bring into partnership to have the change we want to see, and to be open to measuring that in new ways.“

As Standish references, Health Leads moved from collecting only data around patient outcomes to adding longer-term, population-level health changes that all partners agree to and play a role in achieving. For example, Health Leads tracks the number of states where community health workers are making an hourly living wage, which it sees as an indicator of the sustainability of caregiving — an essential input for accessible health resources.

Go beyond counting, and prioritize critical leverage points.

Some aspects of systems change can be counted, such as the number of states enacting Benefit Corporation legislation as tracked by B Lab, or the number of dollars allocated to eyeglasses in the US Federal Appropriations Bill, as tracked by EYElliance. These quantifiable data points are often related to changes in policies, practices, and resource flows and are important to track.

But measuring progress against system change conditions, such as relationships, power dynamics, or mental models, is often much more than a numbers game. Simply counting the number of relationships built provides some notion of progress but has little meaning; rather, these data points are better qualitatively described to capture the magnitude and significance of each relationship. Is there one relationship that is critical to achieve due to the brand and power that organization holds? Is there one shift in power dynamics that will serve as a tipping point to the rest of the system?

For example, a core focus of education non-profit Pratham’s work is shifting the mental model of educational programming from a focus on increasing enrollment to a focus on achieving learning outcomes — which it knows requires buy-in from the highest levels of government. In part due to Pratham’s work, in 2012 India’s erstwhile Planning Commission came out with a learning outcome policy for the country’s 12th Five-Year Plan. The following year, the central government’s education department issued guidelines reflecting many of the core principles of Pratham’s “teaching at the right level” methodology. These guidelines have since been actioned by several state governments in the country — a major step forward in changing the education system and likely a more significant relationship than others counted by Pratham.

Incentivize other systems actors to collect and use data.

In order to collect data at the systems level, interviewees reflected that it is often necessary to collect that data from partners across the system. Since you do not control those organizations, understanding how to incentivize and motivate them is critical — and challenging. Some tips from interviewees include the following:

• Co-creating to achieve buy-in: Incentivizing others to collect and share data begins with the essential, yet often lengthy, process of co-creating data strategies together to ensure buy-in. When creating its data platform, MiracleFeet was clear that if it wanted to create systemic change, it needed data that didn’t just work for MiracleFeet, but for anyone working in clubfoot. Working together with partners, MiracleFeet and others were forced to find common ground, distilling for each partner what data points were “need to haves” versus “nice to haves,” and coming to a consensus about the data that would be collected by all.
• Articulating value for data providers: International Bridges to Justice recognized a desire within the global justice community to“use technology and data as a force multiplier.” In response, IBJ created JusticeHub as a digital home for the global community to share data related to rule of law. The data collected allows the system to detect spikes and “early warning signals” while also providing value to the global community through matching of lawyers with defendants, sharing of universal standards and training materials, and providing a platform to connect community members to each other.
• Benchmarking and the power of community: Some organizations spoke about using the data to build community or allowing for benchmarking by increasing transparency of data collected. For example, according to Houlahan, “At B Lab, we learned that benchmarking really works. Entrepreneurs are naturally competitive — they want to see how they stack up against others.” So B Lab makes its B Impact Assessment scores transparent on the website to allow for benchmarking, publicly celebrates and awards top scoring companies each year, and ensures that the impact data that B Lab is asking to collect is helpful for the organizations themselves so that they have a personal incentive to collect and share that data.

BUT… What if the data simply doesn’t exist?

International Bridges to Justice, a global human rights organization focused on protecting individuals’ legal rights and reforming the legal system in developing countries, faced a difficult challenge when it came to data: the fact that data often did not exist or if it did exist, it was wildly unreliable (sometimes intentionally). For example, many countries in which it works report that there is no torture taking place. Yet, in speaking with lawyers in the country, IBJ found that nearly all reported that individuals they defended had in fact been tortured. IBJ was not deterred by the lack of data and shared several steps it took to address this gap:

1. Using anecdotal data to identify patterns when more comprehensive data is unavailable or unreliable. IBJ works in a field riddled with corrupt data, which founder and CEO Karen Tse knows cannot be taken at face-value. “When we talk to five people and all five say their clients have been tortured, we see that as a sample. A small sample, but at least we can start to analyze. We shouldn’t be intimidated by a lack of data — there is a lot we can do with what we know.” IBJ pieces together what data is available to identify patterns and also trusts its intuition based on many years working in this field.
2. Developing proxies for data that doesn’t exist or is difficult to collect. IBJ has identified what it calls a litmus test to determine the health of a country’s justice system, given the dearth of reliable data to assess the system in many of the places it works. The organization understands that, of all of the interdependent elements of a justice system, the weakest link is usually the criminal justice system — and thus criminal defenders, the group actively responsible for enforcing due process, who usually have a finger on the pulse of the whole system. By looking at data around criminal defenders specifically, IBJ can assess the relative health of human rights in that country.
3. Recognizing that just having a metric is an important step. IBJ has worked with some Ministries of Justice to establish new thinking, access to detention facilities and formal systems around torture, representation, etc. — knowing that even if the data is largely inaccurate today, acknowledgement of the importance of the issue is an important first step. IBJ knows that what gets measured gets done and, with collaborative system-wide collection, it can work to bolster the accuracy and comprehensiveness of the data collected.
4. Moving to a bottoms-up approach to data collection. Recognizing that the data and experience exists — and has tremendous value — at the community level and that a top-down approach has yielded little data, IBJ has moved to developing a digital platform to crowdsource data. JusticeHub will not only provide critical services to stakeholders in the legal system but will also rely upon them to regularly report what they are seeing and experiencing — thus building a significant base of data to inform action at the local, country, and global levels.

Notes:
1. Systems change is defined as “the fundamental change in policies, processes, relationships, and power structures, as well as deeply held values and norms” in Srik Gopal and John Kania, “Fostering Systems Change,” Stanford Social Innovation Review 13, no. 4 (2015).

Read next: As You Have More Data Available: Consider Advanced Data Methods like Machine Learning, or return to see all articles in Data for Scale.

This article was written by Erin Worsham, Kimberly Langsam, and Ellen Martin, and released in June 2020.