From correlation to causation: the importance of mixed methods in impact measurement
Impact measurement is an imperfect science, and as a team that specializes in strategic research, we frequently get asked how we know that the work an organization does is directly connected to the outcomes we see in a community. In a messy world with what can feel like immeasurable variables, how can we ever understand causality?
While it’s nearly impossible (outside of a controlled environment) to fully attribute change to any one program, grant, or community effort, we still try, by gathering information from a variety of sources, using a variety of methods. At the Impact Architects, we use social science research methods to map where communities and organizations measure up relative to a vision and define outcomes that would move them closer to their goals. To measure progress, we have to understand the limitations of available qualitative and quantitative data and use mixed methods — collecting different types of information from a variety of sources — to understand the reality on the ground.
An example of how we can use mixed methods to better understand community health through the lens of news and information is our Healthy News and Information Ecosystems Project, where we use data from the U.S. Census, Pew Research, as well as community and industry organizations, paired with interviews and surveys.
One of the case studies we did was about the news and information ecosystem in Detroit, Michigan. Detroit is one of the few major U.S. cities that maintains two daily newspapers — the Detroit Free Press (owned and run by Gannet and part of the US Today network) and the Detroit News (owned by Digital First Media/Alden Global Capital). With two competitive newspapers, the assumption might be that community members have access to more news and information and are better served than those in other cities with fewer news institutions.
However, veteran journalist Debra Adams Simmons’ 2017 research, which included conversations with residents and journalists, painted a very different picture. Simmons’ SWOT (strengths, weaknesses, opportunities, and threats) analysis of the Detroit journalism landscape outlined weaknesses in the ecosystem due to distrust, dissatisfaction, and lack of diversity and threats from changing preferences, generational shifts, and a questioning relevance of local journalism.
In our research, we compared demographic data from the city with that of newspaper staff to understand and contextualize one potential source of Detroit residents’ distrust in local journalism. As of 2019, 79% of the city of Detroit identifies as Black/African American, 10% as white, 8% as Hispanic/Latinx, and 2% Asian. The Detroit Free Press meanwhile, in their 2020 diversity report, reported their staff as 70% white, 12% Black, 3% Hispanic/Latinx, and 8% Asian. (This is an even smaller percentage of Black employees than the 17% Black reported in the ASNE survey in 2018).
The data itself can’t tell us that this lack of representation in the newsroom is why distrust and dissatisfaction permeates Detroit’s news and information ecosystem. Only conversations with journalists, community members, and leaders on the ground can elucidate the impact of diversity and representation on the content produced and consumed by the people of Detroit.
Quantitative data — such as the demographics of the community and the newsroom, and quantitative measures of trust gathered in third party research or polling — can perhaps tell you the what of your question. But in a world where everything is deeply contextual, and the strengths and weaknesses of a journalism ecosystem is influenced by innumerable changes in the community, only qualitative data that is grounded in the reality of real people’s lives and experiences will tell you why and how Detroit’s journalism landscape, and it’s lack of representation is or isn’t impacting community members. To more directly understand the relationship between representation in the newsroom, and the trust and satisfaction in journalism in Detroit, you need qualitative grounding, running your own community surveys and collecting critical qualitative data through focus groups and interviews.
In Detroit, organizations such as Outlier Media are structured to bring journalism directly to people needing information, through journalists who respond to text-based questions from community members. Outlier measures its own impact regularly through feedback from the community, using both the quantitative data they collect on individual’s information needs, alongside qualitative conversations with Detroit residents.
If you are interested in learning about how other journalism organizations measure impact in their communities and our methods for collecting both quantitative and qualitative data, see our Healthy News and Information Ecosystems Playbook, which can walk you through measuring the health of your own local news environment.
Impact measurement isn’t a static snapshot of a moment in time. It’s a process of identifying a vision of the future, the steps needed to reach it, and the data that would help you know if you succeeded. It’s a process that brings people together to understand the narrative of a community, and where your work fits into its future.
