How do you measure the impact of a story when it’s the result of a collaboration?
By Tara George
Over the last several years, measuring the impact of journalism has become a hot topic in the media industry.
Journalists have long used significant changes in laws, the ouster of corrupt leaders and restitution for wronged people as evidence of their impact. But recently, we’ve seen impact being measured in a more nuanced way; today, U.S. news organizations use all kinds of methods to track how their work is making a difference in communities that go far beyond only the most visible.
As philanthropists swoop to the rescue of quality journalism, journalists are increasingly having to find ways to give their benefactors proof that the investment is paying off.
As a result, methods for tracking the impact of a single organization’s stories have gotten a lot of attention, but little research has been done on how to measure the impact of a collaborative journalism project.
That’s why the work of impact strategist Kara Wentworth and collaborative journalism researcher Sarah Stonbely was particularly instructive when they presented together at the Collaborative Journalism Summit at the Center for Cooperative Media at Montclair State University recently.
Though their findings are preliminary, their message is optimistic: yes, measuring impact on collaborative projects can be difficult, but it can be done.
“Every industry thinks their stuff is hard to measure,” says Wentworth. She added: “In collaborations impact tracking can be more challenging but it can also be more authentic.”
Wentworth has a rare job title: Strategic Impact Analyst at Twin Cities PBS (TPT) in Minneapolis, MN. In other words, she’s one of the few people in media whose job is solely to evaluate the impact of the work her station produces.
Because public media outlets like hers have relatively long histories of collaborating with nonprofit and other partners, much of the impact-tracking that she does is of collaborative projects.
“I focus on the ‘so-what’ of what we do as journalists and media-makers,” says Wentworth, whose position as impact analyst at TPT started in September 2017 and is funded by an ACLS/Mellon Foundation grant. “I’m asking, ‘How are we doing in our mission to enrich lives and serve communities?’”
For the last 15 years, the station has partnered with over 325 non-profit organizations, including those creating impact to enhance arts and culture, health, community affairs, science and environment and youth development in Minnesota. TPT has created over 900 long-form programs and 2,300 shorts in that time, she says.
Wentworth’s arrival at the station last year represents a shift in thinking by upper management that in addition to measuring the impact of a story by traditional television methods, chiefly the Nielsen ratings which calculate viewership, all of their work could be evaluated by looking at impact in a broader sense, and their collaborative work could be looked at by including the goals of their partners as well.
For instance, in June 2017 Twin Cities PBS (TPT) aired an hour long documentary, “Sold Out: Affordable Housing at Risk.” They partnered in the project with a non profit Minnesota Housing Partnership that advocates for affordable housing in the region.
By TPT’s own measures, the package was a success: it reached over 1 million households and there was great online engagement.
But the additional impact that Wentworth was able to measure was more in line with the goals of their partner in the collaboration: the preservation of 1,190 units of affordable housing that she believed she could link directly to the airing of the documentary.
That total, Wentworth says, breaks down to 768 units that were saved when a seller was influenced by the film to accept the bid of the only developer planning on keeping the units as affordable housing. Additionally, she counted 422 units of housing where city council members sat down and watched the documentary and voted to preserve the affordable nature of some housing units up for bid. Afterwards the developer sent TPT a note saying the film had undoubtedly played a role in the council’s decision.
“We are finding ways to blend qualitative and quantitative data to combine numbers with stories,” says Wentworth.
She says impact differs from project to project. In a collaboration between TPT and a national Vietnam veterans performing arts group, The Telling Project, the data that Wentworth used to evaluate impact included the individual experiences of participants and audiences, such as a young woman who said watching her father tell his war stories brought her closer to him.
Wentworth also counted the veterans who came forward after seeing a performance and said they were inspired to seek help, and the fact that the show led to ten more new Telling Project performances in communities throughout the country.
She says that part of her job is to “design for impact,” by making sure that the station doesn’t wait until after a project is completed to assess outcomes. Instead, they make sure that thinking about impact is baked in from the beginning.
Thinking about these yardsticks for measuring impact are part of the worksheet Wentworth has developed to help guide collaborations from the start.
Most organizations will also want to develop indicators that cut across projects, Wentworth says. Often the biggest struggle in collaborations is figuring out who will be the person whose job it is to spend time tracking the impact.
Stonbely is the research director at the Center for Cooperative Media, and she says Wentworth’s work is interesting because many reporters in journalism collaborations are not thinking about measuring impact and, if they are, they’re often measuring it in an ad hoc way.
This was something Stonbely found in her 2017 “Comparing Models of Collaborative Journalism,” and which provided the impetus for her new research project looking at impact tracking in collaborative projects.
She says there’s a mountain of research on the issue of impact in journalism, but very little attention paid so far to assessing it in collaborations.
“The question is: ‘Is assessing and measuring impact different for a collaborative project?’” said Stonbely. “Based on early findings, we think that it is.”
So far, Stonbely has identified some key impacts that are unique to collaborative journalism projects:
- Learning new skills, the result of journalists getting exposure to techniques used in other newsrooms such as editing or digital skills.
- Brand legitimacy, particularly when a smaller outlet partners with a better-known outlet and gets a boost for their brand. This happened when the smaller Charlottesville Tomorrow joined forces with larger more established The Daily Progress in a symbiotic relationship that gave prominence to Charlottesville Tomorrow and resources to The Daily Progress.
- Legitimacy for content, which can happen when a national news organization parachutes into a community for a national story and partners with a local newspaper, a more trusted source in the community.
- Improved content can occur simply because there are more resources available. A classic example is the Panama Papers, in which journalists from over 100 media organizations analyzed a massive amount of leaked documents.
- Greater distribution of content that occurs when multiple news organizations operating on different platforms work together.
- The Black Swan Effect, a term used by Jessica Clark, the director of research at Media Impact Funders, to describe that moment when something magical happens in a collaboration to produce an unexpected result, for example the Dirty Little Secrets collaboration looking at toxic waste in New Jersey involving multiple news organizations, ended up hosting a comedy show produced based on their reporting.
Stonbely says the final report on her research will be out in the fall and will aim to build on these preliminary findings to provide useful tools for reporters working on collaborative projects to map out from the outset how to track the impact of their work.
Tara George is an associate professor of local journalism at Montclair State University. She can be reached via email at georgeta@montclair.edu, or on Twitter at @journoprof.
About the Center for Cooperative Media: The Center is a grant-funded program of the School of Communication and Media at Montclair State University. The Center is supported with funding from the John S. and James L. Knight Foundation, the Geraldine R. Dodge Foundation and Democracy Fund. Its mission is to grow and strengthen local journalism, and in doing so serve New Jersey residents. For more information, visit CenterforCooperativeMedia.org.
