The Journey of Impact Evaluation: Five key steps to measuring the impact of your science films and media.
By Carlos Macher and Maggie Burnette Stogner
When discussing impact, we often refer to viewership numbers, how policy makers are influenced, and whether viewers are motivated by the film. But how to measure impact effectively and concisely continues to be challenging. Our media landscape is increasingly fragmented, making it difficult to know who we are reaching and with what results. And, in today’s highly grant-dependent industry, the better we understand how to evaluate impact, the better chance we have of generating support and interest in our science films and media.
Step One: Envision impact throughout the filmmaking process.
Rather than thinking of impact as what happens after the film is finished, broaden the spectrum of potential impact that your film can have throughout the filmmaking journey. What are the significant effects of your storytelling process from the very inception of an idea until the project becomes part of the continuum of the impact spiral? For example, have you considered the effect of which people and locations you choose to portray, or of the crew that helps to make the film? At each stage of the filmmaking process, there is an opportunity to spark a conversation, review your process, and set a point for a feedback loop — pretty much like planting a seed and caring for it until it grows into a healthy plant.
Envisioning the impact of a science media project as a comprehensive set of outcomes stemming from the very beginning of your project through your final outreach plan is key. Assessing and measuring outcomes is as much a journey as the filmmaking process itself. A common mistake when measuring the impact of a film is to solely base the success of a film on the number of viewers and/or likes, broadcast/streaming ratings, tickets sold. An expanded approach to impact requires filmmakers to define the many opportunities along the filmmaking journey.
Defining the filmmaking approach you use (see innovative trends in the CEF Impact Media Report) can change the landscape of impact that your science media project can achieve. For example participatory filmmaking, in which the director and the local participants work together, is a growing trend. Films such as Our Gorongosa, The Elephant Queen, and the VR 360 project Grandpa’s Reef, formed local editorial boards to develop richer, more authentic experiences for viewers and to formulate impact strategies that would serve the local communities.
Step Two: Hire an impact producer.
The Center For Environmental Filmmaking’s Impact Media research examines several case-studies examine the role of the impact coordinator, and/or impact producer. Their main goal is to identify, systematize and enhance impact opportunities throughout the journey of filmmaking. The four stages are: development, production, distribution, and outreach. Our Visionary Change Model is based on a theory of change approach. You begin by establishing clear goals about what change you envision, then determine who can best help achieve that change, and then create a film that is designed to reach, engage, and inspire those viewers. This vision is then distilled into a comprehensive impact strategy for the film project. Feedback/evaluation loops at each stage will help fine-tune your process.
The large screen documentary The Arctic: Our Last Great Wilderness had a dedicated impact team that worked with local communities to identify and execute actions to help block gas and oil drilling in this area. Early on, the team created an impact initiative named Protect the Arctic that included concrete actions such as the creation of a response campaign through TikTok, providing film clips for TikTok users to make their own short-short videos, and the use of Influencers. They also created micro-films for their partner NGOs, such as The Nature Conservancy and Alaska Wilderness League, to disseminate to their networks. These efforts generated 6.3 million public comments to the US Fish and Wildlife Service, the largest response to a federal comment period in history. Final impact: the Arctic drilling leases were blocked.
As the example above illustrates, having a dedicated impact producer can establish the film’s short-term and long-term positioning as part of the ecosystem of the impact spiral. Each effort complements and reinforces the effect of other initiatives. It is a big job and needs someone who is focused on identifying all these opportunities as early in the project as possible.
Step Three: Determine your evaluation methodologies.
The Impact Producer or team can establish and track the project’s evaluation mechanisms. There are several methodologies to consider. The traditional qualitative approaches include surveys, pre and post evaluation, and adapted Kirkpatrick training evaluation. More complex approaches include discourse analysis and technical methodologies such as sentimental analysis. There are also ways to evaluate online conversations with the help of tools or services like Sprout Social, QUID, or other open-source data-mining software. The important factor is to establish a base-line at the outset to use for comparative analysis. What is the current state of discourse? What are the indicators that provide useful information? For example, you could determine the effectiveness of a certain format, cut, and/or length, by using pre and post surveys, control groups, or if possible, more technological solutions such as eye-tracking software and/or eye-sight heatmaps in the case of VR/360 films.
To see these and other tools, and a glimpse of possible uses throughout the filmmaking process, please visit the CEF’s Impact Media Report website, under Assessing Impact.
The 360 immersive film Walking for Freedom: A Venezuelan Story, the production team analyzed the discourse and conversations in social media, as well as community in-person surveys helped by local producers during the entire filmmaking journey. This resulted in the producers deciding to tailor the film to different audience types. They created three versions of the film, one for audiences in countries that are not aware of the crisis, one for audiences in metropolitan areas to which the migrants are fleeing (such as Bogotá in Colombia, and Lima in Peru), and a third version for towns alongside the borders and the most common walking pathways. These three versions of the film are also accompanied with a community screening toolkit that presents the refugee crisis and migration challenges across South America. Each version takes into consideration the evolution of the conversation within a specific audience type, in order to inspire viewers’ receptivity and reflection.
Step Four: Form partnerships and alliances that expand your impact spiral.
Engaging possible alliances and partnerships can complement the reach of your film in meaningful ways. Take some time to research which organizations are the best for your project. Is the organization’s mission aligned with your visionary change goal? Is the intention of an investor in sync with your goals? How can an organization help further the distribution and outreach of your project? This is particularly important since, “impact is generated through an action-reaction chain that happens inside a context of long-term efforts before and after a film and around the film topic(s),” and it continues living and affecting long-after the film’s initial release.
Walking for Freedom, for example, partnered with the Inter-American Development Bank (IDB) to conduct an assessment of the effectiveness of virtual reality versus traditional film in the engagement of different populations in the IDB’s migration initiative. The Impact Producer can leverage these and other collaborations to highlight the outcomes arising from this exercise, and inform both the post-production process and the distribution/outreach process.
Step Five: Consider the long-term impact your project can have.
We often look at short-term impact, such as ratings and likes, but it is important to consider the long-term effects of your decisions as well. For example, Nómadas, directed by Emiliano Ruprah, was created by an all-local team of Mexican filmmakers and scientists. Typically, blue-chip natural history films about Mexico’s wildlife have been made by U.S. and U.K. production companies with non-local crew. Ruprah and his partners were determined to raise funds to train Mexican filmmakers how to work with scientists and how to use specialized equipment and techniques for filming wildlife. The result is an increase in local capacity that continues to expand and contribute to a vibrant documentary film industry. Nómadas has inspired a new generation of local filmmakers to participate in blue-chip projects, and serves as a model for long-term impact.
Key take-away: As each media project is unique, it also should be unique in its approach to impact measurement and evaluation. There is no standard blend, besides the recommendation of making a comprehensive, holistic, and multi-faceted approach when selecting a range of indicators. Making sure that your methodologies are composed of several qualitative approaches is important in order to grasp and systematize impact. Remember that numbers and data reveal only part of the evaluative story.
I’m excited around all the possibilities! Where do I start?
Assign a team member the responsibility of establishing an impact strategy. Begin this process as early as possible in the project. Make sure to outline a comprehensive and holistic approach to indicators of success. Ensure the impact producer is part of your team, not a partner’s. Most importantly, remember that impact is as creative as the storytelling process, thinking differently will help to push the boundaries of what is considered impactful.
