5 Takeaways from “Civic Media: Impact and Assessment”

Boston Civic Media is a growing network of local academics, community organizations, governmental agencies, researchers and beyond who study and practice media and technology produced for social impact. The network stays connected through a listerv, sharing syllabi, quarterly meetings, and an annual conference. The quarterly meetings rotate among different institutions and academic hosts.

In its third quarterly meeting, Boston Civic Media forum participants examined how civic media is evaluated for success and impact through a series of lightning talks and focused discussions last week. The forum offered a critical and reflective space for participants to deliberate over the rigor, responsibilities, and ethics of impact assessment and evaluation. The day’s efforts were framed and spearheaded by gracious host Erhardt Graeff, PhD Researcher at MIT’s Center for Civic Media.

What defines the success of a civic media project and from types of data should we look to derive meaning? Who are the audiences for evaluations and what are their roles in shaping the reporting? Leaders in the fields of education research, technological arts and humanities, computer-human interaction studies, and media studies shared their provocations in lightning talks viewable here. The audience, ranging from civics education scholars to design graduate students and technologists unpacked the themes of the talks in small groups. Here are some of the takeaways:

  1. “If we value what we measure, then measure what we value”- Justin Reich, Education Technology Researcher. His sentiments echoed Erhardt’s concerns about how evaluation metrics inform the shape of future projects and their evaluations as “successful”. While evaluation is often incorporated toward the end of an implementation project or intervention, in interdisciplinary fields such as civic media, the metrics have the power to dictate how future projects are formed. How the evaluation process occurs affects all of the stakeholders differently: how empowered or inspired research subjects may be to continue the work of a project implementation, how funders prioritize which projects to fund, how researchers understand their work, etc.
  2. “Beauty and fun are not outside the dataset. They’re just not as legible.”- Beth Coleman, Director of City as Platform/XRML, Amsterdam

There are many emergent components of civic media projects identified by producers and participants as highly valuable, but not as easy to quantify, capture, and share. These components often relate to forming relationships, learning processes, gaining self-efficacy, community-building, improving the quality of engagement and so on. These qualities are not always noted as central to the project nor systematically reflected upon. Presenter and civic media scholar Sasha Costanza-Chock related how a majority of community partners shared that “community growth and healing are one of the highest priorities” in media-based interventions. But how can this be measured, as such?

There are important civic media impacts that conventional social science methods may not capture, and we must expand what gets measured and accounted for. For instance, how can we convey the significance of technologies that promote serendipitous routes and exploration of urban art? How might we approach more diffuse impacts, such as creating values and cultural norms, e.g.,through media projects for sharing sensitive and vulnerable stories? When delving into more of the rich, qualitative impacts produced by civic media, the question of time frames (short-term / long-term) and scales (from the personal to the national) is also raised. The ripple effects of impact can also be felt years later, so how might evaluation approaches factor into longitudinal analyses? Getting at these different scales and nuanced layers of emotion that many researchers, artists, community members have witnessed, believe in, and experience as powerful drivers within their civic media projects is challenging; many encountered difficulties in conveying these effects vibrantly.

3. Theorize this. Speaker and human-computer interaction scholar Andrea Parker lamented the lack of sociotechnical theory in civic media, which makes it more difficult to bridge the gaps between novel interventions and impact evaluation. The lack of theory can perpetuate many one-off findings and limited study replication; whereas, theory-building could support comprehensive analysis across systems. Many civic media interventions are so novel that there hasn’t been an effort to move from project-based work to theory-building. International education technology researcher Pedro Reynolds-Cuellar observed this inadequacy in himself and many colleagues in formalizing lessons from the field, much less distilling theory from the work. He feels that a main way of formalizing knowledge has been through crafting case studies and that there is room for growth with theory building. An emphasis on theory-building can support a more comprehensive analysis of interventions across systems. Given the practical nature of a lot of this work and the rapidly evolving media and technology landscape, participants noted that linear approaches to assessment may be outdated in this field and that agile and iterative approaches are more well-suited.

4. Show respect for the different audiences and stakeholders invested in understanding the impact of civic media. The many players in this space with different needs for outputs and measurements can range from project participants, governmental agencies, funders, etc. For instance, a funder may like to see a report replete with infographics, a governmental agency may like access to the network map of community members to work with in the future, while project participants may want access to clear data or tools to repurpose. Similarly, a funder may prioritize being interested in quantitative short-term impacts, while a more scholarly crowd may be interested in qualitative findings with societal implications on a larger, longer-term scale. These are obviously vast generalizations of different audience needs, but the BCM network has expressed concerns about how to navigate diverse the expectations of communications from stakeholders. It may also place the onus on project implementers to take extra steps to connect the dots between the scales and audiences for change. As an example, one group discussant shared the frustrations of evaluating the challenges of public art projects. While there may be a lot of affective responses to art as the primary mode of impact, sometimes the effects trickle out to a policy level- such as the Mayor of Bogota’s civic art intervention using mimes to reduce traffic crimes. One group discussed shifting away from heavily quantitative reports as final evaluations toward narratives that could better encapsulate qualitative findings, emergent project outcomes, and societal impact reflections, as a frame for the the quatifiable measurements.

5. Participatory evaluation: can we elevate stakeholders telling their own stories of connection to the work? One commonly held desire from the day involves uplifting the process and activities of project production, in addition to traditional evaluation of outputs and outcomes. For example, a Boston Civic Media participant shared an example of evaluating the value of hackathons. They reflected that what hackathon participants may cherish the most are not necessarily the project outcomes, but rather the experience of collaboration and “cultural memories” of working together. Moving forward, a way to improve participatory evaluation may be to emphasize the importance of project documentation in each step of the process, allowing stakeholders to reflect on how projects in-progress can encourage a plurality of definitions for impact and meaning-making in the interdisciplinary field of civic media.

Next Steps

While this event was full of robust discussions, BCM would like to support the production of useful materials. In the next year we will grow a repository of evaluation and assessment approaches (ranging from pre-/post-surveys to storytelling platform options) as resources for civic media stakeholders. Other great ideas were shared by participants, such as training young academics to write op-ed pieces to influence social change among general audiences as public intellectuals. While we cannot take every project on, we exist as a resource to help bring together the moving pieces to do so. Please read the live notes and watch the lightning talks here and stay updated on Boston Civic Media by signing up for our newsletter. Contact Becky for to share ideas and learn more: becky@elab.emerson.edu.