My Take: Current & Future Evaluation and Measurement Techniques for Cultural Strategy

This article follows an annotated reading list on evaluation and measurement techniques for cultural strategies, and is focused on my analysis of current themes in the readings and my ideas for the future. As I said in the previous article, it is clear that better and more systematized measurement and evaluation is a “next big hurdle” for our field. I want these two articles to help fast-track that work by spurring conversation and, more importantly, collective action and funding. I welcome comments, others writing articles of their own, individual and group conversations, hack-days to build new tools, and anything else that can get us there.

Current Themes in Cultural Strategy Evaluation and Measurement

The first common theme in the readings and discussion was whether evaluation and measurement of culture is even possible. As Einstein famously said, “Not everything that counts can be counted; not everything that can be counted counts.” Indeed, there were some writers who said it can’t. Others criticized the philanthropic community for requiring evaluation and measurement that often feels like busy work. Some went further to wonder if funders read our evaluations at all, and if our evaluations are mostly used to defend their own job performance. Cultural Strategy: An Introduction and Primer uses a historic and structural lens to say that the difficulty of evaluation lies in the historic lack of funding to cultural organizing and the long time frames required to change culture vs. grant timelines that are typically six to twenty-four months. Regardless of philanthropy’s role in evaluation, it was widely agreed that the learnings of our field need to be re-focused on systematically helping us to do more effective work.

Many articles acknowledge that measurement in culture, while not impossible, is certainly difficult. They talked at length about how change in culture, and the narratives and norms that live in it, will most likely not present in a linear way because culture itself is a highly complex system. Furthermore, no one experiences a single piece of content or culture in isolation to others, and therefore a scientific claim to causation is not possible. Indeed, cultural strategies do not naturally lend themselves to randomized control studies. Rather, we can and should focus evaluation on measuring only the contributions of our work towards larger societal impacts.

Almost all of the articles talked about the need for projects and campaigns to have clear goals or objectives with indicators of success built into them from the start. Several highlighted that these are best when co-designed with a community. (The Equitable Evaluation Initiative has some guidelines for this that can be found here.) While some common goals and indicators might be good to come up with as a field, many will have to be specific to each project. John Bare suggests that we identify tools that can give us constant feedback on these important indicators. He further talks about how “negative feedback” is not actually negative but simply an indication that a change is needed, much like how a thermostat takes temperatures and then adjusts the heat or AC according to the temperature “feedback” it gets.

How can we build thermostats for cultural strategy projects or campaigns? (And yes, it’s hot AF. I live in New Orleans.)

Other articles talked about the need to articulate a project’s specific goals in the context of engagement ladders. An engagement ladder is a framework designed to deepen community or audience engagement — an articulation of how an organization can take a casual supporter and turn them into an accomplished change-maker. Learning for Action describes an engagement continuum that ranged from an individual’s consumption to contribution, and from low to high effort activities. Animating Democracy created a wave-like “continuum of impact” that follows how it believes arts and culture can affect individuals and institutions. Their continuum looks like this: knowledge → discourse → attitudes → capacity → action → conditions. Finally, the evaluation model that Te Papa, New Zealand’s national museum, focuses on measuring key indicators along a visitor experiences in the museum’s physical and digital spaces: attention → reaction → personal connection → insight → action.

Almost all of the articles offered a variety of examples of indicators and metrics. There were a lot of great sample indicators and goals, especially in Doc Society’s “Impact Guide” and Learning for Action’s guide. Animating Democracy’s guide includes categories for indicators and a helpful worksheet for customizing them for your project. Many articles talked about using a variety of measurement tools, including big and small data, qualitative and quantitative, and descriptions of context and unintended outcomes to share our impact or success stories. These tools commonly include things like social media listening, network mapping and analysis, keyword searches, content analysis, surveys, focus groups, in-depth interviews, implicit association testing and more.

Interestingly, many of the tools referenced in articles written within the past five years — like Sparkwise, Capitol Words and Harmony Institute’s StoryLine — no longer exists. The Media Impact Project attempted to create a measurement system and data repository that is no longer active, though the open source code for how to recreate it is available. In other cases, like the Participant Media Index (PMI) or Wise Entertainment measurement systems, there is no public access to the tools. (According to this 2014 article, the PMI was intended to be made available to others, but I was unable to find access or mention of it on their site.) Almost all of the articles, recent and slightly older, emphasized the need for more, and more affordable, evaluation and measurement tools for this work.

To summarize and simplify all of this information, a project and evaluation design process might look like this:

1. Deeper understanding of your audience and its size through a multitude of listening & research techniques;
2. Design your project, its goals & indicators of success, perhaps using an engagement ladder as a frame;
3. Launch project & ongoing evaluation;
4. Deliver on project goals while collecting data against indicators from many different types of sources (qualitative, quantitative, big and small data, etc.);
5. Analyze, evaluate & iterate project design; and
6. Repeat.

Riki Conrey writes, “The unit of culture easiest to change and measure is the social norm.” Creating new norms can be a goal of cultural strategy projects and campaigns. This may seem daunting until we remember that brushing our teeth wasn’t common in America until after WWII. Just 75 years later nearly everyone does it without having to be prompted! (“Nearly everyone” does not include my 13 year old. Grrrr.)

My Thoughts on What is Needed for Better Data, Evaluation and Measurement

The first thing that occurred to me in analyzing these articles is that we need to continue to standardize some of the language we use as a field. Key terms could include impact, engagement, audience, community, etc. Learning for Action’s guide has some good definitions that could be a starting point. I have pulled them out below for our consideration.

The common refrain of needing project-specific goals and indicators that measurement can be tied to throughout the work got me thinking: at the very least, can we create a bank of SMARTIE-style goals and indicators to keep the field thinking big and creatively? Perhaps like this one by the Management Center or this KPI library. Similarly, can we build a measurement & evaluation tool library? So many tools referenced in the literature no longer work or have been shut down. Many resource lists and tool libraries are riddled with these shuttered tools and other broken links. There is currently no central location where we can find all of the working tools for measurement and evaluation. Creating one would allow the field to do better evaluation and help identify gaps in what tools need to be created or found for better systems of evaluation and measurement. If this was created as a google spreadsheet that everyone could access and update, it wouldn’t become as outdated as quickly as other libraries and resources. (Update: I created it and you can find it here.)

An engagement ladder is a framework designed to deepen community or audience engagement — an articulation of how an organization can take a casual supporter and turn them into an accomplished change-maker.

The reading also re-inspired me around how engagement ladders help our projects better meet people where they are and deploy resources to help them take action. An engagement ladder or sequence also allows an organization to create its own “conversion rates” as individuals move from one step to the next and ultimately some of them to the top rung of the ladder. To meet people where they are and take them to new places, engagement ladders must have low barriers to entry, as well as entrance ramps throughout.

Almost a decade ago at Revolutions Per Minute, I implemented an engagement sequence and assessment tool to understand how we could increase the capacity and confidence of artists to take action. We called it the RPM Index. Once operational, it instantly allowed our small and overworked staff to meet hundreds of artists where they were, and deploy the right resources at the right time to get them to go deeper in their activism and have more impact. It also allowed us to conduct real-time evaluation using an assessment tool we created on top of our ladder. As we went along, we tweaked our system to refine our resources and engagement based on what we learned worked. I believe more of these ladders and tools could help others in the field.

Now for some more radical thoughts and questions:

1. What if we, as a field, committed to sharing our project evaluations and learnings with each other somehow? What if grant-makers funded us to learn, rather than merely evaluate? Global Grand Central is a model for a field sharing their evaluation and learnings openly.
2. What if we, as a field, adopted some guidelines for our evaluation based on what is important to us. Things like we want to make sure we are developing measurements that allow for ongoing or interim progress and feedback loops, not only measurements at the end of the project. Or, we want the measurement tools to be a low burden for the organizations and projects that use them. Or, about how we can ethically gather data.
3. One of the conditions often present before periods of rapid innovation is the existence of large amounts of new data combined with the ability to analyze it. If this is true and we want to innovate our field, how could we be more intentional about ethically collecting data or monitoring culture, and sharing it with each other to analyze? What data would we need to do this?
4. More specifically, what are the data points that tell us where people find community and meaning? To understand our audiences and communities better, I believe that we need to gather more cultural affinity data — things about where people form community and come together, like fashion, sports and music. Buying this data is expensive and might not meet our guidelines for ethical collection. On top of that, according to data scientists and marketers that I’ve talked with, even purchased data of this sort is really “messy”. On the other hand, collecting this same data from real people is an act of organizing and relationship building. Asking someone what their favorite music or movies are while door knocking or in an online survey is more fun for everyone. Can we do this and add it to the voter file and other databases that we use for analysis?
5. A lot of the articles talked about how to measure a piece of content like a video or film. Riki Conrey’s piece describes how to measure changes in norms. But how do we also measure both the spread of new narratives and the disruption of old ones? A recent convening by the Narrative Initiative and Civic Hall on Narrative Tech brought together people who are doing this and other interesting narrative work. (See the hashtag from the convening here.)
6. Can we measure fun? Seriously. I believe there is a correlation between winning and having fun. On top of that, if we are talking about self- and community-care, fun has to be a part of it. If we’re talking about sustaining the work over the long-term, fun has to be a part of it. And if we are talking about bringing new people into our movements, fun has to be a part of it. As Emma Goldman said, “If I can’t dance, I don’t want your revolution.”

This could be fun. Or maybe not. (Photo by Yvette de Wit on Unsplash)

In conclusion, I’m wondering if there is a group of cultural and narrative strategists who are really excited about measurement and want to have a deeper conversation about how to answer these and other questions. If so, let’s push up our sleeves and get started.

Definitions From “Deepening Engagement for Lasting Impact” by Learning for Action.

The following are definitions of key terms that the field may want to adopt or adapt. For the full report and context, you can find the report online here.

Impact: (Page 1) “We define impact as change that happens to individuals, groups, organizations, systems, and social or physical conditions. Typically long-term and affected by many variables, impact represents the ultimate purpose of community-focused media efforts — it’s how the world is different as a result of our work.”

Engagement: (Page 18) “All of the various ways that communities interact with, contribute to, and build on your content”

Audience & Community: (Page 11) “While “audience” has traditionally referred to consumers of content, “community” reflects the possibility of readers and viewers as co-creators: contributors to a rich conversation and unforeseen directions and actions, often taking place across multiple platforms. “Community” also emphasizes that the people who engage with your content are connected to each other, and that these relationships are often important for driving higher-level impact such as civic engagement or social change.”

A Sidenote:

1. On Impact: I really appreciated The Fledgling Fund’s distinction between impact and success measurements. They defined success measurements as that which is measured against a project’s stated goals (vs. societal change).
2. On Audience: I actually disagree that audiences are only passive consumers of content. As William Gibson said in the July 2005 issue of Wired, “Today’s audience isn’t listening at all. It’s participating.” I believe that whether we let them participate in our work, is up to how we design our engagement. More on that, perhaps, in a future article.