A Framework for Measuring and Directing Culture Change
#MeToo has touched everyone in the United States; 45% of Facebook users in the first 24 hours alone. And yet a recent NPR/Ipsos poll found that 44% of Americans think that men hitting on women at work is “inevitable.” Obviously, people are engaging with the idea, but it’s a lot less clear if #MeToo is “working” to change culture. Not knowing is costing us. It is costing us time and potentially costing us the opportunity to build on the momentum to implement a coherent strategy for changing American gender culture.
Measuring culture change is hard. There are a lot of ways to define and measure “working” and a lot of ways to think about success in culture change. But complexity doesn’t preclude measurement; it just calls for a robust theory. We can use what we know of the processes of psychological and social change and program evaluation to develop a framework for measurement.
I work with cultural organizers and activists to create measurable, lasting change through story. As a data scientist, I believe that we have the opportunity now to knit #MeToo, the Women’s March, and other moments into a permanent change in the American gender culture, but we can’t do that until we have real knowledge of what is changing, what we are changing, and what should come next.
Culture change is norm change.
There are a variety of ways for cultures to change, and lots of specific interventions that can smooth the way. When we’re measuring culture change, however, we’re not as concerned with how the culture is changing than with:
- Whether the culture is changing, and
- Whether we are the ones helping to make that happen (through our investments, activism, and organizing).
The unit of culture most relevant to progressive change and easiest for measuring our impact is the social norm. Our objective in changing our culture is to create a new normal, a new consensus regarding the gender attitudes, beliefs, and behaviors that are OK and not OK. Perceived social norms “Is it OK?” are a key component in generating behavior (along with intention “Do I want to?” and opportunity “Can I?”).
Social norms that have changed in dramatic cultural shifts in the US include same-sex marriage, condom use among gay men in the 1980’s, and women’s work. #MeToo is establishing a new normal in how people treat one another, in the workplace in other spheres, and how they talk about their experiences of sexual harassment and abuse. To measure our progress, we could benchmark persistent social norms that are still widespread.
To measure our success, we need to measure every step from our activism and organizing of individuals to community-wide consensus:
- Audience Experience: Monitoring Program evaluation practice draws a distinction between process and product evaluation. Before we can evaluate whether our product worked, we need to check that the process happened. If no one engages with our online content or attends our events, then any culture change is a happy event, certainly, but isn’t (necessarily) a result of our storytelling.
- Audience Belief: Persuasion Behavior is made of intention, opportunity, and social norms. On our way to changing those community-level norms, we need to change individual intentions. In this phase, we might ask our audience whether they would like to teach their daughters to do “boy” things and vice versa.
- Audience Behavior: Activation Changing behavior in our target audience is how we intend to change broader norms — by literally modifying the behavior is that the community experiences and changing what feels normal. #MeToo is a great example of a campaign that normalized a specific behavior: speaking publicly about assault and harassment.
- Community Consensus: The New Normal Each stage preceding this one concerned some audience that we had defined and targeted on purpose with our stories. But the goal of that targeting was to create new social stories that propagate into the broader public. Survey is an important tool for measurement here as well, and, once the community’s norms have started to change, we should also see changes in social, political, and artistic institutions.
We can only know what “works” through comparison.
The core question of every program evaluation is: “Is this working?” In social media communications especially, it seems like the answer is often “Yes! Look at all these impressions we got!” That’s is only “working” in a process sense. from impressions, we know we spent money, but we don’t know if we persuaded, activated, or changed a norm.
To conclude that our program is causing change, we need to compare our audience’s reactions to something specific. There are lots of ways to set up the comparisons we need but in general, I use three types of comparisons to learn what’s working:
- Results from a control group. Randomized controlled trials (RCTs) use control groups to determine whether the treatment performs better than some placebo alternative. In a recent large-scale comparison of lots of different messages intended to persuade, we compared social media metrics and survey responses in an RCT to responses from people who saw generic public service announcements to determine which messages worked the best.
- Results across different treatment groups. In the same project, I compared content performance across groups to learn what messages worked best for people with different values frameworks.
- Results from before and after a treatment. Since the culture is a unit, we cannot compare to a control. Our best shot at knowing whether we’re changing norms is tracking them using polls on panels or cohort samples over time. We can also use markers of our own work to see if we are showing up in the broader conversation. This Pew analysis shows that #MeToo has been a big part of the social media conversation, but most legislators who have amplified the message have been women.
We need a research plan to keep ourselves honest.
All the measures, metrics, and comparisons go into one comprehensive plan before we start changing the world. Planning usually makes processes go more smoothly, but in science, it’s the only way to protect yourself from yourself. When you’re really sure your hypothesis is right, it can be irresistible to change that outcome metric just a little bit or exclude just a few outliers. This confirmation bias (and some other statistical problems) multiplied over thousands of studies is how we have found ourselves in the midst of a social science replication crisis.
One of the reforms scientists are undertaking is rigorous pre-research documentation. Documenting the plan means specifying every research question, metric, and statistical analysis before the project starts. That doesn’t mean you can’t do exploratory analysis later, but it means that we have to own all our results including the wins, the losses, and the nothing-happeneds.
We can measure and adapt through phased implementation.
Good communications respond to context — they have to change as the rest of the cultural conversation changes. To be effective, performance measurement should not anchor the program, holding it back from adapting because the plan cannot accommodate change. Instead, the measurement should be an engine for swift adaptation.
Conducting programs and evaluations in phases allows us to draw real, firm conclusions about what worked while using the data to inform new hypotheses and innovations. The goal is not just to optimize, to make our contact with our audiences most efficient, but to learn as much as we can as fast as we can. The goal is to prove that there is change, and that we are the agents of that change. The reason we pursue that goal isn’t just so we can celebrate our current success; it’s so that we have all the tools and information we need to plan our next successes.
Further Reading
- Program evaluation is a mature science informed by decades of work in all kinds of domains especially in health behavior change.
- I mentioned that behavior is made of intention, opportunity, and perceived social norms. There are a number of models of human behavior, but the venerable Theory of Planned Behavior is still very influential.
