The Impact of Cultural Heritage

4 playbook design challenges (and then some)

We’ve established that understanding the impact that we (can) have as cultural institutions on society is important on two levels: it helps us show our relevance to the communities that funds us, and it makes us aware of the things we can do to have more impact (read our previous post on the subject here).

“We do this because it is all too easy to overlook the very real economic and social benefits of cultural activity. At a time of diminishing public funding it is crucial that these benefits are spelled out clearly.” Dr David Fleming OBE, Director, National Museums Liverpool

Step two is to figure out how we can get to grips with this complex beast and develop a method to answer these questions: How do you establish what kind of impact your organisation can have? And once you know what you are looking for, how do you capture that? How do you tell your story and internalise your learning points so that you will do better in the future? How do we lead you through that process in a smooth way? The kind of stuff designers love to sink their teeth in… Only we are not designing a new car here, or even a strategy. We are developing a playbook, a methodology that we hope cultural innovators and the management of cultural institutions will like to use in their daily practice (to get a clearer picture of who we are doing this for we’ve developed a set of personae). Which means we will have to borrow from many disciplines that we are not very familiar with, ranging from behavioural psychology to statistical analysis…

We have teamed up with two companies that can help us work these things out: Sinzer, an organisation with an excellent track record in impact management with whom we are running three pilots. And 30-X, a strategic design outfit that have co-created the book Design a Better Business that we are very inspired by.

In the next six weeks we will produce a beta version of the playbook. On our journey, we will face a couple of interesting design challenges:

1: Marrying Emotion and discipline

Because we want to make this a vey practical ‘how to’ playbook, the wide variety of skills and attitudes needed in each phase might be the biggest challenge we face:

• The ‘design’ phase, where we need to figure out what kind of impact our organisation can have, is highly conceptual, energetic and strategic in nature, which means it is probably best defined through co-creation processes with a full team of stakeholders. This requires some group facilitation skills.
• The ‘assessment’ phase is more intellectual and solitary in nature. This requires people who understand how to turn strategic assumptions (e.g. ‘we aspire to strengthen communities around their local heritage’) into solid research questions (e.g. ‘what is the perceived increase of community feeling in neighbourhood X that can be attributed to the activities of organisation Y’). They should know how to design questionnaires, to crunch data and to understand its validity and statistical relevance.
• Finally, turning the findings into a solid narrative- in whatever shape or form- asks for a mix of creativity and very systematic analytical thinking (don’t think for a moment you can just get away with just a bit of campfire storytelling here. Narratology is a full blown academic discipline these days).

“Sometimes I write drunk and revise sober, and sometimes I write sober and revise drunk. But you have to have both elements in creation — the Apollonian and the Dionysian, or spontaneity and restraint, emotion and discipline.” Peter de Vries in his 1964 novel about poet Dylan Thomas

Which brings this challenge down to: How do we bring these different modes of thinking/acting under one roof in one cohesive methodology?

2. Defining our point of view: circular or linear?

It is said that one of the most fundamental differences between Western and Eastern philosophy is the perception of time as being either linear of circular. Where in Western tradition the world is mostly seen as stable and therefore represented as linear, Eastern tradition is founded on the notion that we are in a constant state of recurring change, like the seasons. Without trying to be too philosophical about it, the design of our toolkit faces a similar challenge. Should we see impact assessment as a linear or a circular process? Are we interested in assessing our current impact as an endpoint? Establishing that is of course not trivial in itself, by any means. Or is the output of the assessment seen as input for our next round of activities? The latter is much closer to the way start up businesses tend to operate these days: ideate, prototype, develop, learn, pivot until you develop a service that actually solves a customer problem. We’ll see. But my gut feeling tells me that we would be wise to adopt some of the wisdom of the Eastern philosophers in this case.

3. Theorising change

Theory of Change prototype

The central element of any impact assessment is the so called ‘Theory of Change’ model (also known as the logic model, or pathway). Now I quite like the ring of that sentence, I must admit. It’s core elements are ‘change’, which is something I feel naturally drawn to contribute to, and ‘theory’ which makes it sound like it has a nicely academic and conceptual foundation. Geoff Mulgan, CEO of British based Nesta, has made some valid critical remarks about this, which can be read here. But in essence no one contests that the Theory of Change is the best attempt to date to capture what impact assessment is about: understanding the causality between activities we deploy, their outputs, outcomes and impact. Nesta made a nice explanation video (kudos to the animation artist, love the voice-over) that explains the concept and their tool.

But literature is not clear about the definition of the components of the Theory of Change (output, outcomes, impact in particular) which leaves a lot of room for confusion.

The differences can be categorised on two main lines of thinking:

1. On the one hand you have those who see outputs as the quantitative measurable results of our activities, but in a broad way: lets say all the artworks you have digitised, the website you have so carefully developed and the amount of traffic that this generates. Basically the stuff any museum, library and archive is used to tracking these days. Outcomes on the other hand are interpreted as the changes in behaviours, knowledge, skills, status, wealth, wellbeing, or effectiveness to mention a few instances, by the communities that are affected by the service. So output is anything that is more or less tangible, outcome is the perceived change in people and thus of a slightly higher order.
2. On the other hand you have those who take a different stance: outputs are only the direct results of my actions (in our example the digitised material and the website). Anything that results from that, be it tangible like visits to the website or change in attitudes, are considered outcomes. Some pinks then differentiate between short, mid- and long term outcomes.

(1) divides the world in quantifiable-qualifiable while (2) divides in ‘us’ and ‘them’.

You may very well think this is splitting hairs but believe me, this makes a world of difference in how coherent the ToC feels and what you subsequently start to measure.

Encore: in both cases there is a need to define the difference between the impact we hope to contribute to (usually big lofty things like ‘unity in diversity’) and the things that can reasonably be attributed to our actions. This is called the accountability line. Where to draw that? Another challenge.

4. Social or economic?

This is the final biggie, the stuff that you can fill whole bookshelves with… Are we interested in social impact or economic impact, or both? And what‘s’ the difference anyway? Anything social can be given an economic value if you try hard enough (this is essentially what Social Return on Investment (SROI) is about). And all things that are seen as predominantly economic (like the amount of jobs a museum creates) have a social side.

I would argue that our society has become completely obsessed with economics and have forgotten that economics are no more than a means to an end (if you want to read something really good and deep on this subject I recommend Tomas Sedlacek’s, ‘Economics of Good and Evil’. A good review of the book can be read here.)

Of course we are only the child of our time, and what is perceived to be important fluctuates like the tides. Only a couple of years back impact assessments were based almost purely on economic terms. An assessment of the viability of an investment in digitisation was conducted by calculating how much new business would be derived from it (e.g. designers using digital culture as fuel for their creative outputs) or how much savings could be expected (all the way to researchers not having to pay a train ticket to consult a work in a distant library and the resulting reduction in co2 emissions, quantified in euros). The equation we were looking for at the time was a straight ‘new business + savings > investment’. Anything ‘social’, like improvements in digital literacy, would end up as an undefined + at the end of the report. We did such an evaluation at Europeana in 2013 which you can read here.

Lately we have seen attempts to capture both the Economic and Social impact in more sophisticated ways. The National Museums of Liverpool, who I cited earlier in this article, have delivered an impressive report about the effects of their activities on the local economy and is appealingly specific about it’s social impact. They have packaged their substantive research in a nicely abbreviated document and a catchy video.

As you can see the economic impact has been captured in quite a convincing way: they reckon they are worth around 53 million pounds a year to the local economy. How is that calculated? Jobs created (1200) directly at the museum, indirectly by attracting visitors to Liverpool who spend money on hotels etc. and induced through the money pent by the people who earn an income from the tourist spending. Pretty straightforward economics, no questions here. What we should keep a sharp eye on is that creating jobs is not why we have a museum. A museum has a whole other mission. Liverpool Museums not only knows that, they act on it. Which makes this study so sympathetic. Liverpool Museums cares about educating the young and helping the old. For example, it runs a dementia awareness programme, House of Memories and with this they are very clear about their accountability line: They don’t claim to cure dementia (which we would call an impact) but they do aim to support caretakers in their job (outcome).

There is one thing that still bothers me. The museum claims an estimated 130 million pounds worth of social impact, on top of the economic. How that is calculated: ‘Regularly visiting museums has been shown to deliver a boost to people’s subjective wellbeing worth £3,200 per year.’ If you dig deeper, the full report tells us that this is calculated based on research by expert Daniel Fujiwara from the London School of Economics. I’m sure this has been done with great care and integrity. We have made similar claims in studies that we have conducted at Europeana, perfectly acceptable practices. But how convincing is it, really, to turn valuable social impact into an economic equation? It’s like asking: “Do you love your mother? How much is that worth to you?” Bit silly isn’t it? I understand the attraction of being able to calculate the return on investment like the next guy. But would it not be more convincing to express social value in social equity? Can we not develop an emoticon, a bitcoin for happiness that expresses univocally the (relative) value we attach to a certain state of well being? Until then, perhaps we should stick to smart user satisfaction surveys and creative ways of expressing the outcomes (film, data viz, …).

This, and plenty of other stuff is on our minds at the moment. We are entering this space as passionate newbies. This has the advantage that we have a slightly naive but fresh perspective. If you’d like to take part in the conversation, join us in this LinkedIn group.

*This article was originally published on impkt.tools