How Cher, Columbo, Harry Lime, a pack of post-it notes can improve your audience research
In this instalment of Tales from the Clipboard I am going to share some approaches I have found to elicit more candid responses in surveys, depth interviews and focus groups. Cher, Columbo, Harry Lime, a pack of post-it notes and a conference table all have starring roles in this story.
Are the people we talk to in focus groups, depth interviews and surveys being completely honest with us? In many instances the answer will be no. It’s not that your audience are telling you lies with maliciously intent. It’s that they need reassurance that you really want to hear what they think and feel; reassurance that you will respect their opinions and actions; and reassurance that you won’t judge them for thinking, feeling or doing those things.
This is an issue all of us working in audience research need to tackle. After all, what’s the point of doing audience research if you don’t get reliable data?
Success in Circuit lies
How can you persuade your audience to be more candid with you? What follows are some formats of questions that I have found to work, over the many, many years I have been doing audience research.
The Cher question
In 1989 Cher posed the philosophical conundrum “If I could turn back time, if I could find a way …”. This illustrates the first type of question I often use to try and elicit thoughtful and candid responses. For example, suppose I was evaluating visitors’ feelings about a workshop they have just taken part in. Instead of asking “What didn’t you like about this workshop?” or “What was your least favourite part of the workshop”.
Instead I prefer to ask:
“If we were running this workshop again, what should we do differently? What changes should we make?”
It is an easier, gentler way for people to make critical comments — by suggesting a future change, rather than a blatant criticism of what has already happened. They don’t have to openly admit to disliking something; they can infer it by suggesting something should be changed.
This format of question also captures constructive feedback from those for whom the experience was a fairly good, or quite satisfactory — but still with room for improvement. These audience members might not even get asked a “what was wrong”/ “what didn’t you like” question.
But a note of caution: be wary when asking your audience about their future behaviour. We all have the best of intentions do exercise more, eat more healthily, volunteer our time for good causes and so on. And we will say this in interviews, on questionnaires and in focus groups. However, all too often we never quite get around to it.
By the way, I am definitely going to take up that gym membership next year.
The Third Man question
A few years ago I was conducting some telephone interviews with teachers about a new educational resource that aimed to help them teach literacy skills. Before asking their opinions about the resource, I wanted to find out about the sort of problems teachers encountered when teaching literacy. My first question was going to be: “What problems do you experience when you teach literacy?”
Just before the first interview I read over my list of questions, and it struck me this really wasn’t a good question. The unintended implication was that the interviewees were, in some way, falling short. I risked appearing to ask “tell me about how you aren’t very good at teaching.” Not a great question, and certainly not a way to start an interview. So, I changed the question to the following.
“In your experience, what problems do teachers encounter when teaching literacy?”
Many were perfectly happy to talk about the difficulties they personally encountered. But the difference is they could — if they so wished — protect their self-esteem and respond in terms of ‘teachers in general’, or ‘teachers like me’, or ‘my colleagues’. It was another example of how to soften a critical question. We find it easier to talk about difficult topics if we can put a little distance between ourselves and what we are admitting to; if we can answer in the third person — someone who happen to be like me.
Interestingly middle aged men with no previous interest in sport, often say they are going to take up gym membership; yet never actually get around it to.
‘Cards on the table’ question — or at least your post-it notes
A while ago I was running a focus group with some teachers about a new educational resource. They were saying nice things about it, but something wasn’t quite right. They just didn’t look or sound all that enthusiastic. Were they just being polite? In a flash of inspiration, I went off-script and asked the following.
“Suppose this table we are sitting around is a rating scale. Where I’m sitting represents the worst educational resource you have ever come across. The other end of this table represents the very best educational resources you’ve encountered. I’d like you to take a post-it note and place it where you feel this resource falls on that scale — from the very worst, to the very best resource”.
And it worked! Far from the bland compliments I’d been hearing before, most of the teachers placed their post-it notes somewhere in the middle, with one or two closer to my end of the table. Not only did this give me a clearer picture of what they were feeling, it also provided an opening to discuss what needed to be improved: “What do we need to do to move this resource closer to the success end of the table?”
There is something about asking people to physically represent their feelings which seems to yield more candid responses. I guess in part that’s because it’s harder to equivocate when you need to put a small square of colour paper on the table. This leads me to the next type of question to consider using.
The closed question
A closed question is one where the person responding is provided with a menu of pre-defined answers from which to choose. Closed questions include multiple choice, yes/no, or rating scale questions. They have many advantages, including one relevant to this topic. With a closed question you have far less opportunity to be evasive or to omit details.
Think of this example. You are looking to buy a house and you want to find out about the risk of it being flooded. Which of the following questions is more likely to yield the information you are after?
“Tell me about the history of this house”
“Has this house ever been flooded?”
Closed questions are widely used in quantitative research — on questionnaires and large scale surveys; but they are also really useful in qualitative research — such as depth interviews and focus groups — where you want to check what you are hearing. For example, at certain points in the discussion I will ask:
“So, what I seem to be hearing is … . Is that a fair summary of your opinions?”
Have I ever considered joining a gym? No, of course I haven’t.
The Columbo question
As the great TV detective Lieutenant Columbo is about to leave the room, he turns and says to the suspect, “Just one more thing …”. And then comes the question that cracks the case wide open and brings the murderer to justice. That’s not quite what I’m suggesting you do, but it does illustrate another approach to getting frank responses.
By the end of an interview or focus group you will have built a rapport with the people you are talking to. They will be more relaxed and so will you; it’s almost over and nothing awful has happened. And they will have begun to understand what sought of information you are seeking.
At this point, interviewers often ask a feeble question like “Do you have any more comments?” Aside from being a closed question likely to elicit a simple “no”, this is a missed opportunity to do something much more powerful. Now is the time to reveal why the research is being conducted — to briefly explain what the project team hoped would have happened; what they feared might have gone wrong; or whatever else it is that’s motivated them to commission this research. And then you ask:
“What do you feel about that?”
It’s remarkable how frequently this elicits really rich, insightful responses. Often interviewees and focus group participants have been holding back these responses, unsure if this is what you really wanted to hear about.
Another note of caution here: you must leave the Columbo Question to the end of the interview or focus group, to avoid inadvertently feeding answers to your interviewees. They will be listening out for any clues about what they “should” be saying. Like Columbo, you must wait until you’re almost out of the room before you ask this question.
Now the reason I’ve been banging on about gym membership is …
One last thing
All of these questions will help but there is also something else you should do in all of your interviews, focus groups and questionnaires. Simply ask your interviewees to be honest.
It’s surprising how rarely we do this. I will always try to start an interview or focus group by reassuring participants that I won’t be offended if they didn’t like something; that I need to hear about the things that didn’t work as we want to improve what we offer; and so on …
Don’t assume that the people you are speaking to will assume you want an honest response. They may genuinely be concerned that doing so will upset you. I often make it clear to interviewees and focus group participants that I didn’t personally build that exhibit, or design that workshop.
And it helps if people feel you genuinely want the information so you can improve things — rather than you are just doing a tick-box exercise for someone else.
The slanted truth
Getting candid, thoughtful answers is hard, but without them there is little point in conducting audience research. It’s not that people are wilfully trying to deceive you. It’s that they need encouragement and reassurance. It takes a bit more effort but as I always say — just because it is difficult to do, doesn’t mean it’s the wrong thing to do.
To quote Emily Dickenson:
Tell all the truth but tell it slant —
Success in Circuit lies
Too bright for our infirm Delight
The Truth’s superb surprise
