The Tricky Business of Close vs. Open-ended Questions Especially in Interviews
Just last weekend, I found myself in a scenario in which I thought:
This. This is what it means to ask good questions! Why am I not recording this?!
The place in question was our latest OpenIDEO Los Angeles workshop, and the scenario was this:
Teams were doing rapid paper prototypes of possible solutions for volunteer retention at L.A. Kitchen. Before hitting the streets to find users to interview, one team — let’s call them Sam, Cam and Tram — wanted to practice their interview questions on me. As I was the facilitator of the event, I agreed.
*Please note: What follows is a reenactment. I don’t remember the specific statements. Apologies.
“Do you volunteer?” Sam asked.
“No,” I said. I saw the answer unnerved Sam by the way her eyes flickered. But she gathered her composure and asked another question.
“Why not?”
“I don’t know.” I said. Sam held her ground. She asked a few more questions like: What types of events did I go to? What topics interested me? And what would make me volunteer?
All of my answers were equally unsatisfying. Because I didn’t generally volunteer, I couldn’t answer any questions about volunteer retention, and all of my other answers just seemed to lead us further away from the topic. Everyone felt it.
Then Cam and Tram jumped in with their paper prototype. It looked something like this.
Cam asked, “What do you think about these topics for our blog?”
“Honestly,” I said, “I have no idea. They are topics. And I guess they seem fine.”
We had hit another road block.
At this point, the team called a timeout, and we talked about what was going wrong with the interview. On the team’s side, they were concerned about validating their prototype within the time limit; there’s wasn’t much time left to make another prototype, redo questions, or screen people on the street for volunteer experience. And I, as their facilitator, was concerned about teaching them better interviewing skills, but my explanations like my answers were equally lame. So instead, I turned to Tram and started interviewing him.
“When was the last time you volunteered?” I asked him.
“About six months ago,” he said. “I helped out with a reading program.”
“Had you done it before?” I asked.
“No. A friend invited me.”
“Have you volunteered their again?”
“No.” He said.
“Why not?”
“It was a really fun experience but I guess they just never followed up with me.” He said.
“Would you do it again?”
“Oh yeah. Definitely.”
I turned to the group, “See what I did there?”
“You asked open-ended questions,” Cam said.
“They helped you dig deeper,” Sam said.
“So we should only do that,” Tram said.
Well, no. But it took me awhile to figure out how to articulate what I did, and that’s why I’m writing this article! Let me take you through it.
#1) Close-ended Questions Aren’t Bad But They Are Often Misused
People generally assume close-ended questions lead to close-ended, uninformative answers like “yes” or “no.” In our scenario, Sam launched the interview with one: “Do you volunteer?” The problem wasn’t that Sam used a close-ended question. The problem was that the team wasn’t prepared for the answers. And in not being prepared, they couldn’t steer the interview into more productive waters.
Because the team hadn’t prepared for a “no” answer, they got stuck, and as such, they lost out on a ripe opportunity. What makes close-ended questions effective in an interview is that you, the interviewer, knows what the answer will be. It’s like a Choose Your Own Adventure Story in that your subject will choose one, and you already have decided on how that will affect your question strategy. No matter where the conversation goes, you can always use a well-placed close-ended question to reaffirm the focus of the interview and get everyone back on track.
#2) There Is No Right or Wrong Question; There Is Only the Answer That Begets Another Answer
An interview is a conversation that happens within a framework. The interviewer understands what the framework is which is where the content of the interview will eventually go. In the case of my team, the framework was a qualitative interview about a prototype. The subject of the interview rarely needs to know this, but the interviewer does because it defines the intent behind the questions they ask. As the interviewer, you need to learn specific things, and as such, the questions that you ask help you learn these things.
But here’s where newbie interviewers get hung up — they worry about asking the right questions to get the right information. In their mind, there is a difference between a “right” and a “wrong” question and a “right” and “wrong” answer. But the master question-asker doesn’t make these distinctions. To the Q-master the only question that matters is the one that gets them an interesting and useful answer. And the truth is that when I’m interviewing, I rarely think in terms of “right” data or “wrong” data. Rather, I want ALL THE DATA because that’s always more useful than some of it. So as an interviewer, the real quandary I face prepping my questions is this: If I want ALL THE DATA but I only have X amount of time with the interview subject(s), what questions do I prioritize to help me get LOADS OF DATA out of the situation. I am rarely picky about what LOADS or DATA means in these scenarios. And because of that, I use my questions as a loose framework to define them.
This is an important mind-set because it means that rather than getting hung up on my questions, I’m just trying to shake my interview subject down for as much information as they are willing to share with me within the time that I have with them. If I can get the subject talking with one or two well-placed questions, then I’m golden. And rather than look for answers that validate my prototype, I’m looking to get to the deep, heartrending ID-stuff ASAP. That’s where all the good information lies anyway.
This is why even if I’m not talking to a target user about my specific design solutions, I’ll still try to get valuable insights.
For example, I eventually took a stab at the prototype. I didn’t want the team to leave thinking that they couldn’t turn it into a useful research tool. I held it in front of Sam.
“What topics interest you?” I asked.
“Ethnic cuisine especially in Los Angeles,” Sam said.
“Why?”
“[Insert a really interesting answer that I’ve forgotten because I didn’t record the conversation, but Sam basically talked about why she loved these topics],” Sam said.
“How do you find out about these topics?” I asked.
Sam preceded to list all the ways she stays passionate about these topics. We then talked about these topics in general and whether she volunteered with any causes attached to them.
See? It had nothing to do with my prototype, but I had to rethink my approach on getting useful data. Once I did, I learned what topics were interesting to a user like Sam, how she interacts online with those topics and I have competitors to look at after the interview.
#3) Open-ended Questions Give You the Most Flexibility in Your Quest for Information
Let’s recap:
“Do you volunteer?” This was the team’s opening question. It’s a close-ended question. There are only two ways to answer this question. And the question itself doesn’t ask for anymore information. The team may have expected their interview subject to say something like, “No because …”. But that’s an assumption, and as an interviewer, I actively try to not leave myself open to those gaps when deploying that type of question. I also want to re-emphasize that starting with a close-ended question isn’t a bad idea. Rather, the team hadn’t considered how to use it to open the floodgates of information. And as such, they got stuck with close-ended answers
Then there was my opening question: “When did you last volunteer?”
That’s an open-ended question, and the reason it’s open-ended is because a million different answers apply. As the interviewer, I don’t know what the answer is, but I don’t need a very specific type of person to answer it. Whether my subject has volunteered only once in their whole life or every Sunday for the last decade, the question applies to them. They have an answer no matter what. And no matter what their answer is, I have two follow-up questions to beget more information: 1) “What was it?” and “Why did you do it?” I just need to create the space for them to say it, and that’s exactly what I want as the interviewer.
Thanks for OpenIDEO and all the ideators who turned out for our prototyping workshop! I’m always impressed by your curiosity and passion to practice design thinking!
Interested in learning what I do daily to stay sharp as a designer and strategist?
Subscribe to my newsletter, and find out for FREE!
Articles in My Series
“The Art of Asking Questions: A Design Thinking Workshop”
“The Question Asker-er: Tactics on How to Use Questions to Get What You Want/Need.”
“The Tricky Business of Close vs Open-Ended Questions Especially in Interviews.”
Update: If you’d like to see how a pro like myself builds user experiences in real time for real clients, check out my online courses for product makers and designers.
