Matthew McKinnon
Feb 18, 2015 · 20 min read

By Matthew McKinnon

Originally published in the August 2002 issue of National Post Business Magazine · Honourable mention, one-of-a-kind articles, 2003 Canadian National Magazine Awards

ED SAYS ignore the one-way mirror. The microphones — two, thin and game-show sleek, hang from the ceiling on white wires — shouldn’t trouble us, either; they listen for Ed’s performance, not ours. And the video camera hidden behind the black dome? Ed has to make reports on these sessions, and he likes to sit on his couch with the tape and a beer while he writes them. Honest.

It is 8:30 on a Wednesday night in October. There are eight of us — plus Ed — gathered on the fourth floor of an unassuming office building on Bloor Street in downtown Toronto. We are:

  • Dan, a pharmaceutical quality-control inspector with sharp eyes and close-cropped brown hair. Dan eats pizza when he watches hockey.
  • Sharon, an administrative assistant and mother who only cooks on work nights. On weekends, she orders pizza for the kids when they get hungry.
  • Isabelle, a ray of sunshine who organizes home offices and routines for the self-employed. She calls for pizza when she’s late coming home from work, which is often. Isabelle’s love of Pizza Pizza is loud and proud; I sense this makes her Ed’s favourite.
  • Regan, another administrative assistant and mother. Her two girls are big pizza fans. So is mom. Regan, like Sharon, chooses restaurants based on whatever coupons are close at hand.
  • Jessie, a legal assistant who accelerates from zero to bored in seconds. She likes pizza, but only from a fancy place most of the group has never heard of, and not too often, because all that cheese can’t be healthy.
  • Terry, a commercial illustrator currently working his way through seventy portraits of twentieth-century cultural icons. Terry likes pizza from the same place Jessie does; he does not enjoy Pizza Pizza, although Ed doesn’t seem to hold it against him.
  • Me.
  • Katherine, a phone clerk at Canada’s Wonderland. She has a braid of hair longer than your arm running down her back, and read a book called Power of the Witch in the waiting room. Katherine eats take-out of all kinds, but likes pizza best.

We are young/old, thin/fat, white/black, single/married. We were strangers before Ed decided we should all get friendly; after leaving here in two hours, we will never meet again. Our bond: “the first choice in qualitative research services,” Toronto market-research recruiting firm Consumer Vision. We also share a professed fondness for home-delivered pizza and the urge to make a quick $50, which is what tonight’s client is paying for our time.

And how about Ed, with his pleated pants, open-necked dress shirt, and go-go-go, wet-toothed lust for life. Ed bounces around the room like a pocket tornado, says “t’rif” instead of terrific, and sounds like he picked up an English accent on a trip overseas and never bothered putting it down. The full effect is a cross between Eric Idle and John Parkin, the man with the bowties who sells non-stick cookware on late-night television. Ed is our fast friend and our guide, directing our thoughts, prompting our answers, encouraging our ideas. “You are here,” he tells us, palms dancing over the tabletop, “because you are pizza experts. This is the only time in your lives that people care about what you think — and you get paid for your trouble.”

Nice pitch, but there’s a catch: at least three of us don’t love pizza like Ed needs us to. There was money on the line, and we lied to get it. Happens all the time.

THE FOCUS GROUP, a.k.a. qualitative market analysis, is aimed at discovering what people think about product innovations, advertising designs, government policies, et cetera. The idea is to find a few dozen individuals with an interest (or, in some cases, no interest at all) in a product/platform/pill and get them talking about their feelings. Ken Solmon, a research buyer at marketing giant Wunderman Cato Johnson’s Toronto office, says focus groups are useful “when you want to look at a range of opinions, range of attitudes, even range of language — how do people talk about analgesics, for example? Do they call them analgesics? Do they call them painkillers? Headache medicine?”

This country’s qualitative-research industry is worth $130 million a year, according to national watchdog Professional Marketing Research Society (PMRS). The typical session lasts two hours, includes between six and nine participants selected to fit a specific consumer profile, and costs between $3,500 and $4,500. High-volume marketers like breweries, banks, packaged-goods and car companies hold sessions weekly or more often. A major consumer-product company can spend $30,000 for one seven-session study, which is generally considered the minimum needed to find out anything important. Some corporations have in-house qualitative-research divisions, but much of the work is carried out by buyers like Solmon. A client calls with a question; the buyer contracts recruiting firms, facility owners and moderators to find the answer.

People in the business tend to have favourite success stories. “Have you noticed how Bounce comes in a Kleenex-style box now?” asks Brett Marchand, president of Toronto advertising firm Roche Macaulay who, a couple careers ago, worked in market research for Proctor & Gamble in the U.S. “Part of that was because we did in-home qualitative and watched one person after another have the box fall apart as they were ripping sheets off the roll.” A popular example is the Freedom 55 campaign. The London Life retirement fund floundered through its first three years until, during a last-gasp round of focus groups, a moderator asked his young-adult participants to shut their eyes and picture their ideal retirement. They described one paradise after another. In fifteen minutes, the session produced the broad strokes for a brand-saving campaign.

Companies go into focus groups hoping for similarly constructive feedback. And though they rarely get it, this hope helps explain why the sessions are as influential as they are. After all, focus groups studies aren’t statistically valid, and they aren’t supposed to be — that’s left to telephone surveys and other forms of quantitative research, which generally does a lousy job of getting inside people’s heads. But while most marketers will acknowledge that qualitative research isn’t scientific, focus groups regularly produce major changes to product and promotional strategies. Earlier this year, for example, Danone Canada altered its English-language television campaign for Silhouette yogurt after a round of focus groups. East Coast brewer Alexander Keith’s recently mothballed a successful, long-running ad campaign after hearing calls for a fresh direction from focus group respondents. “It was pretty overwhelming,” says Rob McCarthy, Keith’s national marketing manager, in explaining why the company did no other research. “In an ideal situation, with all the time and money in the world, we’d love to go back and test again. But at some point, you have to trust your judgment and your experience.” Keith’s runs groups just once a year, but McCarthy considers them essential to the brewery’s direction. “It was amazing to hear consumers say how much they love Keith’s,” he says.

In their enthusiasm at eavesdropping on real live customers, companies tend to forget that focus groups are only as strong as their components. In Canada, there is no mandatory set of qualifications or training regime for moderators to follow. Recruiting companies like Consumer Vision operate at the mercy of respondents, who routinely lie about their experiences and preferences in pursuit of honorariums. Once in session, there is little to prevent the participants from recalling false memories or kowtowing to moderators by under- or overselling opinions. The industry presumes that the large majority of respondents answers questions honestly. In practice, every group runs the risk of placing a sullen, barely responsive participant like Jessie two seats down from a model of virtue like Isabelle. Increased awareness of marketing techniques among consumers means that plenty of respondents supply jargon instead of opinion. Throw in a dedicated cabal of professional respondents — men and women with fake names who live to “focus” — and the painstakingly collected and richly paid-for findings start looking dubious.

Lindsay Meredith, associate dean in Simon Fraser’s faculty of business administration and former chair of the school’s marketing department, suggests thinking of a focus group as a power tool. “There’s no question, the chainsaw is one of the most useful tools I own,” he says. “It’s also the one that will take your leg off the quickest if you get stupid with it.”

CONSUMER VISION is the biggest firm of its kind in Canada. Its offices can accommodate nine simultaneous sessions. The recruiting room runs two full-time shifts of forty telephone operators, even on weekends. Most of the 45,000 respondents in Consumer Vision’s database were obtained through the purchase of phone lists, but I got mixed up with the company by filling out a questionnaire on its website. It’s a fairly innocuous survey, but I wasn’t entirely forthcoming: prompted for title and employer, I entered editor and designer for a trade publication (a commitment that occupies me one out of every eight weeks) but neglected to mention my job as a journalist.

One Saturday afternoon a couple of months later my cellphone rang. The woman on the line said she was a recruiter from Consumer Vision and asked if I was available for a two-hour session. “I just have to ask you a few questions,” she said.

I have no recollection of her first two queries. This was the third: “Are you or is any member of your household or your immediate family employed in market research, marketing, public relations, any media — print, radio or TV — ACTRA or advertising?”

“I’m a freelance writer,” I blurted out, figuring I had eliminated myself from the race before I’d even started running.

“Oh . . . just a minute,” she answered. “I know we can’t have people working in that field.” She left me on hold for ten seconds, then returned and said, “Okay, no problem. Do you smoke, drink or visit casinos?”

She rattled off a dozen questions, mostly about alcohol, then told me to bring my glasses for the television commercials that I may or may not be watching, and hung up.

A couple of calls followed — it’s Consumer Vision policy to double-check its respondents’ answers — and I answered the questions well enough to earn a trip to a Holiday Inn conference room on the northern edge of Toronto. I arrived early, cued the microcassette recorders I had hidden in my pockets, then made for the reception desk. Women in skirts and turtleneck sweaters carrying clipboards moved through the small crowd. One approached me and said, “Quick creative question — if you could be anyone else in the world, who would you be and why?” I gaped. She was serious.

“Uh, am I allowed to say myself?”

“Sure. Why?”

“. . . because I’m happy being me?”

“Okay! Thank you!”

At 3:30, we filed into a conference room. Two women stood huddled around a table laden with VCR and computer equipment. One introduced herself as Moderator A — no, not really, but my hidden recorders caught nothing but hiss — and showed us how to activate the wireless keypads we’d been given outside. Our first task: watch eight television commercials (Gap, Captain Morgan Original Spice Rum, a forgettable floor cleaner, etc.) on a giant video screen, then try to list the names of the company or product being pushed by each. It was a simple knowledge-retention drill, but it didn’t come off as planned. As we started writing our answers, one of the moderators (who would spend the entire hour with her back to us, pecking away at a laptop) rewound the tape without killing the video feed. Fifty-six eyes lifted from page to screen, twenty-eight pens scribbled the lineup in perfect reverse order. A burly guy on my left mumbled, “What a moron,” causing half the room to break up. Neither moderator flinched.

We watched the commercials again, this time using the keypads to score our interest in each on a scale from one to five. We were to enter a value once per second while the commercials ran. Months later, I would find out why: sessions like this are built around one key commercial. After warming the crowd with a few placebos, moderators hone in on the target ad, which will have been broken down “almost to the vowel,” says Paul Lavoie, president of Taxi Advertising and Design. “People are interested in anything they see,” Lavoie explains, “so when the commercial comes on, everyone goes whoosh and peaks real high. Then they all go phshoo, right down, because [the ad] is no good. When I first saw this, I laughed and called it the multiple orgasm. They want you peaking all the way.”

I, however, didn’t peak at all. I spent most of my time drumming a steady one-two-one rhythm into the keypad, particularly during the Captain Morgan spot. Which pretty much killed my shot at the group’s second phase: a discussion. From four o’clock on, we spoke of the Captain and no one else. Moderator A rattled off a list of questions about the commercial — mix equal parts sweat-soaked dancers, mood lighting and keening rock music, add one boat in a field and one silhouette of a pirate against its sail, serve with guttural laugh and brand logo — which we answered by pressing buttons on our keypads. With the hour nearly up, I took my recorders out and flipped the tapes in full view of anyone who bothered to watch. Nobody did. After that, Moderator A read a series of statements (“I could feel the spirit of the Captain throughout this commercial”) and we pressed numbers to indicate our level of agreement.

Then it was over. Moderator B read off eight numbers and the people holding those keypads got to stay for the discussion. Everyone else — including me — got a $40 honorarium (the discussion paid an extra $20) and had to go.

“THE FINNS have the right idea,” writes George Silverman, founder and president of American research firm Market Navigation. “I’m told they hold focus groups of mixed men and women . . . in saunas, naked. Talk about getting them to let it all hang out.”

Other Silverman observations, culled from “How to Get Beneath the Surface in Focus Groups,” one of the many musings about qualitative research on Market Navigation’s website:

  • “When you get into the room, have [respondents] help you rearrange the furniture. . . . You’ve then already formed a group, performed a common task and established an atmosphere of relaxed informality.”
  • “It is a real art to be able to steer people down a particular line of inquiry without influencing the content of their answers. Running a successful focus group without asking any questions at all would be the equivalent of pitching a no-hitter.”
  • “People often say more in one inflection than in tons of words.” Paralinguistics — meaning choice of vocabulary, speed of talking, hesitations, emotional level, et cetera — are as important as words.

According to Silverman, projective techniques, which can be anything from drawing pictures to filling in empty thought bubbles on comics, are a moderator’s bread and butter. “[People] will fill in the blanks by ‘projecting’ their thoughts and feelings onto the stimulus,” he writes. “[Projective techniques] are serious ways of helping people get in touch with and reveal things that they cannot directly access or communicate.”

After my recent focus-group experience, I wanted to know what happens when these techniques are misapplied. “What happens is the same thing that happens when people play psychological parlour games,” Silverman tells me on the phone from his office in Orangeburg, New York. “You say what kind of animal would you like to be, they” — the moderator, the client, whoever — “make a bunch of guesses based upon what you say that may or may not be right. . . . Those things have to be verified. I’m not a big fan of Freud, but even Freud said that sometimes a cigar is just a cigar.”

Simon Fraser’s Lindsay Meredith teaches his marketing students to respect focus group findings only after they have been bolstered by methodological triangulation. “It’s what a surveyor does,” he says. “How does he figure out where your fenceposts are? He has to pick up three other survey points. In marketing, you have all kinds of different statistical measuring tools. You have focus groups, interviews, testimonials, delphi groups — that’s checking the wizards in the industry to see what they feel. You’re getting a lot of data. And you better be hoping that after a while this information keeps taking you back to the same place.” Focus groups, he stresses, “tell you where to start mining, but if you get the wrong information, you dig in the wrong place. You might find what looks like gold, but it usually turns out to be fool’s gold.”

For Taxi’s Paul Lavoie, focus groups’ biggest weakness lies in their tendency to inhibit innovation. “New ideas are hard to get through focus groups,” he says, “especially when consumers are presented with a change in attitude, the reinvention of a brand. They can only recognize what they know.” Some famous focus-group rejections back up his point: the automatic teller (too impersonal), the Seinfeld pilot (lacks a strong supporting cast), the Sony Walkman.

Prior to co-founding Taxi in 1992, Lavoie spent six years as creative director at Cossette Montreal, Quebec’s largest advertising agency. “The French creatives used to pronounce it ‘fuck-us’ groups; that said it all,” he says. Today, Lavoie notes that eighty percent of his clients don’t use qualitative research at all. “[Focus groups] are good if you take them for what they are, which is research and development. Beyond that, they reduce, homogenize and compromise fresh ideas.”

Roche Macaulay’s Marchand agrees that focus groups have inherent pitfalls. “When you’ve got five strangers in a room, you don’t react the way you do when you’re sitting at home with your dog.” Researchers try to get around this problem by observing consumers inside their homes. To improve their odds of getting accurate research, companies are also tightening the criteria on what they run through focus groups. Geoff Craig, Unilever Canada’s marketing director of new business development, says that “historically, we would just dive right into qualitative. Nowadays, before you go into focus groups, you should be developing and testing hypotheses.”

Unilever runs about 100 groups a year, and relies on well-tailored recruiting specs to find the right respondents. “For any project, any brand, any initiative, you have what we would call a bull’s-eye consumer — a woman aged twenty-four to forty-four with kids in the home, that sort of thing,” says Craig. A sophisticated set of questions then probes the respondents’ values and attitudes.

Craig watches most of Unilever’s sessions from behind the one-way mirror. He often runs parallel sessions, having his team brainstorm the same problems the moderator presents to the consumers. “They’ll come up with something, we’ll noodle it about in the background and then send it back into the room.” (At least that’s what is supposed to happen on the other side of the glass. “In the worst case scenario,” Craig says, “it’s chatter and wise-ass commentary.”)

Craig is also careful about the moderators he hires: he has a list of independents in Toronto and Montreal that he would wait two months to work with; there are others whose services he wouldn’t accept for free. That said, Craig knows not to take focus group suggestions as gospel. He recalls a Lipton chicken soup commercial that had its burping finale massaged into a hiccup by finnicky respondents. “If you listen too much, you remove all the risk and arrive at the lowest common denominator,” he says. “It’s a Pablum proposition.”

THE NEXT phone call — the one that got me to Ed — wasn’t pretty. The recruiter had a horrible cold, so when I told her I was a writer, she said, “Are you serious? (sniff) Hang on (sniff) for a second,” and then, “Would you (sniff) like to change your occupation?”

It got better. In my previous interviews with recruiters, the questions were all over the place, so I wasn’t sure what product or service I was being recruited for (otherwise I might have shown the Captain some love back at the Holiday Inn). This time, however, the questions were almost entirely about pizza (which I eat only occasionally) and how often I had it delivered (I almost always eat it in restaurants). I’d seen how far the truth was going to carry me, so when she asked how many times I’d taken delivery in the last month, I answered one instead of none. “I think (sniff),” she said, dropping her voice, “you should say three times. Got it?”

“Right. Three times.”

“Good. Just make sure (sniff) you say that when they call you to confirm. Okay?”


Which brings us back to Bloor Street. Now that we’ve all gotten acquainted, Ed wants to know more about our pizza-eating habits. What makes Dan think of pizza when he watches hockey? Isabelle usually eats alone, so what does she do with the leftovers? Who decides which toppings to order, Regan, Sharon or their kids? My reading had told me there is concern in every focus group about dominant personalities taking over, even swaying other respondents’ opinions, but Ed is careful to give us equal face time.

Next, we identify Pizza Pizza’s competition. Pizza Hut gets mentioned more than anything else, so Ed zeroes in. What are the differences between the companies, he wants to know. Terry compares Pizza Pizza pizza to the cardboard box it comes in; Isabelle glowers. Sharon and Regan point to Pizza Hut’s superior taste and wider variety of toppings; Jessie and I say it’s too greasy, earning smiles from Ed and Isabelle. Ed writes everything down on an easel beside the mirror, then steers us into a couple projective techniques.

“I want you to imagine that you’re at a party,” Ed says. “There’s some good music on the stereo, all your friends are there. The doorbell rings and two people walk in. I want you to imagine that one of these people is the personification of Pizza Pizza and the other is Pizza Hut. Now tell me what these people are like.”

The conversation lasts about ten minutes, but this is the gist of it:

Regan: “Pizza Pizza is a twelve-year-old boy.”

Isabelle: “He’s dressed like Superman!”

Me: “Pizza Hut needs a shower.”

Ed: “T’rif!”

With that settled, Ed opens a manila envelope and spreads dozens of single-page magazine ads, which look to be selling everything under the sun save pizza, around the table. He tells us to choose two. One should represent Pizza Pizza, the other Pizza Hut. He tells us he will step behind the mirror for a moment — “Just to make sure I haven’t missed anything” — and leaves us alone to work. (A smart move, according to Silverman: sometimes respondents will talk to the moderator but not each other. Eliminate the moderator, and the problem usually disappears.) Everyone, even apathetic Jessie, looks lost in concentration.

Ed returns after five minutes and has us pin our ads to a wall-mounted corkboard. All eight Pizza Pizza selections include pictures of children. Ed asks why, and I say I picked mine because any adult who eats Pizza Pizza is probably letting their kid make too many decisions. There’s more variety in the Pizza Hut projections. Terry chooses a photo of two people hugging on a beach (because Pizza Hut is a nice place to go — “It’s a dining experience”). Dan the hockey fan picks a heavily made-up model drinking a beer. I select the head shot of a pig.

And that’s it. Ed thanks us for coming and shows us to the door. He gives Jessie a little smile and tells her he hopes the experience wasn’t too painful; she mumbles an answer and pushes past him to collect her honorarium. On the way down the elevator, I break the news that I’m a writer and tell the group that I’d like to call them later to discuss what just happened. Isabelle, bless her heart, wants to know how much I’m paying. Dan, Sharon, Regan and Jessie oblige and give me their phone numbers. Outside, Jessie tells me she’s spending the $50 we just made on marijuana.

“YOU CAME under false pretenses. This is really upsetting. You came in and you lied to get into a focus group, and that means you’re not honest.”

That’s Phyllis Friedman, Consumer Vision’s general manager. I’ve called her looking to talk about my time with Ed, and just told her all about what I’d done to get into the group. Needless to say, I’m not expecting chocolates on Valentine’s Day. “The group was made inauthentic by you coming in there and not being the right person,” she says. “We try to be so honest, we try to keep to what the client wants 100 percent. They’re spending a lot of money to make sure they’re getting the right people.”

As it turns out, I’m not the only one Friedman should be angry with. When I call Jessie, she says she has a friend who works at Consumer Vision. He filled out the entry questionnaire for her, making sure the company database would find her responses attractive. Sharon, who attended two other focus groups before our evening with Ed, tells me that when recruiters call, she gives the responses she thinks they want to hear. Regan says she’s not much of a pizza eater but somehow made it into the group anyway. Dan says he has lied too, although he doesn’t want to say when or for which of the groups he’d attended.

Friedman directs me to Natalie Gold, an independent moderator who frequently works with Consumer Vision. Gold got her start working the phones as a recruiter, and has been moderating for twenty years. “Oh my god!” Gold says when I tell her about the recruiter’s coaching. “I know this goes on, but if recruiting companies knew about it they would be appalled. People would lose their jobs.” (Turns out she’s probably right. Friedman says the recruiter in question no longer works for the company.)

From Gold’s perspective, it’s not a hopeless scenario if bogus respondents infiltrate a group. “I can tell,” she says. “Anybody who has been around the business long enough has an intuition about when someone is fudging. . . . It’s easiest just to let them be, ignore their comments and not calculate them into your analysis.” That’s probably easier said than done. I thought Ed did an admirable job of managing our group, but I doubt he voided the input from Jessie, Sharon, Regan, Dan and me.

Ed might have caught some of us by using a rescreener in the waiting room before the session. Rescreening “is a relatively new thing,” says Mike Harper, an independent moderator who sits on PMRS’s qualitative research board. Harper’s other safety tactic is to over-recruit. “Of the ten people I recruit, I only want six, so I identify the six who are honest, who seem like decent people who will put an effort in.”

“There’s a certain human nature here that we can’t overcome,” Harper continues. “If people choose to fib a little bit on the phone or have a couple of aliases, which they do, there’s nothing a recruiter can do to stop it completely. Steps are being taken.” Aside from re-screening, there’s now the requirement, made mandatory for all PMRS members last year, for respondents to show photo identification.

The first line of defence, however, is Central Files, a PMRS-supported data management company that pools and compares the respondent databases of more than thirty Canadian research houses (including Consumer Vision). In place since the mid-’80s, the system contains the names of all people in a given geographic area who have exceeded specified attendance restrictions, plus those moderators have fingered as having been disruptive, drunk or otherwise unsuitable. Each year, Central Files flags seven to eight percent of the names it receives as “professional” or undesirable respondents. In industry jargon, the pros are further divided into two categories. Repeaters bounce from one recruiter to another, registering under different names each time, but tend to participate in the discussions in good faith. Cheaters, on the other hand, typically look to focus groups as a principal source of income and will say and do anything to get inside and get the cash. A cheater haunted the Toronto qualitative scene during the late ’90s, arriving at sessions uninvited, hiding behind fake names and disguises, sometimes even switching genders to avoid detection. By the time recruiters caught up to him, he had acquired thirty aliases and fifteen telephone numbers.

Everyone I speak to assures me that the new safeguards are working. Only Robert Hutton, executive vice-president of Consumer Vision and IFOP Westwego, the French marketing company that owns Consumer Vision, allows that dishonest respondents are just part of the game. The best thing he can do, he says, is ensure that his recruiters stick to the script and don’t supply respondents with the desired answers. “We screen and monitor our recruiters,” he tells me, pointing out that they train one-on-one with supervisors and have to follow a manual. “But everyone once in a while, what can I say, it’s going to happen.”

According to a focus group regular, it does. Often. “[Focus groups] are a bunch of crap,” Regan told me when I called her. “For most of the groups I’ve been to, they have asked me to lie and say I used one product more than the other, so they’re not really getting [honest] opinions.”

© 2002, Matthew McKinnon

Photograph by Jasmine / CC By-ND 2.0

    Matthew McKinnon

    Written by

    I used to be a writer. I still am, but used to be, too

    Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
    Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
    Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade