Designing a Quantitative Questionnaire: 5 Things You Should Always Do!

Questionnaire design is both an art and a science. It entails much more than creating a “tossed salad” of questions and throwing them into a document. Designing market research questionnaires requires the researcher to recognize what an effective rhythm for a survey “feels like”; it requires the ability to balance the objectives of a client — who may want to try to squeeze as many questions and topics possible into the instrument — with the realities of yielding reliable, projectable data.

Although there are few “absolutes” in life, some do exist. So here are a few guidelines to help you recognize some of the things that you should virtually ALWAYS do when designing a quantitative questionnaire

5 things you should always do when designing a questionnaire:

1. Word each question in a style geared to your respondent audience.

Just as you would likely adjust your conversational style if you were speaking with your cardiologist versus your child’s football coach, you should be mindful of how you address your market research audience, as well. You would likely use a somewhat more formal style, for example, if you were surveying oncologists about a soon-to-be-launched treatment for breast cancer than if you were seeking to collect data about juice preferences from parents of toddlers. That’s not to say that you would communicate in a less respectful or professional way with either group… just that you would adjust the specific wording and “tone” that you use accordingly.

2. Design screening questions that will capture a fresh pool of qualified respondents.

Many potential research participants are, unfortunately, fairly savvy about our screening techniques by now; as such, it’s not uncommon to see the same respondent names popping up over and over. And over. The better you’re able to camouflage both participation and disqualification criteria, the better your pool of actual respondents (and the more reliable your data) will be. Including too many “frequent flyer” respondents in your sample can result in stale, skewed perspectives that don’t provide clients with the real-world insights they need. So instead of recycling the exact same five or ten screening questions for every study, try to change the wording of the questions or the prelisted response options you provide. If possible, ask a simple question or two about a topic that’s unrelated to the actual research you’re conducting; it will be more difficult for respondents to hone in on the focus of your study and second-guess how they can qualify.

3. Use a design approach that’s compatible with how your survey will be administered.

Ideally, those of us who design questionnaires should become at least somewhat conversant with the market research survey software used to program those questionnaires. Having a basic level of familiarity with the capabilities and limitations of your survey data collection software will enable you to structure questions in a manner that the software is designed to most efficiently and “elegantly” accommodate. Similarly, if you understand the capabilities and limitations of the delivery mechanism/medium being used — telephone pen-and-paper versus online versus mobile phone versus CATI or CAPI, for example — then you can also be mindful of the specific needs of that specific medium as you design your questionnaire. For example, you would not want to ask respondents to provide detailed feedback on visual stimuli (e.g., several potential package designs) if you are using a CATI (computer-assisted telephone interviewing) data collection method.

4. Confirm that the questionnaire is programmed correctly. Then, check it again. And, yet again.

In market research’s halcyon days — as recently as just 25 or 30 years ago — most quantitative surveys were conducted by an interviewer, in real-time. Today, however, studies — whether conducted online or via mobile phone, CATI, or CAPI — are primarily administered by way of proprietary market research survey software. The reliability of the data hinges on the quality of the instrument programming, so making sure that these surveys are programmed perfectly is of paramount importance.

• Pre-launch testing: Once programming is complete, it’s critical to have internal research staff test your survey on whichever medium will actually be used in the field. During these run-throughs, not only will all of the text be proofed against final hard-copy versions, but the skip logic will be verified and the technical aspects of the survey — from sliders used to designate responses to ratings questions, to the appearance of embedded photos and illustrations, to cumulative tallies of responses that must sum to 100% — will be checked.
• Small out-go in field: Once internal QC’ing is complete and all necessary revisions have been made, then fielding may begin with a “small out-go” (also called a “soft launch”) across quota groups. Fielding during a small out-go is based upon a limited amount of respondent sample, and designed to test all aspects of live fielding before the study is rolled out in earnest. This cycle of proof-revise-proof-revise may seem like overkill, but it’s always better to catch any errors prior to a full launch than to have to discard a slew of completed interviews because there are programming glitches.

5. Include open-ended questions to gain additional insight and nuance into the reasons behind perceptions and behavior.

Using quantitative data culled from closed-ended questions is an excellent way of making clear, “apples-to-apples” comparisons between various data points. However, sometimes a sea of percentages and statistics can lack the “beating heart” added by more descriptive responses. Open-ended questions are an extremely effective way to add more of this beating-heart quality to quantitative research. Open-ends are best used as a follow-up to key questions … those questions that, together, create the “story” leading to the resolution of the client’s study objectives. These open-ends allow us to find out WHY respondents within our client’s target market purchase specific brands of orange juice, WHY they’re likely to prescribe a new anti-anxiety medication, or WHY they perceive a Congressional candidate positively or negatively. From timing, staffing, and cost vantage points, open-ends can be a drain on resources, so it’s prudent to use a limited number of these questions where they’ll provide the most valuable insights — for example, following questions that ask for overall satisfaction ratings, attribute preferences, or rankings of brands by purchase likelihood.

Now that we’ve talked about how to design a quantitative questionnaire, we should talk about what to consider when setting up an email campaign to reach the most respondents. Download our white paper now!

Loved what you just read?Check out more similar blog posts here