How to Write Unbiased Poll Questions
When conducting a PickFu poll, one of the biggest benefits is accessing an audience of people who have no familiarity with your product, logo, book, or whatever it is you’re testing. They approach the question without bias… but as the poll creator, do you?
Avoid these common mistakes and poll respondents will answer openly and honestly.
Mistake 1. Leading words
Your question may include a positive or negative bias — words that consciously or unconsciously lead the respondents toward a certain kind of answer.
• How much did you enjoy this YouTube video? (positive bias — implies that the viewer enjoyed it, and leads respondents to answer more favorably)
• Should responsible parents vaccinate their children? (puts respondents on the defensive by insinuating that parents who do not vaccinate are irresponsible)
• A recent poll found 80% of Americans disagreed with this government policy. How unhappy are you about this policy? (negative bias — not only is the question phrased negatively, it also includes a statistic that shows many unfavorable views, leading the respondent to feel as though he or she should feel that way, too.)
How to fix it
Remove leading words and phrases and structure the questions as objectively as you can:
• On a scale of 1–5, with 1 being the worst and 5 being the best, please rate this YouTube video.
• Do you think children’s vaccinations should be required?
• Please rate your level of agreement with this government policy.
Remember, biased questions lead to biased results. And if you’re using polling to make important business decisions, you want your feedback to be as objective as it can be.
Mistake 2. Making assumptions
Sometimes questions include an opinion. For example, “Do you agree that dog owners should be able to walk their dogs off-leash in the park?” This assumes that dog owners want to walk their dogs without leashes, which may or may not be true. But by including this assumption, you are leading more people to respond in agreement.
• Is your favorite color blue? (assumes the respondent’s favorite color is blue)
• As long as nobody minds, is it okay to smoke indoors? (assumes that nobody minds)
• Where do you like to party? (assumes respondents like to party)
How to fix it
Structure your questions around facts, not opinions. Make sure you’re not painting respondents into a corner so that they can only answer one way. In the first example, it would be better to ask, “Do you think dog owners should be allowed to walk their dogs off-leash in the park?” Similarly, these questions eliminate assumptions:
• What color do you like the best?
• How do you feel about smoking indoors?
• What do you like to do in the evenings?
Mistake 3. The double-barreled question
Sometimes questions ask about two disparate things. These double-barreled questions do not make good survey questions because respondents will often only concentrate on the one topic that means the most to them.
• How do you feel about our two new flavors, French Vanilla and Hazelnut Cream?
• How have teachers and students responded to the new dress code?
• Which title and subtitle do you prefer?
How to fix it
Double-barreled questions should be separated into two distinct questions and polled separately from one another. When questions are worded similarly in a multi-question poll, it’s also good to distinguish them with underlined words or italics, as below:
• How do you feel about our new flavor French Vanilla?
• How do you feel about our new flavor Hazelnut Cream?
• How have teachers responded to the new dress code?
• How have students responded to the new dress code?
• Which title do you prefer?
• Which subtitle do you prefer?
Creating better PickFu surveys
PickFu focuses on preference testing. This means that in general, you’re testing options against one another, not necessarily open-ended or yes/no questions. Therefore, a question like “how do you feel about smoking indoors?” or “How have teachers responded to the new dress code?” likely wouldn’t get asked through a PickFu poll.
That being said, the same rules apply to preference testing as in general surveying: avoid leading words, don’t make assumptions, and only ask about one thing at a time.
For example, if you’re testing mobile app icons, the simplest unbiased question would be, “Which app icon do you prefer?” To embellish the question with anything else may introduce bias. Sometimes pollsters want to know whether their icons read a certain way, so they’ll ask something like, “Which icon seems more kid-friendly?” While not biased, per se, this question asks respondents to judge your icons differently than they might otherwise.
Because PickFu respondents include comments, one approach to avoid any hint of bias is to ask the simple question, “Which app icon do you prefer?” and then read the comments to see if kid-friendliness is something the respondents took away by themselves, rather than being led to see kid-friendliness or not.
Another approach is to include factual statements (unbiased, of course!) to explain what you’re testing, and then ask the respondent’s preference. For example, “Our app is a game aimed at children under 10. Which icon do you prefer?” These approaches may all yield various results, so some experimentation might be necessary to see what’s most applicable in your case.
Keep it consistent
Consistency is key in testing creative options. If you’re testing book covers, make sure that each design option includes the same information. If you tested two cover designs and each cover had a different title, for example, you wouldn’t know whether respondents preferred the design layout or the title on that design layout. Remember, only test one thing at a time.
Below is an example of an author who tested two covers. However, the two titles are different, only one option included a subtitle, and only one option included the superfitdads logo. Any of these variances may have skewed the results.
This poll would be less biased if each cover included the same title, subtitle, and author attribution. Then, the only test parameter would have been the cover design.
Here’s a better example. This author is testing two cover designs. The title, subtitle, author attribution, and even graphic layout is the same. The only difference is the “Action Plan” stamp. By only testing one thing at one time, this author knows the stamp helps his cover.
In both general polling and preference testing, it is important to keep these main ideas in mind as you design your survey:
1. Avoid leading words that may sway the responses positively or negatively. Phrase your question objectively.
2. Ensure your questions are fact-based, not opinion-based. Do not make assumptions about your audience.
3. Test only one thing at a time. Do not use double-barreled questions. When creating comparisons between two creative options, include only one test parameter per question, such as layout, title, or color palette.
4. You may need to experiment with the level of specificity in your question. Generally, the simplest form of the question will be the least biased. However, there are instances where you will need to direct the question a certain way rather than leave it open-ended. Just be sure to do it according to the three tips above.
Originally published at The PickFu Blog.