Random Reminders for Survey Research

“Research-speak” neither describes nor improves reality.

Photo by Jon Tyson on Unsplash

• Don’t use scales that start at 1 — life starts at zero / none/ zip/ nada.
• Likewise, no fives or sevens, please — who uses these in real life? Do we talk about “a perfect seven?”
• Don’t use numeric scale points at all if you don’t need to (most of the time, you’d be better off with a categorical scale unless you are hypothesis testing)
• Don’t use semantic differentials like Somewhat Agree / Agree/ Strongly Agree, that a respondent will spend more time parsing than you will ever spend acting on. Most likely, you are looking for a 0–10 scale, or a five-point categorical scale that begins with “Completely Disagree.”
• Offhand, I can’t think of any reason to have words like “satisfaction,” “preference” and other abstractions as the subject of a question. When did you last use these IRL?
• Before you offer up such a question, ask yourself: What do I really need to know about what the respondent believes, knows, thinks s/he has experienced, etc.? Then write the actual question, using frequency, duration, intensity or some other real-life construct in natural language.
• Often, “about half the time” or “50/50” are perfectly sound midpoints.

You should be seeing a pattern here. For a respondent’s answer to be worth something, the question preceding it needs to be naturalistic, easy to answer, precise, unequivocal, single-topic and probably feature at most one adjective or adverb.

GIGO.

Above all, represent your respondents. If management is drawing a fallacious conclusion from “your data,” don’t be a doormat. Represent the truth until you are told straight out that the truth is not of interest. Then start looking for another job.

Originally published at https://choiceandchance.typepad.com.