Designing a better survey
Last time we did a deep dive into a survey, but this week I wanted to come back with some basic things that can help improve your future surveys.
1 — Ask questions if you only want the answer (or include constraints or context)
This is probably something that seems obvious when you say it, but it’s a trap that I fall into from time to time. Most recently when our office ran the Teaching Community Conference I surveyed faculty about the timing of the event and whether there was a better time for it, but our committee knows that certain periods aren’t on the table (e.g. because of room booking constraints, when faculty are on campus, etc). Because we know some boundaries of the best window we have, I’ve set up the survey to briefly state our window and ask faculty to let us know about conflicting events that we should be aware of.
By honing this question it makes it easier for us to get useful information but also explains why the date of the conference may not change much and prevent some faculty from feeling like we’re just ignoring them.
2– Know what your answers mean
Perhaps one of the most famous internet surveys is the Political Compass. The survey is a poll consisting of a 4 point (or forced) Likert scale. The scale forces people to take positions on questions, but that might mischaracterize their responses. One question where this can be problematic is: A significant advantage of a one-party state is that it avoids all the arguments that delay progress in a democratic political system. If someone taking this survey believes that that is a fact of one-party states, but believes that arguments are valuable might agree with the statement but disagree with sentiment and must try and discern the author’s interpretation of the results.
3– Frame the question to give participants an out
Criticism can be hard for people to give, just as it can be hard to receive. I learned this strategy from a favourite rapper of mine, Dessa. When she has a series of songs and is readying an album and is trying to decide what stays and what goes, she’ll go to her friends, family, and colleagues and ask. But instead of asking her reviewers which songs they like or don’t, especially since music is such a vulnerable experience. So to let her friends not say they don’t like a song, she asks them which ones are must keeps for the album and since space is limited, which songs they would leave out. Likewise you could gauge response to course topics, activities, etc.
So hopefully this information, as well as our look at that previous survey provide some help in crafting your future surveys! And as always, if you need support our office is here to help!
