How to Create Effective Research Surveys

Part Two: The Layout

By Alexie Touchette

In Part One: The Questionnaire, we discussed tips for writing effective survey questions. Today, we’ll touch on how to design an effective survey layout, which can make it easier for respondents to understand and answer your survey questions — and in turn, reduce the number of misread questions, skipped pages and non-responses.

The majority of these tips have been summarized from Dillman’s Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th Ed [2].

1. CONSIDER THE ORDER OF YOUR QUESTIONS

1.1 Strike a Conversational Flow

The order of questions should simulate the logical flow of conversation, and should be grouped together by topic [3]. Jumping from one topic to another can give respondents the impression that the questioner is not listening to their answers. Further, responses are less likely to be well thought-out when the topic is constantly changing.

1.2 Logic is Key

Overall, questions should be placed in a logical or sequential order. For example, ask about things in the order that they happened, or ask descriptive questions about an activity before asking about an evaluation of the experience.

1.3 Order by Importance…

Where possible, Dillman [2] suggests ordering questions from most to least important. Having the most important questions first helps meet respondents’ expectations of what the questionnaire will be about. For example, a questionnaire which starts with a series of demographic questions (i.e., age, gender, job title, nationality, etc.) might help meet the study’s objectives and might be easier to answer, but they do not address the study topic.

1.4 …But Save the Tough Stuff for Last

With that being said, the first page of the questionnaire should not immediately jump into abstract or attitudinal questions, as these are harder to answer and might lead to respondents skipping questions, or might even scare them away from the questionnaire altogether. Questions that tend to make respondents uncomfortable (such as questions regarding income, sexual behaviour, religious activities, or criminal behaviour) should only be seen after the respondent has become interested and invested. Pilot testing your questions can help you identify which questions seem to be the tough ones.

1.5 Carryover & Subtraction Effects

Though beyond the scope of this post, order effects such as carryover and subtraction are important considerations for question ordering (for further reading, refer to [3]). Don’t assume questionnaires are simply a series of independent questions that have no effect on one another. Each question needs to be evaluated for its own individual content and within the larger survey context.

2. MAKE THE CORRECT PATH OBVIOUS & EASY TO FOLLOW

2.1 Format questions so respondents only have to read them once

Example:

Poor layout:

☐ Very satisfied
☐ Somewhat satisfied
☐ Not at all satisfied

Please check the appropriate category above to indicate whether you were: Very satisfied, Somewhat satisfied, or Not at all satisfied with the quality of your meal the last time you visited our restaurant.

The problem: The answer choices and instructions are provided before the question, which requires respondents to reread the question and retrace their steps.

Better layout:

The last time you visited our restaurant, how satisfied were you with the quality of your meal? (Check one box)

☐ Very satisfied
☐ Somewhat satisfied
☐ Not at all satisfied

2.2 Place instructions exactly where the information is needed

Place special instructions inside of questions. Optional or occasionally needed instructions can be separated from the question statement by font or symbol variation.

Example:

Poor layout:

Thank you for taking the time to complete this questionnaire. The directions for filling it out are provided with each question. Because not all questions apply to everyone, you may be asked to skip certain questions.

- In order to get comparable data, we will ask you to refer to the week of April 15, 1993, when answering most questions.
- If a ‘skip’ instruction is provided, move to the next question.
- Either a pen or pencil may be used.
- When answering questions that require checking a box, please use a “X”.
- If you need to change an answer, please make sure your old answer is either completely erased or clearly crossed out.

The problem: The instructions are too detailed and not immediately relevant. Instructions regarding time frame and skipping questions when certain responses are given (also called ‘skip logic’) shouldn’t be provided until they are actually needed.

Better layout:

Question 7: Were you working for pay (or profit) during the week of April 15, 1993? This includes being self-employed or temporarily absent from a job (e.g. vacation, illness, or parental leave), even if unpaid. Mark your answer in a box with a ‘X’ using pen or pencil.

☐ Yes (SKIP to question 8)
☐ No

7. a) (If no) Did you look for work at any time during the four weeks preceding April 15, 1993?

☐ Yes
☐ No

2.3 Ask one question at a time

By asking only one question at a time, you can better control how and when respondents answer each question — this includes avoiding the use of complex matrices (asking multiple questions about multiple variables at the same time).

Example:

Poor layout:

7. Please indicate the extent to which each of the following services of our organization are important to you, and for those that are very important, please indicate how frequently you have used that service during the past six months.

The problem: Having to answer two questions in one requires survey takers to toggle between questions, which can make it hard to remember exactly what is supposed to be done for each question.

Better layout:

7. How often have you used each of these services from our organization during the last six months?

Customer credit (number of times used): _____
Next-day delivery (number of times used):_____
Personalized offers (number of times used):_____

8. Thinking about each of these same services, how important is each of them to you? Circle one answer for each service.

Customer credit (level of importance): Not — Somewhat — Very
Next-day delivery (level of importance): Not — Somewhat — Very
Personalized offers (level of importance): Not — Somewhat — Very

3. USE CONSISTENT VISUAL NAVIGATION CUES

3.1 Increase size and visual impact of written elements to attract attention

This includes changes to font size, font type, bolding, italicization, and the use of colours to attract and focus respondents’ attention to where and when you want them to look.

Example:

Poor layout:

The problem: By bolding the middle of the question, the reader’s attention is drawn away from where they should be starting.

Better layout:

3.2 Use empty space to group information

This is pretty straightforward — information that is grouped together is read together, and information spaced separately is read separately.

Example:

Poor layout:

17. Thinking about the last time you were enrolled in classes, were you primarily interested in (mark one answer): ☐ Obtaining a degree or certificate; ☐ Learning a new skill; or ☐ Both?

The problem: The example above combines the questions and answers into one paragraph — spacing would help improve clarity and avoid respondents missing any information.

Better layout:

17. Thinking about the last time you were enrolled in classes, were you primarily interested in (mark one answer):

☐ Obtaining a degree or certificate
☐ Learning a new skill
☐ Both

3.3 Clearly identify the beginning of each successive question

Use a simple and easy to follow numbering scheme that introduces questions consecutively from beginning to end.

Example:

Poor layout:

Section 1: A, B, C, D and Section 2: A, B, C, D or IA1a, IA1b, IA2, IA3a, etc.

The problem: Restarting questionnaires for different sections is common, but can be confusing and discourages the use of skip logic. Numbering should be simple, consecutive, and consistent throughout.

Better layout:

1, 2, 3, 4… to the end of the questionnaire

3.4 List answer choices vertically

Provide answer choices in columns instead of rows, and avoid double and triple answer banks as these can result in respondents missing potential answers.

Example:

Poor layout:

The problem: Placing the choices horizontally puts the responses on more than one line, and makes a linear connection more difficult. Additionally, the second or third column of response options may be missed by respondents who are used to looking for just one column of options. A better alternative would be to list all choices vertically in a single line.

Better layout:

3.5 Maintain simplicity, consistency & symmetry

Example:

Poor layout:

The problem: The lack of simplicity, regularity, and symmetry leads to confusion and frustration. Keep number format, font size and format, spacing, answer choice boxes (format and location), and use of space consistent and aligned with the tips above.

Better layout:

4. Use additional visual cues to interrupt established navigational behaviour and to redirect respondents

This step is used intentionally to disrupt the pattern of reading and responding established by following the above strategies. This is useful when there is a critical change in the questionnaire (such as a change in time reference from the past month to the past year) that could otherwise be missed or ignored by respondents.

4.1 Use formatting to properly direct skip logic

Note that this strategy speaks predominately to paper formats, as most online survey services have a skip logic option, which automatically directs respondents to the most appropriate next question based on their previous answers.

Example:

Poor layout:

The problem: The distinction between “go to 24” and “go to 28” is too small to assure compliance. Respondents who are used to the routine of reading a question, selecting an answer, and beginning the next question might unintentionally miss the skip logic instructions. Use different visual cues to direct respondents to the appropriate next question.

Better layout(s):

Option A: addition of four visual elements: (1) differently shaped directional arrows; (2) word changes; (3) larger font; and (4) redundant instruction to define skip pattern.

Option B: addition of alternative visual elements, i.e. change in location of response boxes with spacing change to define the skip pattern.

4.2 Highlight changes

Words or phrases that introduce important (but easy to miss) changes in respondent expectations should be visually emphasized with consistency, but done so sparingly.

Example:

Poor layout:

The problem: When wanting to introduce a new important clarification, words should be emphasized in the same manner (not in three different ways like the example above).

Better (but still poor) layout:

The problem: Although this example uses emphasis in the same manner (i.e., underlined), too much of the sentence is emphasized. This can lead respondents to focus only on the words that are emphasized and not read the whole question.

Better layout:

As with question design, survey layout considerations have a significant impact on responses. Ensuring that the questionnaire appears interesting and important, is logically organized and easy to navigate, and does not contain any errors is key to optimizing your questionnaire. Although these strategies may help to create a better survey, there is no substitute for evaluating and pilot testing your survey.

References

1. De Leeuw E, et al. International handbook of survey methodology. New York: L. Erlbaum Associates, 2008.
2. Dillman DA, Smyth JD, Christian LM. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th Ed. Hoboken, NJ: Wiley, 2014.

About the Author

Alexie Touchette is a Masters student in the Department of Community Health Sciences at the University of Manitoba.