The Ultimate UX Survey Design Guide

How to design, iterate, and analyze impactful user surveys

I recently did a deep-dive to better understand and use surveys. Gathering meaningful data with a set list of questions requires some careful crafting.

I’ll walk you through a survey analysis that I used to validate qualitative findings. You will see my process, survey checklist, results, and ethics.

Reference: I used the survey for a project on the Calm mobile app to explain my survey design process. I hosted the survey on Google Forms and recruited participants with tailored posts on various Social Media and Slack channels.

All on with a $0 budget :)

By the way, surveys are one of the most accessible quantitative methodologies. Check out Norman Nielson’s article on quantitative research for a breakdown of the most popular methods, includes information about purpose, cost, difficulty of collection and analysis, and whether its more attitudinal or behavioral.

Step 1: Determine Research Objectives

Surveys are often used as a quantitative research tool to validate qualitative findings. For example, you might start a discovery research project with in-depth user interviews and then dive deeper into user themes with a survey.

For this project, I conducted seven in-person interviews. I focused on these key topics: meditation, habit building, and meditation apps.

Pairing Qualitative and Quantitative Research

Key User Themes (Qualitative)

  1. Meditation is a long-term relationship for most people with phases of consistency and inconsistency.
  2. People who meditate have described it as “game-changing” and often recommend it to friends and families.
  3. However, the struggle to build and maintain a meditation habit was a consistent pain point.

Research Objectives

I chose to dig deeper into these research questions to help users build the meditation habit and to help the meditation app increase daily user rates.

When and why do people start meditating?
Why do people stop meditating?
What other apps behaviors have been more consistent?
What are people’s mental models about meditation as a practice?

Step 2: Make sure you get the right participants

Finding the right people to take your survey will greatly impact your results. If you already have a list of users, great! If not, start defining potential users. Write out the key topics around product usage. Consider necessary psychographic qualities that your users may have.

As a rule of thumb, it’s good to have at least two screener questions: a task knowledge question and an internet knowledge question.

For example, for the Calm app meditation survey, I used these screeners:

Do you have experience with a meditation practice?
Have you checked out one of the apps?

Step 3: Write the First Survey Draft

Reference: Chris Gray, UX Mastery, “Better User Research Through Surveys

Iterating and testing the survey before sending to a larger sample size helps you to achieve desired results. Before showing the survey to anyone, I created a survey checklist from secondary research. Here are a few to keep in mind:

Survey Design

  • Did you group similar questions?
  • Are the questions appropriate for the audience?
  • Are the questions easy to understand?
  • Did you avoid questions with two concepts?
  • Are rating scales balanced?
  • Do you include “don’t know” options when relevant?
  • Did you practice this survey before sharing it? (ideally with a smaller sample size?)
  • Did you keep your participants in mind throughout the whole survey design?
  • Did I avoid double negatives?
  • Have I used open-ended questions?
  • Have I asked why?

Secondary References

Okay, now maybe you’re ready to share with someone. First I asked a UX designer friend to take the survey and provide some initial feedback.

Step 4: Iterate the Survey. Test it. Iterate Again.

After gaining initial feedback, I made some adjustments in the question order and reviewed my research goals to be sure it lined up. Then, I decided to test the survey with a slightly larger sample size:

Testing Round 2: Recruiting Participants

I found seven people who fit the criteria via social media and Slack group recruiting. I made sure the post hit on these points:

  • Provide context of the study
  • Include the purpose of findings
  • Ask screener questions
  • Consider possible incentives

I received positive feedback from this Slack message:

I used similar criteria when crafting the survey intro, just a bit more formal.

Testing Round 3: Widening the Sample Size

Those responses, especially from open-ended questions, helped to narrow the survey focus. After additional iterations, I reached out to a larger sample size.

The survey contained 14 questions, and I recruited 30 participants.

Step 5: Evaluate Survey Results

After all that work, getting real feedback from your survey is so exciting!

Start with the simpler results and see if they validate or invalidate your initial hypothesis. For example, I was curious about users’ key motivations for initiating a meditation practice. I was curious about how often work performance and physical health were prioritized. But I assumed mental health would be a key factor. Well, according to 30 responses:

A desire for better mental health definitely tops the list. This can be really helpful when assessing which features to implement. Additionally, having quantitative feedback to give your team will give you credibility in your design recommendations.

Quick Note on Survey Ethics

Develop a sense of trust with your users and with your clients or stakeholders by always using quality ethics during your surveys. Here are a few helpful checklist questions to ask yourself when evaluating your survey:

  • Did you explain the purpose of your research?
  • Did you mention who you are doing the research for?
  • Did you share how you will use the results?
  • Are the questions appropriate for the audience?
  • Do the benefits of the research outweigh potential risks?
  • Be mindful when communicating the results of your study
  • Have you been sensitive of the subject matter? (i.e. include something like, “There are no wrong answers.”)