If we try to separate error types in participant responses, we will see the following causes of errors: memory, motivation, communication, and knowledge. The good news, however, is that there are methods to combat these error traps and obtain truthful information.

Let’s go over the causes of such errors and what we can do to recognize and prevent them.

The error of memory

Our memory can be quite fallible sometimes. Chances are any information you ask a participant to recall is no longer sharp in their memory. Activities and experiences may be forgotten, or the time at which they happened may be remembered incorrectly.

Big or important events (like the date of your child’s birth, school graduation, or getting fired from a job) generally tend to be remembered well. So do events that are repeated frequently, like the daily commute to work. Other events, however, may have various levels of sharpness in your memory, and the information may easily become distorted in time.

How to prevent it:

First, make your questions as specific as possible. When interviewing a participant, ask for behavior within an exact time period, rather than generally asking about participants’ usual behavior.

While generally, people think that shorter questions are best (as they reportedly have a higher answer rate), I advise giving some additional context or conditioning to questions that inquire about past events.

Remember that for regular, frequent behavior, respondents will estimate the number of events by using the basic rate they have stored in their memory. The accuracy of these estimates can be improved by asking about exceptions to respondents’ normal behavior.

Secondary records, like checking the bills, the mobile photo gallery, or asking other household members for input will help reduce or eliminate errors caused by memory distortion, as well as possibly provide additional detailed information.

Another method you may utilize is diary studies, an excellent source for collecting qualitative data during a prolonged period of time through participant entries of their activities or experiences.

The error of motivation

Sometimes participants have their own motivations that may contradict their ability to provide you with honest answers to your questions. For opinions that are controversial, somewhat shameful, or otherwise sensitive, participants may consider the truth somewhat too personal to share, fearing the consequences, and thus opting to give false and more neutral replies.

Particularly in the case of in-person interviews, there’s an increased probability that the respondents will try to present themselves to you in a more favorable light.

How to prevent it:

If there is any chance that the respondents may be afraid of other parties overhearing their conversation with you and the consequences it may have on their careers or relationships, it’s best to hold the conversations in a neutral location. For example, an employee might be reluctant to talk to you about his personal life while in the office. Similarly, a teenager might deny ever smoking cigarettes if her parents are present.

Questions that can be labeled as sensitive or polarizing need to be formulated differently to get honest information, even on polarizing topics or socially undesirable behavior experiences.

Note: the book Asking Questions by N. Bradburn, S Sudman, and B. Wasnik has a whole chapter devoted to carefully formulating threatening questions about behavior in a way that keeps the reports un-skewed.

The error of communication

Sometimes respondents do not fully understand what they are being asked and answer the question in terms of their own (somewhat limited) understanding.

How to prevent it:

The same words may mean different things to different people.

Vague words may be interpreted less broadly than the researchers may have intended. Examples of such words include “enough, young people, generally, regularly, anyone, anybody, sometimes, fair, most.” Try to avoid vagueness and be specific about time periods or other categories you’re inquiring about.

Also, avoid any special terms or slang that may be unfamiliar to the user. Use words that virtually all respondents will understand or explain the term in the question (this isn’t always a good idea for all types of interviews). If the sample is homogeneous and most respondents would use the same slang, however, the use may be helpful to aid in the flow of the discussion.

The error of knowledge

Sometimes the user just doesn’t know the answer to what’s being asked of them, yet still answers the question without indicating their lack of knowledge.

How to prevent it:

Respondents will generally give more accurate information about themselves than about relatives, friends, or coworkers. Answers about behaviors and experiences of others can still be valuable to your research but be careful to refer to them as speculations on the participants’ side, not factual data.

The best thing you can do to ensure the participants are answering the question you meant to ask is to probe further and request examples or stories about the topic. Keep in mind that you should be careful not to put the participants in a situation where they feel they are being caught for telling a lie or that they’ve somehow failed a test.

User research can be an invaluable source of knowledge. Acknowledging that humans are flawed and may at times provide faulty information is not a reason to abandon the research practice but rather a call to take the specificities of being human into account when shaping your questions and methods.

4 Reasons why research participants (unconsciously) lie and how to prevent it was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

So, you’ve worked relentlessly on creating your new product or adding the coolest new feature and are impatient to hand it off to development. But are you actually confident that what you’ve designed solves a need, is going to be usable, and have you considered all the flows the users might go through to achieve their goals?

To feel more confident in your design, consider building fake façades, imitations of products that will capture the experience without going through the extensive process it takes to build the real thing just yet. Prototype tests do just that.

Why do it?

✅ To validate the ideas

✅ To minimize costs

✅ To uncover opportunities

✅ To get unbiased feedback

✅ To discover minor bugs

How to do it?

Well, that’s what this article is about!

In my experience the easiest way to dissect a prototype testing experiment is by separating it into three phases:

1. Script writing
2. Preparing files
3. Moderating

On their own, each phase has its best practices and popular pitfalls, but together they form the experience the test participant will have from the session and each will directly impact the results you obtain.

Script writing

1. Test for what’s important, focusing on a few high priority hypotheses

Create a hierarchy of priorities of hypotheses you need to validate foremost before moving forward with product decisions. Especially if you have a complicated product, de-emphasize the actions you’re not prioritizing to test within that research session and focus on topics that will enable you to tick the big questions off the list.

When considering the priorities, remember to get the input of other stakeholders, since there may be hypotheses that the Product Manager, UX Designer, or Engineering Manager may have that you didn’t consider previously but which are critical for overall product development.

Focus on a maximum of 2–3 topics for the research during each session. Jumping from one topic to another will break the storytelling process of the user and might result in missing information that the interviewee would otherwise give outside of just answering the questions or performing tasks.

Also, be careful not to fall for the loophole of having few but very vague goals. It’s one thing to say, “I want to test the prototype,” and completely different to say, “I want to explore how the users navigate through the hospital signup flow when they are in an emergency.” Unfortunately, you won’t be getting far with the first approach.

2. User test VS Usability test

Once you have your goals in mind, it’s important to decide what exactly you’re trying to achieve — gathering qualitative feedback or a list of usability issues and ideas. Most prototype testing sessions I’ve observed have focused on the usability of the prototype, trying to see if the product is usable or not, how participants interact with it, where they get stuck, etc.

Qualitative user research sessions on the other hand would be focusing on the need of the users in the first place — what are the user’s expectations, how do they perceive the product, when do they imagine they might need it.

While both test formats are completely legitimate and useful, it’s important to decide what you want to focus on. Is it to validate the need in the first place or to see if the users can add items to the cart and proceed through checkout?

3. Start with a mindset or scenario and utilize the Jobs To Be Done Framework

Participants must have a certain level of contextual understanding before they jump into testing your product. When will they feel the need for your product? Where will they be? Are there any external or internal influences to be expected? If you’re familiar with the Jobs To Be Done Framework, this could be the perfect time to use a Job — what is the thing the user is trying to achieve in their life when using your product?

4. Ask participants about their expectations

This can be done on a more general or local level. Explore what the participants expect to happen after they click a button or when/how they expect to interact with the product. Questions on expectations are crucial, especially towards the beginning of the conversation before the mental model of the participant adjusts to the product you’ve presented them with.

Some great follow-up questions to a task the participant has performed are to probe, “does this meet your expectations?” and “what did you expect instead?”

5. Dry run

Most of the time, prototypes aren’t as complete with all of the actions users might consider doing during the test. The best way to minimize wasted time re-running interviews and usability sessions with your precious users is to perform a dry run (also known as a pilot test) with other people before the session. The pilot tester can be a colleague that hasn’t worked on your product or even a family member if your product category permits it. The important part is that you keep it as realistic as possible, using the same script as you plan to with the user.

Preparing Files

6. Pick the right fidelity level

The depth of the insights you gather from your prototype test is in direct correlation with the fidelity level of your prototype.

Low fidelity / Paper prototypes

Excellent for: Initial architecture and idea validation

Advantages: Quick, easy, no commitment, easy to modify

Disadvantages: Somewhat unrealistic experience, little visual presentation

Best used for: Small usability tests and initial product feedback

Wireframe or mid-level fidelity prototypes

Excellent for: Getting more product-focused feedback

Advantages: Building blocks can be reused and adapted quickly

Disadvantages: Dummy texts and wireframe components may provide an unrealistic experience

Best used for: Multi-page prototypes with different flows and user subtasks

High fidelity or complex prototypes (including developed product tests)

Excellent for: End-to-end user flow experience feedback that can include feedback on elements of UI

Advantages: Realistic, easy to test

Disadvantages: Hard to modify, drastic changes can be too late to implement

Remember, when building a prototype not EVERYTHING needs to be functional, but rather just enough to have a smooth testing experience to generate data points.

7. Use realistic, non-distracting content / scenarios

Form fields with Batman’s personal information and adding ‘lorem ipsums’ are easy options, but will keep the scenarios unrealistic, taking the concentration off from the product. Avoid these extremes whenever possible and test with content that will help users focus on the task at hand rather than on the distracting made-up content.

8. Don’t forget prototype-specific test factors

Most prototyping tools have a feature that highlights the clickable area whenever the user clicks anywhere on the screen. Obviously, this is not going to be the case when the product goes live and the hints may guide the users where to click to proceed, so you might consider disabling the highlights for a more realistic test.

Will the users be under any time constraints when using your product in real life? Account for that as well or you risk validating a solution that will be inconvenient in the real usage environment.

An example of this may be prototype testing ticket selling machines at train stations. These users are most likely under stress since their train is about to leave, there’s a huge queue behind them, and they’re not performing this type of action daily, so it is important to account for this during testing.

9. Test in the right environment

What is the context in which the users will be interacting with your product? The ideal place would be the actual environment in which the product will be used. Unless you expect the real use to be in a calm environment with no distractions, avoid testing at usability labs as much as possible and opt for more close-to-reality options, like testing at participant’s home, a busy cafe, or even in traffic (ensuring safety first, of course).

Another tip would be to test outside of your company premises, as the excitement of visiting a cool office or the view of a team looking busy working on the product may lead the tester to highlight positive feedback only.

Moderating

10. Informing users

Prototypes are different than final products, obviously. You know it, the Product Manager knows it, but the user might not be as familiar with the concept. It’s best to inform the user that:

• This is a prototype test so some of the functions will be disabled during the session.
• You want them to stick to the scenario, however, provide verbal feedback about anything else that they’d expect to see in the live product.
• In case you have the highlight feature enabled for the prototype, the clickable areas that the user will see will be highlighted. This is not part of the product itself but the testing tool, so remind them it won’t be highlighted when they use the product in real life.

11. Encourage participants to think aloud and share “It would be great if…”

Like with any qualitative research, the point of prototype testing isn’t just to receive answers to the specific questions you ask or tasks you give, but also to learn about the thought process of the testers, their expectations, what mismatches they experience, and more. Ask users to think out loud as they navigate the product and encourage them to share additional feedback and contribute ideas.

12. Reverse the questions addressed to you as a moderator

During any moderated prototype testing session, you will hear questions like, “can I do X with this?”, “why did this page reload?”, “what should I do to get to the next screen?” A LOT. You will naturally be inclined to answer them (at least some), but this is highly inadvisable. To begin with, the participant will stop feeling like the one doing the testing and will feel more like they are being tested, causing them to try to answer all the questions in “the right way”.

The best response you may have to such questions addressed to you is to reverse them and ask the participant what THEY think is going on. Variations of, “what do YOU think this change means?”, “what makes you say that?” and “how would you explain what you should do next?” are here to save you at such times.

Final Thoughts

In his book, Zen and the Art of Motorcycle Maintenance, Robert Pirsig has a quote that 50 years later, stays true, no matter the field of study:

“An experiment is never a failure solely because it fails to achieve predicted results. An experiment is a failure only when it also fails adequately to test the hypothesis in question or when the data it produces don’t prove anything one way or another.”

Similarly, the point of prototype testing shouldn’t be confirming what you expect and moving forward, but testing hypotheses and gathering data that will enable you to make informed data-driven decisions. Most of the time this will include iterating, iterating, and iterating again until you get a product version that has just the right balance of everything.

Prototype testing — the complete guide was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article will tell you about 3 survey types that you can carry out to measure your customers’ satisfaction and see your progress quarter by quarter.

Organizing and conducting impactful user experience (UX) research is often harder than it seems — there’s always a backlog of tasks marked “urgent”, dozens of usability tests, preference tests, competitor researches, and many more.While these types of research are definitely important, they are not giving you a full picture of where your product and your brand are at the moment, as well as how the changes in design or functionality are affecting the user perception and experience.

To learn how your product is performing, you need to carry out repetitive, identical research that will monitor the growth or decline of user satisfaction through time and changes.

By conducting and analyzing UX research quarterly, you will be able to track overall user satisfaction and compare how your efforts have been affecting their perception of the product, brand, and specific features.

Why do this quarterly?

Well, if your product is evolving and your designs are changing periodically, you NEED to have a clear understanding of how satisfied your current users are with the product in general and its specific parts. Moreover it is imperative to be aware of how well your brand is being perceived and how easy it is to interact with. Without this, you might struggle to successfully cater to your users’ specific needs.

1. User Satisfaction Survey

Also known as Customer or Client Satisfaction Survey (CSAT), this type of survey will help you measure how satisfied your users are with your brand, its services or features.

Knowing which areas your users are most satisfied with, as well as what they find worthy of improvement, will help you narrow down your list of priorities and decide on potential improvements to make.

How to create a User Satisfaction Survey

A typical Satisfaction question would be along the lines of: “How satisfied are you with the product/service you used?” with a rating scale from “Completely Dissatisfied” to “Completely Satisfied”. The rating scale will help you compare the satisfaction level of today with that of last quarter.

Another good question to ask your users is “What do you like/dislike the most about our product?” This will be a good opportunity for them to describe what irritates or delights them the most about your product.

2. Net Promoter Score (NPS) Survey

The Net Promoter Score helps you understand how loyal your customers are to your company. It also allows you to identify which user groups are less likely to recommend your product. This valuable information will come in handy when you make future marketing and product decisions.

The main objective of this survey is to divide your users into three main categories:

1. Detractors (dissatisfied users who will discourage other people from using your product);

2. Passives (who will neither bring nor discourage future users);

3. Promoters (satisfied users who will bring in new users by being your advocates).

With NPS, you can measure the satisfaction and praise of your customers on multiple levels — from overall product experience to individual features or actions taken.

How to create a Net Promoter Score Survey

The NPS survey is probably the most hassle-free way you can go about measuring customer loyalty.

The main question is “How likely is it that you would recommend this product to a friend or colleague?” with a scale from 0 to 10 as quick answer options.

People who rated the product from 0–6 are the Detractors — they are the unhappy, dissatisfied customers that will most probably quit using your product, and may even discourage others from trying it.

Those who rated 7–8 are the Passives that are satisfied but are not brand advocates. These are the users that may switch to another brand if it offers better quality or pricing.

Finally, those giving a 9–10 are the Promoters — the satisfied, enthusiastic users that will stick with your brand as long as they can, and will also encourage their friends or colleagues to give your product a try.

The NPS is measured on a scale from -100 to +100, with higher scores meaning more loyalty.

To calculate the NPS, subtract the percentage of Detractors from the percentage of Promoters, while ignoring the Passives.

NPS = %Promoters — %Detractors.

If you only want to measure your user satisfaction score, one question should be enough. However, if you are eager to understand the reasoning behind the score, you can set up some additional, detail-specific questions. It is important to note that follow up questions will not be directly a part of your NPS score, however, they will give you the insight you need to enhance your product. One possible follow-up question may be “How can we make your experience better?”

You can also ask a demographic question about the user’s age, gender, location, and more — but do so only if you’ll be using the information to interpret the score and if it will be valuable to you.

3. System Usability Score (SUS) Survey

The System Usability Scale provides a quick, easy to analyze and reliable information about the usability of your product.

How to create a System Usability Score Survey

As outlined by John Brooke in 1986, it consists of a 10 item questionnaire with five response options for respondents; from Completely agree to Completely disagree.

Here’s the list of questions that your interviewees/testers need to agree or disagree to:

1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.

Interpreting the scores can be a bit tricky. So here’s a detailed step guide:

1. Convert the scores to 1–4 points (with Strongly Disagree = 1 point, and Strongly Agree = 4 points) and add the scores.
2. Multiply the sum by 2.5 to move from a 40-point scale to a 100-point scale.
3. Compare your score. Generally, a score above a 68 would be considered above average and anything below 68 is below average.

Putting it all together

Carrying out consistent research quarterly will give you a better idea of your users and the way they feel about your product. This valuable information will eventually pave the path to success for your product.

3 Surveys to do quarterly for ux monitoring was originally published in UX Planet on Medium, where people are continuing the conversation by highlighting and responding to this story.