How to recreate a bad customer experience

A product manager’s guide to understanding problems in an existing product

A customer reaches out to you about a frustrating experience. It’s never a good thing, but it happens. How do you take action from this type of open-ended feedback (email, Facebook post, customer service chat, etc.) to improve the experience? How do you identify potential fixes that don’t feel like guesses?

Many organizations have all sorts of tools that can help a product manager explore data. This data can help tell a story about what is going on, but sometimes it can feel like you’re just poking around, making decisions based off the first signal you find. Conversion rate is down! I think it’s in step 2 of the funnel! Run tests — stat!

A giant list of opinions

If you work with other people who are doing the same thing then you can end up creating what I affectionately call the giant list of opinions. As a product manager, a lot of times these opinions come from a variety of your key stakeholders. You can’t dismiss these opinions, but you can’t let them be the main guiding force for improving the experience either.

Many organizations have a lot of resources and tools. They also have a lot of opinions.

It’s a slippery slope. Some of these opinions on what to do may have a positive impact. But how do you know if these items will make a difference at all? How can you reduce the risk of these assumptions? What will you learn to help you understand what to do next?

Rushing to the solution

The real risk in the giant list of opinions is it rushes you to delivery instead of trying to understand the problem. It gives the illusion of progress: you seem to have enough information to understand what to do and therefore you can move towards taking action. Perhaps this is the way your organization operates — valuing delivery over continuous learning. Let’s see what this looks like when compared to the Lean UX Process.

The Lean UX Process as found in the 2nd edition Lean UX book by Jeff Gothelf. Comments added to frame our problem. Since the product in our example is in the market already, we should be starting at the Research & Learning step.

So what do you do?

The following approach is for the product manager who has struggled to confidently find what to do after discovering a problem— the details behind the “Research & Learning” step in the Lean UX Process. Together we will cover a specific case (customers frustrated with an existing product) in the hope that it conveys the mindset needed to have a better understanding of your outcomes, assumptions, and hypotheses.

Don’t outsource thinking

We will go through the entire approach by looking at an example. One important thing to keep in mind is that this is not some linear 10-step process. We are more or less digital detectives, working with what we have to solve a problem — without guessing. This requires a lot of critical thinking and keeping track of your work — not marking off items on a checklist.

Moving on to our example

Here’s an email from a customer:

Your website is frustrating. I tried to order pictures and pick them up at Target but it wanted me to have them delivered. What a waste of my time! I don’t even have the pictures I need!

Ouch. It doesn’t seem like this customer had a good experience. Let’s say, for our example, that we found a significant amount of similar feedback from other customers. Let’s say we also have a set of key metrics that are below their targets.

We know we need to improve this experience, but how do we do it?

Replace the opinions with a list of questions

This simple approach will help reframe the conversation — shifting from opinions about what the problem is and how to solve it, and replace those with a list of questions aimed at increasing our understanding of the problem. With a greater understanding, we can reduce some risk in our assumptions and the resulting experiments.

Here are the the things we want to better understand:

  • The goal the customer is trying to accomplish.
  • The tasks the customer thinks she needs to complete in order to accomplish her goal.
  • The current experience accomplishing these things.
  • How we will measure success?
  • What will improve the quality of this experience?

A little bit more about our example

Let’s provide some background on our example before we continue digging into the process. In this case our example will follow an experience of trying to accomplish a task on Shutterfly’s website. Shutterfly is an internet-based company specializing in creating photo products.

My wife has used Shutterfly’s various photo-related products over the years. She’s the one who had the frustrating experience.

I asked her to write a quick little letter to Shutterfly to share her experience. We didn’t send the email — but I wanted her to document her real thoughts.

What we learned from the customer’s email feedback

This diagram helps us understand what steps the customer thinks are important to reach her goal. At this point we only know what we can gather from that single customer email.
  • The goal: Get prints of pictures.
  • Was the customer successful? Nope. She gave up.
  • Additional sentiment: The website is difficult to use; it was a frustrating experience.

Understand the customer’s goal

That one source of information gave us a signal — but we don’t know if it’s accurate. Is the goal really to get prints of pictures? To start building our understanding of the customer’s goal and associated tasks, we need more information.

Questions we want to answer:

  • What was the customer’s end goal? Do we understand it enough to help improve the experience?
  • What tasks are involved?
  • Who else is involved in the tasks?
  • In order to accomplish this goal, what is the customer’s mental model (expectations, thought process)?

This is more of an atomic list — it’s meant to give you an idea and it isn’t exhaustive by any means. Your unique mix of product, customer, and your organization will provide some additional relevant questions to ask. When starting off with your research, you should always have questions you are trying to answer. If you don’t, you might be in trouble — and you’ll probably start adding your own ideas to the giant list of opinions.

Talk to customers

Now that you have a list of questions you’ll want to talk to 3–5 customers who have a similar problem. You want to start building your understanding, but it should be as quick as is possible. The goal here is to learn more about the mental model of the customer.

After interviewing this particular customer, we learn she wanted to order 5x7 prints and needed them the same day. It was for a gift. The print was going into a frame she still needed to buy. She usually orders prints from Shutterfly and they come by mail. All of the pictures she needed were already up on Shutterfly’s website. She remembered that the last time she was on the site she noticed an option to send the prints to a local store in her area.

What we learned after interviewing one customer

We are now starting to see a more complete understanding of what the customer was trying to accomplish.
  • The goal was to give someone a framed picture as a gift.
  • There is a sense of urgency. It was a time-sensitive task.
  • The tasks were perceived as being easy to accomplish.
  • Shutterfly was selected because it was convenient.
  • Shutterfly itself made the customer aware of the service.

With this information we now have a better understanding of the customer’s story. Now let’s see what the experience is like for ourselves.

Understand the current experience

To start building our understanding of the current experience we want to know the answers to a different set of questions (are you noticing a theme?).

Some questions we want to answer during our current experience discovery:

  • What areas are potentially causing: Confusion? Friction? Distractions?
  • Is the current flow to accomplish the task intuitive?

The customer interviews may help shed light on some of these questions, but we need to step through the experience for ourselves, having some sort of narrative in mind when you do this is helpful.

Go and see the current experience

We’ve established what this one customer was trying to do. Now we need to “go to the Gemba” and see this for ourself. Gemba is a Japanese term that means “the real place”. In lean manufacturing — this phrase is used to say, “Hey, you in the office — come out to the floor and see how the work is done”. In the digital realm where we have visitors in their homes, offices, coffee shops, traveling, etc., we need get as close to that as possible. The importance of the problem we’re trying to solve will dictate how much effort is needed here.

The purpose of this task is to get a sense of what our customers are experiencing. It provides perspective for the feedback they’re providing.

Do one initial pass without taking any notes. Focus on what you know about the customer’s overall goal, frame of mind, and empathize with them as you would a character in a story.

If you’re playing along, go to Shutterfly’s website and step through the process of ordering prints for pick up at a local store (the flow looks similar as of 01/22/2017). We know from our customer interview that the customer was alerted to the service by seeing it in the top navigation under Prints.

This is where we know one customer located the pick up prints service on the Shutterfly website. This can give us a good starting point for the flow of the task.

Create a diagram of the current flow

This map will give us a visual to help us understand where people are actually going. I find it important to have some sort of visual that can tie all of the detailed work together. This will be built out over time as you gather data from qualitative sources, like customer interviews, as well as when you start gathering from quantitative sources like web analytics.

For this diagram I’d recommend names that will help you remember the actual page. This could be the page title or whatever the purpose of the page is.

The map of the current flow helps you keep track of where to go next, to make sure you’ve analyzed every page, to keep track of the pages you discover once you dig deeper.

Understand how the customer’s mental model lines up with the user flow

Now let’s combine the current flow with the previously created goal model. Use this as your empathy guide when researching the problem. It’s a great way to try to understand what the customer might be thinking at a given step in the flow.

Mapping the customer’s mental model to the user flow gives us a great visual of which steps in the flow fail to help move the customer forward. It also serves as a way of understanding that our products might be assisting with some tasks, but other tasks remain for the customer to reach their actual goal.

Identify potential problem areas

Now we’ll go through the website again. This time we will ask questions. My words of advice: don’t make assumptions. We need to be humble — to always have a learning mindset. We don’t know what the real problems are here. Our goal is to objectively evaluate the experience and create a list of questions. We can start to get an idea based on all of the data gathered so far — but this pass through will test your susceptibility to confirmation bias.

Take screenshots and make annotations

Looking at the flow map we created as reference, we will go through every single page and look for areas that may cause confusion, friction, or distraction from the call to action. We also check the entire flow of the task. I like to take a screenshot of each page and then annotate each image with my questions. Here’s one of the annotated pages. I initially used Notable to keep track of the annotations for this example — but if I was working with a team then I’d use Invision to keep track of the annotations. I find it helpful to recreate the paths with hotspots. If you want to see the whole collection then you can access them here.

The screenshots with the annotated questions help provide context for every question. We will eventually add them to a spreadsheet, but this is a good way for a team to review a website together.

Keep track of your detailed work so you don’t get lost

I don’t know if you’ve noticed, but we end up asking a lot of questions and amassing a ton of data. My recommendation is to keep track of all of the questions. You’ll end up with a chain of questions, research, findings, more questions, research, findings, etc. If you don’t keep track you might get lost, you might miss something, you might waste time. Well, I do at least… so I use a spreadsheet.

Here’s a template I use to keep track of the questions I have on each relevant page/step in the process. I’ll use multiple worksheets if the research trail gets long.

This worksheet template is how I think of my work. Always look for what make makes sense for you. This isn’t a dogmatic approach.

Here’s every piece of information I include; adjust for what makes sense for you and your organization.

  • The number (#): Purely for reference and helpful when you’re working with a team.
  • Question: This is the actual question from the website review. This is a good opportunity to refine your initial thought as well.
  • Status: Keep it simple: to do or done. I want to know if I still have to act on this question or if I gained enough insight and direction to move forward.
  • Page Name: However you or your team refers to the page, put that here.
  • Page Area: An additional description to help team members who might not be as familiar with a given page.
  • URL: The location of the page. Where can I go and see this?
  • Next Steps: Critical. If you get to this and can’t think of a next step, it’s time to rethink/reframe your question.
  • Findings: What did you learn? Be as detailed as possible. If it’s complex then write it up somewhere else, summarize here, and link to your work.
  • Next Steps RD 2: This is where some things might simply end. No more follow-up digging. Perhaps you have enough here to help formulate an experiment. Note that in this section and mark the research as done.

All of the questions here will eventually go into my spreadsheet.

This is the completed spreadsheet for our Shutterfly example. Round 1 is the first pass. You will do an assessment and generate a list of questions. Round 2 is the follow up. Once you’ve gone off and done the next steps, you’ll keep track of your follow up questions in round 2.

Understand what to do next

Everything up to this point has helped us understand the problem space by performing some research. We dove really deep and created a list of questions to help guide the next actions we take.

At this point in our research we haven’t dug too deep into many data sources; we’re still in our first research cycle. After updating our assumptions we can create hypothesis statements, work through design ideas, build, and run experiments.

Here’s where we are so far with our assumptions:

  • User of the print pickup service: A mother with young children who is currently a Shutterfly customer. She is busy and is looking to save time. She has family that live close by. This family enjoys receiving pictures of her kids.
  • User outcome: “I want to print pictures out from my Shutterfly account. I want to feel like this is an easy thing to do since I’ve been a Shutterfly user for quite some time now. I want to feel like this is saving me some time while also giving someone I care about a meaningful gift.”
  • Business outcomes: Increase conversion rate through the funnel. Increase the number of repeat customers for the print pickup service. Increase customer satisfaction with the service.

Out of these three elements the most risk right now is around the business outcome. This has to do with the fact that we don’t work for Shutterfly. We don’t know how big the print pickup business currently is.

Quick note on metrics

I find Google’s HEART Framework to be a good source to help you think about metrics to measure the user experience. Increased revenue is great for a business, but I find user-centric metrics to be a bit more inspiring for a team trying to do meaningful work.

Understand where to learn more

The role of a product manager is continuing to evolve, but creating meaningful products and experiences is at its core. The Lean UX process provides a fantastic framework for doing just that.

For additional information on specific research techniques I suggest the following as a start: