Key Experience Indicators: How to decide what to measure?

When you measure smaller parts of the experience (only important ones, not everything), you get an indication of what specifically works well and what doesn’t. You can then dive deeper into the problem areas to understand why.

Chef Antonella opened her new restaurant six months ago. Everything has been going really well, it seemed. Revenue is up and to the right, tables are reserved in advance, and the restaurant is full and lively every single night. Business metrics show a very positive picture.

Chef Antonella also measures two key experience indicators she cares about. These paint a different picture. Close to 100% of customers are new. She has almost no returning customers. Second, on average, 70% of food is left on the plates. Something is wrong although business is great.

🔑 Why measure experience?

Key Experience Indicators (KEIs) provide a quantitative score of a specific, important, and actionable phenomenon related to using a product or service. Measuring KEIs has the following benefits:

  1. KEIs provide information to decision makers.
  2. KEIs precede or predict business outcomes.
  3. KEIs give insights into qualitative findings and customer anecdotes.

🔬Considerations of KEI measurements

What vs. why: KEIs will never explain themselves. To understand a KEI, one must invest in qualitative research.

Results of experience-related measurements for three alternative designs. Design a is better than b and c, yet you don’t know why until you make an effort to learn.

Good vs. bad: It is not always clear whether KEIs are showing a positive or negative phenomenon.

Can you tell if these are good or bad?

  • Average time to first click is 49 seconds.
  • Average time on site is 27:38 minutes.
  • Homepage bounce rate is 99%.

Attitude vs. behavior: what people say does not necessarily match how they behave. People’s attitude is interesting yet behavior is much more telling. Give more attention and weight to behavioral rather than attitudinal metrics.

Chef Antonella’s restaurant is providing great coffee. Yet if you ask one of Antonella’s customers to rate her satisfaction with coffee immediately after she found a mistake on the check, coffee might be getting low scores. Coffee consumption, however, measured in the same exact time, might tell a more realistic story.

Beware not to measure behavior through attitude. Don’t ask about behavior in a survey. Instead, measure it directly.

Overall vs. specific: Measuring specific phenomena gives you the ability to focus efforts and save resources. When you measure NPS or overall satisfaction, you get scores that do not help you understand what goes well and what needs fixing. If you get a score of 66 for your product’s NPS, do you know why you lost 34 points?

Alternatively, when you measure a lot of smaller experiences, for example, at a hotel, you may find that while your overall UX score is 93%, you lost 7 points because coffee, room cleanliness, and the online checkout process got low scores. You wouldn’t know why and yet you can start exploring these three areas rather than guessing what’s wrong.

Instrumentation vs. analytics: Measuring the user experience sometimes involves instrumenting dedicated sensors within your product or service (physical or digital) because different analytics packages don’t always give you the indicators you need.

In vs. out of context: When measuring attitude (what people think), it’s very easy to fall into the trap of sending a survey once in a while, yet a survey is not “close” to the experience. The best way would be to ask during or immediately after people have a certain experience. Leave surveys to questions that don’t make sense to ask otherwise.

At WeWork, we experimented with measuring attitude toward different parts of the experience. For example, we asked about satisfaction with phone booths and printers in those spaces, about events using an app we created, and left asking about the office chairs to a quarterly survey (chairs don’t change much and it didn’t make sense to put a happiness meter on each one).

🥇Only measure things you have full intent to change

Spoiler: you can change anything in the experience you created. It’s all a matter of motivation to do so. This one is important. Your customers will stop cooperating with your attitudinal measurements and get angry with you if they notice you don’t change the experience after they provide feedback. 
Test question for the team: Will we take action once we have data for this metric?

🥈Reach team agreement and shared understanding

Agreement within the team and senior stakeholders that a certain KEI is important. Disagreement results in ignoring KEIs and not taking action. Make sure to create a shared understanding and acceptance of what’s being measured.
Test question for the team: Do we agree this metric is important to measure?

🥉Only measure things that are core to the experience

Measuring everything is measuring nothing. Focus on measuring things that are truly central to the experience. Together with your team, reach agreement on what these are. You may have some debates here but for the most part, those core things are going to be easy to identify.
Test question for the team: Does this metric measure a thing that is core to the experience?

🎯 In a separate series of articles, I share a set of useful KEIs to measure based on the Google HEART framework:
Measuring user happiness
Measuring user engagement
Measuring user adoption
Measuring user retention
Measuring user task success

Meanwhile, please 👏👏👏, share, and comment. I’d ❤️ to hear from you. How difficult (or easy) is it for you and your team to decide what to measure?