USER Metrics

A framework for measuring enterprise user experience

Illustration by Stela Stamenkova-Din

How do you measure the user experience of an enterprise product? As a user researcher, I get asked this question a lot:

• Designers want to understand how people use their features
• Design Managers want to justify investment in user experience
• Product Managers want to fix the biggest pain points in their feature areas
• Executives want to set measurable goals for UX across their product lines

On the VMware Design Team, we’re rolling out a framework that we call USER metrics, to measure user experience in production environments and to get continuous real-time user feedback.

The USER metrics framework provides guidelines for measuring four aspects of enterprise user experience: Usage, Satisfaction, Ease-of-use, and Ramp-up.

USER metrics are for designers, design managers, product managers, execs, and anyone else who makes decisions that impact user experience and wants a way to measure what they manage.

Why measure user experience?

Metrics help teams prioritize what to work on and show the impact of their work to justify future investment. They complement qualitative customer feedback by estimating the impact of usability issues and feature requests. While we still design workflows to accomplish functional goals (“As a [persona], I want to do [task], so that I can achieve [goal]”), we prioritize our roadmap based on the change we want to make in the world (e.g. “Increase the percentage of [user segment] who [complete step of funnel]”).

Design for functionality → Design for outcomes

Why do we need a new framework?

USER metrics are based on the Google HEART framework, supplemented with metrics from HCI/UX literature, and adapted for enterprise products and services. We built on Google HEART as follows:

• More prescriptive than Google HEART, to make it easier for teams to get started, and to have comparable metrics to manage across products
• Prioritized, both for order of implementation and order of analysis
• Optimized for enterprise products and services by measuring organization-level behavior

What are USER Metrics?

USER metrics are measures of good user experience design, in priority order:

1. U = Usage: are customers using it?
2. S = Satisfaction: are they happy about using it?
3. E = Ease-of-use: are they happy because it’s easy?
4. R = Ramp-up: is it easy to get started and grow?

While the Design Team will always champion usability, the prioritization of USER metrics reflects our point-of-view that an enterprise experience must first be useful, helping users and organizations achieve their goals.

Each category has a primary metric and secondary metrics. We prescribe specific types of metrics to make it possible to compare across products and feature areas. See below for the metrics we use in our current iteration of USER metrics at VMware and let us know how well they work for your enterprise products.

U = Usage

The usage metrics are:

• Org engagement (primary metric) (org = customer organization)
• User engagement
• Active orgs
• Active users
• Feature adoption

To measure usage in an enterprise context, we distinguish organizational engagement from individual user engagement. An organization (a.k.a. “org”) corresponds to a customer account; some large enterprises have multiple accounts for different business units. We prioritize org-level engagement because: (a) our customer is an enterprise or business unit, not an individual user, (b) org-level engagement is a good proxy for customer value, and (c) it’s proportional to our business revenue.

Org engagement measures how many units of service customers utilize, summing usage from all individual users and machine automation. It’s a unit of customer value, rarely the exact unit of billing. For example, here are units you could use in various enterprise products:

• Documents created in an authoring tool (e.g. PowerPoint)
• Message threads or chat sessions sent in a communication tool (e.g. Slack)
• Transactions processed by a payment service (e.g. Stripe)
• Data (bytes or records) stored in a datastore (e.g. Hadoop)
• Objects tracked with a monitoring or record-keeping tool (e.g. Salesforce)
• Tickets created in a workflow tool (e.g. ServiceNow)

User engagement measures the frequency of key user interactions, such as dashboard page views or searches in a monitoring tool. It tends to indicate value within our monitoring tools, but in some cases, human interaction is a bad thing, indicating less efficient workflows. The ideal user experience might be automation that reduces the frequency of human interaction. Therefore, even for highly interactive information-worker tools such as monitoring and data analytics tools, organizational engagement works well as a primary usage metric, and individual user engagement comes in as a close second.

The timeframe for active usage is set based on the expected frequency of use, which can be a work week, month, quarter, or year. Active Orgs are associated with org engagement during the timeframe, and Active Users are associated with user engagement. Feature Adoption uses the same timeframe, and can be org-level or user-level depending on the feature.

S = Satisfaction

The satisfaction metrics are:

• Product satisfaction (primary metric)
• Product satisfaction follow-up questions

Satisfaction is a good user experience metric because: (a) it’s easy to interpret, (b) it supplements behavioral data with attitudinal data, (c) it captures issues that happen off-screen, and (d) it’s a leading indicator of usage.

Satisfaction surveys are also a good opportunity to recruit customers for interviews. Per GDPR, only people who opt in can be contacted for follow-up.

Product satisfaction (“PSAT”) is the average rating for the survey question: “Overall, how satisfied are you with [product]?” Free responses include feedback about all aspects of the customer experience, not limited to product design. Therefore, in addition to the overall product satisfaction question, it’s worthwhile to ask follow-up questions on satisfaction attributes such as quality, performance, ease-of-use, support, interoperability, and whatever other issue areas are most frequent in free responses. This makes it easier to redirect feedback to the cross-functional team that can follow-up for more details and fix issues.

E = Ease-of-use

The ease-of-use metrics are:

• End-to-end workflow funnels (primary metric)
• Customer Effort Score
• Service requests

Ease-of-use measures workflows. We focus on end-to-end workflows (journey-level, not task-level) because:

• Enterprise projects are often multi-month journeys involving multiple people. At VMware, we design use cases end-to-end, from intent to value.
• Journey satisfaction is more predictive of business outcomes than touchpoint satisfaction (McKinsey 2013)
• We initially tried task-level funnels, but we didn’t find insights because the hard part was earlier in the journey during planning, not task execution.
• There don’t seem to be good tools for collecting and reporting on micro-interactions at scale. The tools that collect all UI interactions (FullStory, Pendo) don’t meet our requirements for reporting and analysis, while the tools that have stronger reporting and analysis capabilities (Amplitude, Mixpanel) require manual instrumentation per event. We’d like to expand to task-level usability insights in the future when it’s more feasible to collect and report on micro-interactions at scale.

We apply standard task-level usability metrics at the end-to-end journey level:

Definition of Usability (ISO 9241–11) The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use.

• Funnels capture completion rate (effectiveness) and time-to-completion (efficiency). Funnels also capture repeated steps, which may signal errors. Completion rate (within a timeframe) is our primary metric. Initially we wanted to use time-on-task as our primary metric, because it correlates with complexity and errors, but it’s only feasible to capture time-to-completion, and it’s hard to interpret what’s good.
• Customer Effort Score (a.k.a. Single Ease Question) surveys capture perceived effort, which is conceptually similar to workflow satisfaction. They can be used at the task level or journey level. The score is the average rating for the survey question: “How easy or difficult was [task]?”
• Service Requests (customer support tickets) are strong signals of errors. Other measurable behaviors associated with diagnosing and fixing errors include engaging with troubleshooting features and/or repeating a step in the funnel. Per the definition of usability, effectiveness means completing a task accurately, minimizing errors. Qualitative evidence indicates that errors are a strong driver of satisfaction and perceived ease-of-use.

Some workflows like troubleshooting and data analysis have a lot of variability, so it’s harder to define workflow-based funnels. To measure the ease-of-use of our analytics products, we start by measuring guided workflows that enable analytics, such as connecting data sources.

The Design Team is most responsible for ease-of-use, however, like the other USER metrics, ease-of-use is driven by the work of the cross-functional team. Documentation and customer support help customers complete workflows efficiently and effectively, and engineering quality affects perceived effort.

R = Ramp-up

The ramp-up metrics are:

• On-boarding funnel (primary metric)
• Scale-up funnel

Ramp-up funnels track the customer lifecycle with a product, similar to customer journey maps. In contrast, ease-of-use funnels track the workflow for a particular use case. We considered organizing the entire framework around the customer journey but decided to prioritize the use phase where enterprises get things done. Regardless of priority, we began by piloting funnels on ramp-up before ease-of-use because there’s a clear business impact, so it makes a nice proof-of-concept for product analytics tools.

On-boarding funnels capture the bulk of retention issues for enterprise products. Churn mostly happens during the on-boarding period, so we combine the adoption and retention sections from Google HEART into a single on-boarding section. In enterprise, product-market fit is signaled by on-boarding from discovery to proof-of-concept to production, not retention like in ad-supported consumer SaaS.

Scale-up funnels are the next step in the customer journey. They are the journey from “Use” to “Grow”; from “Land” to “Expand”; from “Aha moment” to “Habit moment”; from “Active User” to “Power User.”

In addition to funnel steps, we track events that increase or decrease the likelihood of continuing through the funnel. We do workshops with cross-functional teams to hypothesize positive and negative drivers of funnel success, as well as org types that may be more or less successful. When possible, we include touchpoints beyond the product, such as marketing emails and viewing documentation. For researchers who focus on usability friction, it’s good to remember that there are also positive drivers that educate and motivate users to make progress.

We segment our ramp-up efforts between customers who have Customer Success Managers, and self-service customers who rely on our product design, documentation, and the support community. The growth PM coordinates the Marketing Team, Design Team, and others, to drive changes that improve the on-boarding funnel for self-service customers.

What’s not included in USER metrics?

Some important UX attributes are not included, because they’re owned by other teams, or because they’re measured by heuristic evaluation and/or periodic user studies instead of on-going analytics:

• Quality is a strong driver of satisfaction, but we don’t include quality-of-service measures such as availability or performance in the framework because our engineering team already has metrics to ensure technical quality, and we ask about perceived quality in satisfaction surveys.
• Accessibility, privacy, and other aspects that are found by heuristic review are not included, because this is focused on metrics to be collected and reported with data analytics tools. Analytics prioritize the behavior and attitudes of the majority, so it’s important to have other processes such as accessibility testing to draw attention to smaller user segments.
• Visual appeal isn’t included, because we prioritize usability-oriented enterprise design principles such as simplicity, learnability, and error prevention. The visual style of our design system, Clarity, doesn’t change frequently enough to monitor with analytics, and satisfaction surveys have been sufficient for us to monitor enterprise-relevant visual design issues such as data density and dark theme.

Wrapping up

USER metrics help the VMware Design Team and our cross-functional partners prioritize our work and track our progress in helping customers meet their goals and grow with ease and delight. It’s been an eye-opening reality check to see real-time feedback from real-world customer environments, and it’s been rewarding to see how simplifying design has real impact. We encourage you to try USER metrics yourself and share back your stories of measuring and improving enterprise experience.

We’re hiring!

Interested in helping us define the user experience for next-generation enterprise solutions? We’re hiring talented UX researchers and designers.

Thanks to Jehad Affoneh for initiating and sponsoring this project; Leo Yeykelis for reviewing the framework and providing guidance on operationalizing it; Baker Woods for partnering on a proof-of-concept; Jahnavi Patel and Nancy Cheng for support on queries and data infrastructure; David Joyner for partnering on in-product feedback; Kevin McBride for leading design beyond usability to start with customer value; Zhao He for demonstrating measurable design impact; and Grace Noh, Bonnie Zhang, and Josh Johnson for reviewing the article.