Measuring design excellence

How do we know we are achieving design excellence?

This seems like an easy question to answer. Don’t we have all this data indicating that what we did was excellent:

• User quotes state the new interface is fantastic.
• All the milestones were met, and all the work is completed on time.
• The backlog of issues is small.
• The team leveraged the company design language and widget library in new design work.

However, most of those items are subjective since we are just asking research participants, “Is this better than what you have today?” Our answers for these indicators are typically in pass/fail style. What we need to answer this question objectively and to inform our decisions are metrics.

For those conducting user research, your first reaction maybe
“We already collect design metrics throughout the design process. We are counting the number of user hours, and we get numbers from the Likert scales in our surveys.” Yet, most of the metrics we collect today are reflective of the new work only. We never took the time to establish goals for the release, measured the current state of our offering, or evaluate the competition’s offering. For example, you report to the business that you did 30 user hours. Is that good or bad? For a two-week sprint, you may be overachieving, whereas for a three-month iteration, you might be underachieving. Did those 30 hours provide any insights? Without a goal and baseline measurement, we can’t tell if we’ve accomplished excellence. These goals will help you understand when you’re on track or falling short, thus being able to correct course throughout the project.

What we shouldn’t measure

The answer really depends on several factors. Look for metrics that you can directly influence during the release cycle. It’s tempting to set an ROI target, but this metric is a team sport — not just something that Design drives. Business metrics the management team uses tend to represent the cross-disciplinary effort on the project, not just Design. That will make using business metrics hard for your design team to achieve alone.

You also don’t need to track your progress and work if you are including your efforts in the tool that Development is using. For example, most development tools, like JIRA, ZenHub, or GitHub, will track story-point completion, capacity, burn-down rates, and anything else the release manager is tracking. If you include the Design work as epics, stories, and issues with the same tool that tracks Development’s effort will automatically track Design’s effort. You should also avoid process metrics, like “Did we do a workshop, create an MVP, or hit that milestone in the plan?” The release manager is already monitoring the progress and step completion of the process.

Getting started with metrics and goals

It can be hard to determine which metrics to use. Start the conversation by focusing on the goal you want to achieve.

Ask yourself these questions:

• How do the business objectives translate for the design team for this release?
• Where have we fallen short in the past efforts?
• What new behavior do we want to drive for the team?
• What measurement will show our user’s success as we improve the experience?
• How do you know your design meets the majority of user needs?
• Which parts of the design language matter to this effort? How far do we go beyond pass/fail for our design language?

These are not easy questions to answer. Answering them will give the design team a better understanding of what is at stake and the priorities of the entire team. Our design work is a collaboration, so should be the creation of our metrics. Start by setting up a meeting with your partners in crime, an offering manager (OM) — the business leader ­– and an architect — the development leader. Begin with your understanding of the design goals. Explain what these goals are going to accomplish for the release. The outcome is a list of targets where designers should spend their time for the program increment. Don’t expect to walk out with a list of metrics. We’re just looking for agreement on the design goals. Gaining alignment on release goals is an important step that you may already be doing, so keep it up.

Once you know the goal you’re attempting to achieve, then figure out what metric or metrics would indicate that you’re making progress towards that goal. Look for metrics that you can measure throughout the various stages of the development cycle. Remember to think about the maturity of the entire team; let that guide you to establish the metrics. For example, a team just starting to work with design most likely has done little user research in the past. As a starting point, set a goal to focus more on user hours and user satisfaction with the improved task. Don’t jump to some complex or lofty metric. Our goal is to showcase new behaviors and the impact of Design on the release. As the team matures, you can start adding more metrics like task completion rates, complexity analysis metrics, A/B comparison metrics, and many more ways to measure that you’re meeting the goals.

What could we measure?

User research will be the core of metrics and provide the most insights. You’ll want to start with a combination of user hours overtime and an insight metric to tell the story. Insight metrics like time on task, success rates, issues found from a user study, number of insights from a persona interview, or increase in user satisfaction tell the impact of those user hours. Establish a baseline by collecting the same metrics before the design work to compare after the design work is complete. This will showcase the impact of your design work. As the maturity of the organization grows, you can start to measure something more complex, like “Are the visuals in the offering communicating the correct meaning?” Ask users to explain their understanding of a visual and rank the effectiveness of the visual.

A heuristic evaluation or review can also provide insights into the experience of a given task, although nothing will replace user research with actual users.

A heuristic review is typically conducted by three to five experienced practitioners who understand the best practices for user interfaces. Take screenshots of the flow as you go and map out the entire journey the user experiences while performing a task. Evaluate the steps to achieve the goal for showstoppers, the number of hops, and other issues to determine a final score. At IBM, we call this a Design and User Experience (DUX) review, which includes a list of ranked problems and a letter grade at the end of the study. Here are just a few of the many articles that provide information about how to conduct and score heuristic reviews on the internet.

• https://www.toptal.com/designers/usability/usability-analysis-how-to-run-a-heuristic-evaluation for more details about doing a heuristic review.
• https://www.nngroup.com/videos/heuristic-evaluation/ for another view of running a heuristic review.
• https://uxpajournal.org/heuristic-evaluation-quality-score-heqs-defining-heuristic-expertise/ for information on refinements the NN/g method to measure and scoring the heuristic review.

For more articles, search on the terms “heuristic review” and “scoring heuristic review.”

Establish a baseline for how the user performs the existing task; even if the task is done by hand, scripts, or command line, these are flows you can measure. Once you have a baseline, establish a goal to reduce that number and compare it with the final design.

Socialize and record the metrics

Now you have a list of metrics you’ll be measuring; you need to socialize them to the key stakeholder and devise a process to record these metrics. Leverage the Object and Key Results (OKR) framework as a way to define a goal and metrics, which indicate success. Find ways to incorporate the design metrics whenever business and process metrics are discussed. Reporting regular updates on the metrics will help you celebrate success and start the conversations when you need help.

As for recording, any tool that allows basic trending and graphing will do. Most start with a spreadsheet. Unless you want to be the gatekeeper, pick a tool that will enable collaboration across the entire design team. You may even want to open up to partners like an offering manager who talks with customers about roadmaps since they are collecting feedback as well. Unfortunately, the horror stories about spreadsheets are true. Eventually, the file will get clobbered, duplicated, or encounter replication errors. You might want to consider an online version, if your company allows such a tool, or an online database like AirTable. Don’t go off to build something new unless you are solving this for the entire company and other options fail to meet corporate guidelines. Above all, don’t make a tool to be a barrier to collecting metrics.

After going through this effort, you’ll have several metrics aligned with the release goals for your offering. Collecting metrics will be difficult at first, and it will be easy to lose track of them during a development cycle with all the other demands. Keep the design team accountable by reporting the metrics and progress throughout the development cycle. Determine a cadence for metrics reviews that strikes a balance between time required to collect new data and timeliness of acting on that new data. Use these metrics to showcase the impact of the design work and proof that you’re achieving excellence. After successfully using metrics during a development cycle, evaluate and update for the follow-on release. You’re now off and running. Good luck!

BJ Scheid is a IBM Distinguished Designer at IBM based in San Jose, CA. The above article is personal and does not necessarily represent IBM’s positions, strategies or opinions.