Complexity Analysis for UX Research at IBM: What it is and how to get started

Several months ago, I gave a presentation during a Best Practices in Design Research session alongside Rick Sobiesiak (Design Research Lead, IBM Security) and Tim O’Keefe (Design Lead, IBM Systems) about complexity analysis. Rick and Tim developed the method with IBM Research and I was fortunate to learn from them. I’ve applied the method on my team’s product, IBM PowerAI Vision, to quantitatively demonstrate the value add of user experience design in terms of lowering the complexity of a task or sequence of tasks for the user. After this presentation, I received many Slack messages and emails from other product teams wanting to implement complexity analysis. I believe it’s critical for user experience researchers to develop a working competency in a few quantitative methods and always use qualitative and quantitative methods together for a panoramic and robust user story (read more about this here).

I have also linked Rick and Tim’s original complexity analysis publication at the end of this article under ‘Resources’ for those looking to dive even deeper! Let’s get started.

Method Overview

Complexity analysis is a usability inspection method to assess the usability of a specific task or set of tasks that does not require direct user engagement and provides quantitative outcomes. Because complexity analysis does not require user engagement, the evaluators of the product task are typically members of the product design/development team. I would recommend a pairing of user research and user experience design roles if possible. However, be mindful that having the lead designer also be the evaluator could introduce bias when trying to objectively review the difficulty of the task, so engaging a team member that may be focused on a different project or tangentially related to the space could be beneficial.

As the evaluators step through the task being tested, they are looking for elements that get in the way of the user reaching their end goal. This is why it is critical to understand both your user and their ultimate job to be done before performing complexity analysis. Otherwise, you will have no frame of reference when conducting the evaluation.

Unlike qualitative methods, complexity analysis will quantitatively measure elements that get in the way of the user’s end goal through complexity metrics. These metrics are derived from the evaluators assigning ratings for each step or interaction in the specified task. The rating categories include:

  1. Context shifts: moving the user within the product to complete the step.
Ex: A modal is a minor context shift versus taking a user to a new page.

2. Navigational guidance: provided support to begin and complete the step.

Ex: Intro text on how to get started can remove guesswork for core capabilities.

3. Input parameters: info the user has to specify to complete the step.

Ex: User input required for training is two radio buttons versus text/image input required in other steps.

4. System feedback: system response to user interactions during the step.

Ex: A real-time notification log documents the steps the user has taken.

5. Error feedback: system response to common error situations in the step.

Ex: This error message tells the user why a step failed but does not give suggestions for resolving.

6. New concepts: information the user needs to understand for the step.

Ex: Modal provides feature overview, but link for more info will take the user out of the product.

Each of the rating categories above have their own rating scale, and Rick’s assets provide examples of low and high scores for each respective category. This is because the complexity analysis algorithm that computes a final quantitative score is weighted based on the level of influence each rating category has on the overall complexity of the task. Luckily for us, we don’t have to do the long form calculations ourselves! But it is critical to understand the importance of weighted algorithms and why each scale is unique.

When to Use

I have found complexity analysis to be a rather flexible method in terms of when to use and how to adapt it to your product team’s specific needs. However, there are a few requirements for use:

  1. You must be in the evaluative research stage post generative research.
  2. You will have identified a target user(s) and their job(s) to be done. This identification process means that you not only know their job role and responsibilities, but also know what concepts are known to them versus what could be perceived as new. You will have conducted enough research to know their areas of expertise and areas of uncertainty.
  3. You need representative wireframes of specific tasks that the target user is looking to complete (wireframes do not have to be hi-fidelity but they do need to be comprehensive and inclusive of all necessary components to complete a specific task).

With these requirements in mind, I have used complexity analysis in several different use cases during the product design process:

  1. To compare a previous UI to a newly designed UI (This can quantify the value-add of a design team to product management and development).
  2. To compare the same task flow in a competitor product to our product (Great for conveying where UX and development work is needed to ensure a product is up to par with the competition).
  3. To compare two different design flow possibilities for the creation of an entirely new feature/capability in the UI (This helps when new innovations are introduced to the product or the stakes are high for integrating a certain feature and the UX team needs to ensure they get the right flow before validating with users).
Use case 1: comparing a previous UI to a new UI (note that the lower the complexity the better).
Use case 2: same task flow in competitor products (note that the lower the complexity the better).

How to Use

There are three core steps to conduct complexity analysis.

  1. Breaking down a user task into discrete steps and interactions.
  2. Rating the complexity of each step or interaction in the task across the 6 rating categories.
  3. Analyzing the complexity metrics generated and determining next steps.

1. Breaking down a user task into discrete steps and interactions.

This is the most time consuming stage of complexity analysis, but it sets the foundation for evaluation. Additionally, it’s a refreshing view for the product team because you can catch any issues that our own familiarity can lead us to overlook. The most difficult part of this process is having a strong understanding of what qualifies as a task, a step, and an interaction.

Ex: breakdown of tasks vs. steps vs. interactions.

A task is a discrete component of the user’s overall goal or job to be done. The steps are the the individual elements that work towards the completion of a task. The interactions are the literal engagements the user will have with the product to fulfill a step. You will rate the complexity of either the step or the interaction depending on the rating category.

2. Rating the complexity of each step or interaction in the task across the 6 rating categories.

After you have outlined your tasks, steps, and interactions, you will need to move through the product and rate each element appropriately. Do not try to do this from memory! Go through the product or prototype as you score.

As you work to apply your ratings, you will need to reference the unique rating scale and criteria for each rating category. This will help you understand what determines a 3 versus a 5 for navigational guidance for example.

Complexity metrics rated at the step level: navigational guidance, system feedback, error feedback, and new concepts.

Complexity metrics rated at the interaction level: context shifts and input parameters.

If using the complexity analysis assets we use within IBM, your rating inputs will be automatically converted into complexity metrics using the underlying algorithm. These complexity metrics will be shown both numerically and graphically.

Zoomed-in view showing what data YOU need to input (note a,b,c are interactions) — remember, some complexity ratings are assigned at the step level while others are assigned at the interaction level.

3. Analyzing the complexity metrics generated and determining next steps.

Once you finish entering your complexity ratings and the complexity metrics are calculated, you will need to understand what the results mean in terms of actionable next steps. While the data and charts are valuable, they are much more meaningful when coupled with suggestions or proposals, especially when sharing with development and product management teams.

Complexity metrics in bar graph form.

In the above example, we have the use case discussed previously where you can use complexity analysis to compare two competing products that both allow the user to complete the same task. Since we want the complexity to be as low as possible, we can see that Product B outperforms Product A in the context of this method. We can also see that navigational guidance is the biggest complexity difference between the two products, followed by new concepts. Therefore, we can propose that Product Team A prioritizes changes that reduce the complexity of navigational guidance and new concepts. In addition to conveying this information, the team who conducted the complexity analysis can suggest specific ways to reach these goals including more info icons or on-boarding elements to step a user through a task and keep them informed along the way.

Best Practices

Based on my experience and guidance from Rick and Tim, I’ve outlined a few best practices below that help ensure an ideal complexity analysis outcome:

  • Work in pairs to assign the steps/interactions in a task BUT assign initial number ratings alone and then compare (This reduces the tendency for people to agree with one another verbally and potentially assign an incorrect rating. What is critical here is if one person assigns a 5 to a step and another assigns a 2 to the same step, then you can be sure to go back and review).
  • Choose one scoped user goal to focus on. Don’t do the whole product flow your first time around (It’s simply too overwhelming and time consuming in a fast-paced sprint environment; choose a single goal that is not too broad, either isolating the most problematic flow or the most important capability to get right for the overall user experience and product mission).
  • Share your final results with the larger product team, especially product management and development (It’s critical to conduct this method with the intention of sharing. Be sure to clearly communicate generated bar graphs and results with the larger team and propose next steps to address any problem areas).

It is important to note that in addition to calling out problem areas, complexity analysis is equally vital in drawing attention to parts of the flow that are low in complexity and performing well. These experiences must be protected as new designs and features are integrated.

Complexity analysis is a powerful method for quantifying the user experience of specific and imperative tasks in a product. In my experience, it has been immensely helpful when advocating for the measurable impact of design and prioritizing work across disciplines on a product team. Whether you are looking to implement complexity analysis in full or want to leverage some of it’s core tenants towards more informed cognitive walkthroughs or other methods of task-based evaluation, understanding this method will provide more value and range to your user research toolkit.

Resources

For IBM employees: access in-depth resources and assets to get started provided by Rick Sobiesiak here and to join the Complexity Analysis Community, reach out to Rick Sobiesiak directly.

For non IBM employees: access the research paper, co-authored by Rick Sobiesiak and Tim O’Keefe here

A huge thank you to both Rick and Tim! All thoughts expressed are my own. http://www.gabriellacampagna.com/