The Evolution of Design Debt: The Design Consistence Score
We started using the Design Debt method a few weeks ago and ended up enhancing it to fit our needs. In this article I will explain what Design Debt is, how to measure it, which challenges we encountered while using the method and how we optimized it to create our own UX metric called “Design Consistency Score”.
The concept of Design Debt
Design Debt is a concept that is derived from the term technical debt. Technical debt describes a software code that was developed by many different people over the course of time (or even at the same time) without code compliance. Since each developer has their own style, the code for similar functions can differ widely.
So the „debt“ in these terms means that you collect a lot of inconsistencies over the time that should be cleaned up afterwards — which usually doesn’t happen once the team moves on to their next projects. For design elements this can happen as well. Think about the time after a website relaunch during which new sections, new landing pages etc. are created — leading to additional styles for buttons, teasers, form fields, tables etc. And this can result in inconsistent (possibly negative) user experiences (UX). In the past few years we started building Design Systems in an attempt to reduce Design Debt, but still, even with a Design System in place, the Design Debt can emerge.
“Design Debt affects the integrity of the user experience.” (Austin Knight)
Design Debt as UX metric (KPI)
With the Design Debt method we can measure these inconsistencies as a quantitative UX metric and use it as a Key Performance Indicator (KPI) for the UX of our digital products. For example, we can use this metric to compare our digital products before and after a relaunch or even with other digital products from within the same industry.
Quantitative methods normally require many participants (users of the digital product) to get statistically significant results. But the Design Debt method is a heuristic expert evaluation that can be executed by only one or two UI Designers and therefore lets us calculate a quantitative metric in a nice and simple way. We used this in a recent project to enhance quantitative data from user tests with a small budget.
How to measure Design Debt?
In order to measure the Design Debt of any digital user interface you have to:
- define the most important elements within your digital product or your whole digital eco-system (e.g. atoms, molecules and maybe organisms according to Atomic Design)
- count the number of variants for each element and the instances of each variant.
In other words, you will answer these questions:
- How many variants of a single element exist within your digital product?
- How many times does each variant show up (on all sub pages)?
Furthermore, the original Design Debt method states, that only one variant of any element should ever be displayed to users. We started calling this “perfect score”.
Then you can use this formula to calculate the Design Debt Score for each element:
Design Debt (of element x) = (variants — 1) x (instances x 2)
As you can see the perfect score is subtracted from the total number of variants and the instances are multiplied by two to give them a higher weight. More instances of any variant lead to a worse score (the higher the score, the worse the consistency). This places the focus of the original Design Debt method on the number of instances.
For example:
- 3 different buttons across 40 pages = design debt score of 160
- 2 different buttons across 20 pages = design debt score of 80
- 1 consistent button across 20 pages = perfect design debt score of 0
After you calculated all the scores for each element, you get your total sum of Design Debt as an absolute number.
Our challenges with the original Design Debt method
When we started using this method we worked under these parameters:
- Our clients’ websites often consist of more than 500 pages
- We identified more than 30 elements we wanted to look at
- We wanted to calculate the total score as percentage to integrate it into an overall UX Score based on different KPIs — all measured as percentages
The combination of the first two factors resulted in an extremely high effort (more pages x more elements). Calculating a percentage with the original method is possible, but much more difficult and not really self explanatory.
How we developed the Design Consistency Score
We started experimenting with adjustments to the Design Debt method that would deliver the same meaningful results but make it easier to execute:
1. More than one allowed variant for each design element
In some cases, like campaign landing pages, for example, UI designers need a little more creative freedom. So we wanted to allow more than one variant for a perfect score. We also wanted to define different perfect score numbers for different elements.
2. Define a maximum number of allowed variants
In order to make calculating the total score in percentages easier, we then defined a maximum number of allowed variants per element and called it the “insufficient score”.
Now we have three scores for each element:
- Perfect score
- Insufficient score
- Actual score
The first two scores have been defined once and in an optimal state, they are valid for all websites on earth. They will only change according to the size of the website, so if there are more sub pages, then more variants will be allowed (in the next chapter we will explain this in detail).
Example for a single element:
- perfect score = 3 buttons
- insufficient score = 5 buttons
Different actual scores result in these Design Consistency Scores:
- 1 button → 100% consistency score
- 3 buttons → 100% consistency score
- 4 buttons → 50% consistency score
- 6 buttons → 0% consistency score
If an element is not used within the digital product we are studying, we are not scoring it (which lowers all total scores). To calculate the overall consistency score for a digital product, we add up the numbers for each of the three main scores (only of the elements that exist for that specific digital product) and calculate the overall score from the three totals (don’t calculate the average from the percentage scores of each element!).
Example for an overall score:
- total perfect score = 30
- total insufficient score = 80
Different total actual scores result in these total Design Consistency Scores:
- actual score = 9 → 100% consistency score
- actual score = 37 → 86% consistency score
- actual score = 99 → 0% consistency score
The formula shown as a graph:
In this example, we have a lot of creative freedom for up to 30 variants for all elements. For each new variant the Design Consistency Score decreases in small steps — until the total insufficient score (80) is reached and the score remains at 0%.
3. Integrating the total number of pages
We were saving a lot of time by not counting instances anymore, but we still wanted to factor in the size of the websites we were looking at. Our reasoning behind this is that larger websites with a variety of different purposes and sections (i.e. product pages, job listings, blog posts, or company information) need a larger variety of elements than, say, a microsite with only 10 subpages.
Basically what we did was to define that our perfect scores and insufficient scores for each element are valid for 100 pages. Then, we used a SEO crawling tool to count the actual number of pages within a digital product (e.g. screaming frog).
At last, we calculate the difference of the total number of pages from the defined 100. We use this to adjust all perfect scores and insufficient scores — the higher the number of pages, the higher the perfect and insufficient scores for each element (and as a total) and vice versa (for websites with less than 100 pages). Let’s call this “adjustment score” — and we only weigh it by 10%, so the impact on the final score is rather small.
After we applied all these changes to the method, we decided to call it “Design Consistency Score”.
Benefits of the Design Consistency Score:
- You don’t need to count each instance of all elements and variants on hundreds or thousands of sub pages (less effort)
- Because of this, we can base our score on a larger number of design elements (we are using around 30)
- The total number of a website’s sub pages is considered as factor (the more pages, the more variety = creativity is allowed)
Our workflow to calculate the Design Consistency Score:
Now that we know what numbers we need, we use this expert evaluation with this workflow:
- A designer browses through the website and collects all variants of all elements in a Sketch file (on separate artbords).
- We count the total numbers of variants for each element (actual score).
- We crawl the website to count the total number of sub pages.
- We paste all numbers into our excel template with the above formula (including the pre-defined perfect and insufficient score for each element).
- We have our score!
Additionally, the designer is asked to write a short review about the design of the tested digital product to explain the final score — like a management summary to highlight the most important wins and challenges of the design consistency for this digital product.
About mediaman
mediaman is a Germany based agency with additional offices in Boston (USA) and Shanghai (China). We help our clients to stay relevant in an increasingly digital world by providing services like strategic consulting, data & design sprints, UX/UI design as well as UX Research.
