DesignOps: What can we measure?
I like to think DesignOps is everything that needs to be done in order to create and support a Design Ecosystem to scale and amplify the design practice. To do this, we need to find out what’s in our ecosystem and how it’s working today, after all, we cannot change what we cannot see.
Metrics for design are pretty well documented: heuristics, conversion rates etc, but all of them focuses on the outcome, in order to have something more specific we started doing some research on how other Design Orgs are using metrics to measure the day to day processes and interactions. A great starting point was Dave Malouf‘s’ article on Measuring DesignOps, it speaks about the value of metrics for the whole practice (i really recommend you to follow him).
At Dell Digital Design we decided to define a set of KPIs for our Design Org, that way we could have a snapshot of what’s happening at a given time in the form of a dashboard, with this, we could define our road maps for initiatives and measure if we’re taking the right actions towards success, this is now our primary tool to make informed decisions.
Where to start?
Once we identified the need for metrics the next step was to ask ourselves “What can we measure?”, we needed to align all these metrics with our goals, and at the same time, we needed to show our stakeholders something that makes sense to them (hours, $, resources), most importantly, we needed to make Designers life easy!.
The approach was as follows, we defined four large buckets for our KPIs:
Each bucket represents a key area of improvement for our organization, now we could focus on metrics for :
- What are we doing,
- How we are doing it,
- What are we delivering
- How are we feeling about it.
After that, based on industry standards and evaluating our own requirements, the whole list looked like this:
For the first MVP, we tried to use KPIs that we could easily collect data and start comparing it with other time periods.
Below you can find a brief definition for each term.
Productivity
This bucket is focused on how we are doing regarding the time we spend designing and in how are we using our resources.
Maker Time
It’s defined as the number of hours a Designer spend doing actual hands-on design work, you can easily track the number of ours the designers use for these activities. With this KPI we now have a mechanism to measure a efficiency and remove blockers in our workflows and processes. One important thing is Maker Time is not equal to overall productivity, having a MT of 4 hours a day doesn’t means that someone is 50% productive, it means that maybe someone is spending too much time in meetings not having the right software tools or doing something redundant in a process.
The whole concept of Maker Time is very interesting and totally worth diving into in, it talks about Design as a craft, the creative process, designers schedule, creative space, etc.
Exposure Time
Exposure time is any time spent watching another person interacting with a prototype or product. The observation can be in a usability lab, at a customer site during a field study, or as a remote usability study. Jared Spool has a great article about this topic, our approach for this metric is based on his findings.
Team Health
This one is related on how are we doing emotionally, as you know this is a huge aspect in any creative role.
eNPS
This is a metric used to assess employee loyalty. The eNPS is an important tool in the arena of employee engagement and employee experience, it is simple to use and yields a single number that easily can show trends over time.
The eNPS is calculated by the percentage of unhappy employees subtracted from happy employees, as determined by their answer to the question “On a scale of 0 to 10, how likely would you be to recommend this company to a friend or colleague as a place to work?”
Job Fulfillment
Based on Margaret Gould article this qualitative metric is intended to be used by our Team Leaders as a tool for Designer growth. The outcome of the exercise is a spider web chart which can be used to identify: Opportunities for significant change, goals and areas where peers, managers or the company could help with Job Fulfillment
Team Output
Here we’re measuring the quality of our work, after all, we need to show value to the product development teams across the company.
Product Quality (Heuristics)
Having hard data about the quality level of our deliveries is a key part of our strategy, we need to ensure that our team is crafting the best experience for Dell’s digital products. Product Quality is defined as a value between 0 and 100 based on Nielsen’s 10 general heuristics, those are basic industry standard principles for interaction design.
Design Debt
Design debt is all the design concepts or solutions that were skipped in order to reach short-term goals, like its technical counterpart, it implies that the cost of having to go back and fix problems after launch is always higher than launching ideal solutions in the first place. Like tech debt, Design debt is often incurred when designers and researchers are working under tight timelines or project constraints.
Be sure to check this article by Michał Mazur What is design debt and why you should treat it seriously
Team Growth
Growth and Attrition Rate
These quantitative indicators measure the growth in the team’s headcount, these values are important to plan our resources, designer allocations on projects, scalability and licensing models among others.
Designer Developer Ratio
During the DesignOps Summit Keynote last November, Doug Powell from IBM talked about the Designer Developer Ratio as an indicator and how it can help in planning resources, also it’s a key insight on the participation we have inside the development teams. Again this metric speaks very clearly to our stakeholders.
The challenge: Assessment and gathering data.
After defining these KPIs we came to realize that gathering data for all of these indicators is not as easy as it seems, for a number of reasons known to everyone involved in Enterprise Ux some of these require heavy use of resources (ie heuristics assessment in dozens of ongoing product developments), some of them require buy in from our stakeholders (Move Design Debt to a Product backlog) and some of those take a lot of time because of their own nature (job fulfillment).
We’re following different strategies to get data for each KPI, but at the end, our approach is to demonstrate our team that the time they invest giving feedback is worth it.
What’s next?
Based on my experience so far, if you’re defining metrics for your DesignOps strategy I’ll recommend:
- Define a few KPIs (low hanging fruit) and start collecting data.
- Use that info to define a clear road map for DesignOps initiatives.
- Discover the major pain points in workflows and processes.
- Show the value of our practice to the rest of the company.
So far, this approach has been working for us and we’re starting to see results, we’re now exploring new, complex KPIs like Value Stream Mapping outcomes.
For us the key has been keeping an open mind, being flexible and LISTENING to the designers and stakeholders. After all we’re in DesignOps to Design for Designers…
Are you doing something similar? What metrics are you currently using in your company? Let me know your thoughts on this topic!.
