Measuring DesignOps
Appeasing the bean counters.
I have been touring around the world (sorta) leading workshops to help introduce people to the workings of DesignOps, and how they can be moving forward with their own DesignOps initiatives. Early in these workshops I offer everyone the opportunity to tell me why they came. The top reasons are:
- What is the ROI of DesignOps?
- How do you get buy-in?
- How can designers and devs work better in “A”gile organizations?
There are other questions for sure, but these always rise to the top. What’s interesting is that I never had the first two in the workshop before. The last one, I definitely cover.
Since people have been asking, I have been answering. One of my favorite activities is to answer questions on the spot. I’m serious. It’s like playing a game show. But doing this helps me evolve my thinking and construct new frames to better help people understand the core of what I’m trying to communicate. In the case of the first question, I think I’m almost there, and this is a further attempt to sketch and learn.
I must admit I have a negative reaction to the acronym, ROI (Return on Investment). I hear this request all the time. What’s the ROI of design? What’s the ROI of DesignOps? What’s the ROI of standup desks?
So let’s look for a moment at this. What is being asked is defense and justification. These are businesses, of course, and profit is generally the primary goal of almost all initiatives in an organization.
What gets me in a knot about the question is I almost never hear it asked of Engineering and Product Management. Now I’m not saying that these orgs get everything they need. Most are sorely over-worked. But the way these decisions are quite different. They are just budgeted and resourced against those budgets. We hope to make this much revenue on our product given these sets of features. If we want to make a margin large enough to justify our existence, we can only afford this much resource in people, tools, and supporting structures. When we ask what the ROI is for design, or designOps what we are saying is that design is a value-add, and not a core of the company. We are implying we can live without it. E.g. if there is no appreciable, understandable return, then we don’t need it.
The other issue I have with this, is accountability. Totally makes sense for a product team to be held accountable together for meeting a well understood and measurable product, business goal. But to suggest that only design by itself magically increases product value is kinda absurd. Of course it is a team sport, and even if your defense (sorry, sports ball) is awesome at reducing points, if your offense doesn’t score at all, the TEAM still loses. Your designs can be amazing, but if dev can’t make it, and the operations don’t support it, then what’s the point, right?
So stop with the ROI, and move to asking “how do you measure”?
I think it is incredibly fair to ask the question, “How do I measure the success of my initiative?” It would be irresponsible to not have a way to appreciate in some way your success/failure.
There are other parts of your operational organization that have this problem and have found great ways to overcome the need to measure qualitative criteria in ways that are seemingly quantitative and thus easier to appreciate. One such part of the organization is the team that manages people — often called Human Resources. For example, a common measure for folks managing people generally is “Engagement”. How do you measure a quality statement like engagement? Well, through likely correlations to other criteria, and then the trend across those correlations.
A great tool that helps orgs do this is Culture Amp. It is a bit tedious, so I don’t enjoy it as an employee too much, but it is kinda painless. As a manager I totally get the value it provides. It is a simple survey, where employees rate their (dis)agreement level to a host of questions across different categories. Answering just one question won’t get you an answer, but averaging all of them, give you a picture with much higher confidence that the answer is accurate.
The method here is to take any category of measure, e.g. “trust in my manager” and apply it to the entry, “I trust my manager has my interest at heart.” 1-strongly disagree < > 5-strongly agree. Keep asking these types of questions or a similar format and you can measure it.
A similar method is how Forester and others suggested measuring heuristics on a similar scale. I applied this to how to measure the quality or maturity of your ResearchOps.
Now take that framework and apply it to a measure of success you believe your DesignOps will have as it matures.
Is the product organization aligned in their understanding of the value of designing?
There is no alignment across the product organization.
There have been gains in alignment seen by open trials of design and research activities and processes.
Alignment is growing, as seen by more non-designers participating in design activities.
Design value is well understood and consistently articulated across the product organization.
In many ways ResearchOps is way more mature across the board than DesignOps. It has the advantage of stronger focus. This is why it was easier for me to come up with those qualities of measuring maturity than it is for DesignOps. At this point, all I can offer is that as you create DesignOps initiatives, you should create some success criteria and the questions you need to ask and answers you need to get to to advance from nothing to awesome over time. The criteria will be different per organization for the next couple of years, is my guess. But you’ll still need to be able to do the work.
Besides the heuristics of course, make sure you create outcomes like …
- Is the percent of design intent to product execution increasing?
- Is attrition among designers reducing?
- Is design increasing its involvement as a valuable tool in strategy development?
etc.
I realize for many this won’t get them that ROI answer they are begging for, but for many more it will help them create a plan of action with measurable goals that will help them be held accountable to these initiatives success in a way that their leadership can appreciate and respect.
Does your team have “vital signs”?
Another metaphor I like is that of medical vital signs. In many organizations there are monitors with dashboards of measurements and other information strewn throughout an office. These measures are easy to interpret as and compare to vital signs that a patient has in a hospital setting.
When I was working in infrastructure IT, it was clear to me that support personnel has this mindset as well. They required a dashboard of their infrastructure so they can determine the health of the system. It wouldn’t tell them what was wrong, but it would guide them where to look. A fever doesn’t tell you by itself what is wrong, but it tells you for sure that something is wrong.
Similarly there will be less volatile measures that you can put together to determine if your operations are working well that your Design Operator can keep tabs on in a similar way that a Program/Project Manager might have a Jira dashboard they keep open to track velocity, estimation accuracy, etc.
Whatever you do to help observe success/failure in a way that is reportable to many, you need to be sure to start all initiatives with this in mind. Designing any system requires also designing the instrumentation of that system.
