Getting questions about the ROI of UX?
3 ways to refocus the conversation to opportunities
by Aaron Powers, Sr. Manager of Experience Measurement, & Scott Mackie, Chief Experience Officer
Alan Cooper & Jared Spool are both giving you bad advice about the ROI of UX.
We never thought we’d start an article that way — Jared & Alan are both luminaries for the UX field who’ve delivered a lot of good advice over the years. But Alan Cooper recently wrote that if management ever asks about the ROI of UX, it’s time to get a new job. Jared Spool replied that getting a new job is a good idea but may only work for the most privileged. Both these approaches have an assumption in common — they assume that when management asks to quantify the value of UX, this is a bad sign. Both Alan & Jared think teams shouldn’t need to have this discussion. We strongly disagree!
Let’s start with some things we can all agree on: many UX’ers hear a lot of desire to quantify the value of UX. That question does inherently cast doubt on the degree of value that UX design delivers and therein our individual contribution. Note the italicizing of degree — if you’re an in-house UX designer then someone relatively senior at your company, at some point, made a decision to stand up that function and hire full-time design employees. That’s undeniable proof they see at least see some ROI from UX!
Let’s move to something we should agree on: All well managed businesses should constantly be thinking about the best possible marginal allocation of resources. Where should we spend our valuable dollars to grow and retain our valuable customers? It’s natural for teams to advocate their own cause over another, and rational evidence-based debate between advocates should help a business arrive at an optimal allocation strategy. If you respond to the challenge well, then it’s a great opportunity to help grow your team & improve your business.
Here are 3 concrete tips for turning the sour grapes of ROI conversations into fine wine.
— —
1/ Debate the value curve for UX for your business, product or project, not the value of the entire field
Designers could interpret questions about ROI as an attack on the entire discipline, and indeed some antagonists will frame it this way. Don’t fall into that trap. Both Alan & Jared talk about studies that show the value of UX — but citing these studies to your management team will not answer their real question. They are not interested in whether UX design is sometimes helpful in the world of digital products — they hired you because they already agree with that in principle. Management is asking about the quantification of UX because they really want to make decisions about staffing and product strategy trade-offs for their business. Sometimes it’s higher level — how many UX staff should a company have and what parts of the product should get the focus? Other times it is more immediate — should we hire another designer or another engineer? Should we spend the next 3 sprints on a better workflow for our existing functionality, or on adding new functionality? Should we delay the release by 4 weeks to give the team time to improve the usability? There is a decision that needs to be made and these stakeholders use the phrase “ROI of UX” in a call for one overarching piece of data that could help them make those decisions each time they arise.
UX Design used to be a somewhat siloed function at our company. UX was a cost center, and designers were doled out by central design leadership authority. Individual designer allegiance was to the UX team and their goals were sometimes in conflict with the goals of the business. As part of our shift to a scrum team structure, two things changed for the better.
First, we chose to stick fairly firmly to a “1 designer per scrum team” model which meant some scrum teams had to go without — a painful situation for everyone (read more here). Second, product design headcount budgeting got matrixed with product business unit budget. Initially we all hated this, it took full hiring authority away from design leadership and made life a lot more complicated for product leaders and finance alike.
Over time however, the value of these structural constraints and the ways they force everyone to consider the ROI of UX have become apparent. When a new scrum is spun up to pursue an opportunity, the scrum-level product owner and product leaders need to decide whether to include a designer in that team. They have to make that trade-off decision to assign 1 FTE of budget to a designer (rather than another engineer or PM), and they’ve now felt the pain of teams forced to go without, that built a beautiful back-end engine that nobody could properly use or who stalled out wondering what problem they were really solving or how they’d actually bring their idea to life. Before, UX headcount was sometimes seen as wasted resource, now we have engineers and PM’s anxiously ensuring they get their designer and associated returns!
— —
2/ Find proof points within your business
These can be large scale, even approaching the big ROI question, or small scale anecdotes from users or first hand experiences that help business leaders viscerally understand the value.
In 2017 we finally gathered and properly analyzed our User NPS data against key financial outcomes of client attrition and successful client referrals. Previously, no useful correlation had ever been found in our Buyer NPS data. The end User data showed a 6:1 odds ratio for attrition in clients, an especially strong relationship where mean Likelihood To Recommend was very low. Better still, end user assessment of usability correlated more strongly with that Likelihood to Recommend than other areas such as product reliability. Having this data didn’t end all conversations about ROI, far from it, but it made our arguments much stronger. We’ll publish a full article about this regression and mediation analysis soon, and about the additional work we have yet to do on this front.
While we believe in the power of quantitative data (geeky pun intended), some of our greatest successes in changing minds have come from the heart. In mid 2017 we ran a cross-functional leadership event to share findings of a large UX heuristic audit of core workflows across our product (read more here). We played soundbites from users who struggled, we did cognitive walkthroughs and gave them a 101 course and practicum on Jakob Nielsen’s 10 heuristic principles. Senior product leaders laughed and cringed at how bad some of our product experience was, and mostly took it to heart. Conversations have been a little easier ever since. On the flip side, recently a design leader compiled a video of a beta user glowing about our new mobile workflows, saying: “Alongside getting a dedicated nurse assistant, this is the biggest improvement to my practice of medicine in the last 10 years!”. That video went viral internally and now gets quoted by other disciplines. Most designers have experienced this visceral impact on stakeholders at some point but we often forget, and fail to work to create these moments. Human-centered design works on business leaders too!
— —
3/ Measure UX impact — the good, the bad and the ugly
Over the last year we’ve worked very hard to modernize the behavioral analytics approach at our company. At many places the product management or engineering teams lead this effort, but because our cloud-based platform provided a modicum of insight via server log data the product development teams always just “got by”. We recognized that to do our UX design jobs well we needed better behavioral data across the user base. We pushed for a new vendor to get HIPAA compliant (thanks Amplitude!), spent our own budget dollars on an engineer to stand up an internal platform for using the tool, and advocated at every level of the organization.
Now we’re starting to be able to quantitatively show the bad (how long users get stuck in complex workflows, how many times they fail) and the good (how those issues vanish once the workflow is simplified, and how many hours of user effort we’ve saved). We’re also able to show the ugly: the times when our design efforts did not have the impact we intended or wanted. But even that has its benefits. When we suck it up and admit failure, we earn our stripes alongside the product owner whose initiative failed and the engineer whose code caused a crash.
— —
A parting note to Alan & Jared: rebut this if you wish; let’s start a conversation. And if you want a really antagonizing aspersion to tackle, we’ve got one for you! We get concerned when someone starts saying that the proper role and value of UX is solely in usability and aesthetics — we’ve all been there: “hey UX, we built this already so polish it up please”. We know that the greater value lies in helping discover and define the best product to build — you know, the one that is worth prettying up!
