Could QE/QA be the Missing Piece of Our Software UX Puzzle?
In a previous story, Why We Never Seem to Have Enough UX Designers in Enterprise Software, I had a chance to share my deep respect and appreciation for my UX colleagues for their ability to make software products delightful to use.
In this story, I would like to shine a spotlight on another set of colleagues, our QE’s (Quality Engineers), a.k.a. QA (Quality Assurance), or often simply referred to as our trusty “testers” even though the modern day QE’s role encompasses so much more than testing.
While there is an abundance of best practices content regarding how a PM can best partner with their Dev lead or with their UX designer, there is far less content regarding how a PM should partner with their QE. There is even less mention of QE in one of the hottest topics of software development today — software UX (user experience).
This is a shame because, in my 30 years in Product Management, I have found QE to be every bit as mission-critical a partner to me as my Dev and UX counterparts in many aspects of building great software, and software UX should be no exception. Yet as I look at how enterprise software UX’s are designed and built, QE is often left out of the picture. In fact, our industry is so comfortable with letting PM+Dev+UX lead the charge that we sometimes affectionately call this trifecta “the three-legged stool of product development” — lacking any one leg, and our product would likely tip over. But how stable and robust is this three-legged stool really? Despite huge investments in user research and UX design in recent years by all of the major software vendors, mostly led by PM+Dev+UX, enterprise software remains notoriously difficult to use, indicating our three-legged stool is still a bit wobbly.
This suggests our UX challenge may be more than just a UX design challenge, and further investments along our current trajectory may not yield drastically better results.
In this story, I will put forth the thesis that our UX challenge is both a UX design challenge and a UX quality challenge, and if we want to conquer this UX challenge, we must pay as much attention to the quality of our UX implementation as we pay to the original crafting of our UX designs. And who better to help us deliver a high quality UX than our QE’s?
In other words, while our three-legged stool has served us well to address our UX design issues, it is time to consider an upgrade to a four-legged chair consisting of PM+Dev+UX+QE.
A Fun Little Case Study
To kick things off, let’s start by looking at a simple UI issue and explore how such an issue could have been allowed to reach the market. The sad truth is bad software UIs are so abundant around us it takes almost no effort to find a bad UI to talk about. Since enterprise software UIs are complicated and differ by industry, for this exercise I will simply pick on a consumer mobile app from Jamba Juice that everyone can relate to.
I was ordering a smoothie over the holidays using the Jamba Juice mobile app, and I came across the screen below. As eco-conscious as I try to be, it surprised me a little that the app defaults to “No Straw”. I couldn’t help but question how many smoothie drinkers consume their smoothies without a straw or bring their own re-usable straw.
Since I didn’t have my re-usable straw with me, I proceeded to click on the “Yes Straw” button. Instead of changing the selection from “No Straw” to “Yes Straw”, I was confronted with an error message in red (see screenshot below) telling me to “select one item” (i.e. informing me “Yes Straw” and “No Straw” cannot be selected at the same time). Even as a software guy, this error message confused me a bit at first, but I quickly surmised the developer must have used the wrong UI controls (E.g. they used multi-select buttons where they should have used single-select buttons or even just a simple toggle switch). This is a classic newbie mistake, but I couldn’t help but feel sorry for the average consumer who would have no idea why the app would make it so hard to order a smoothie.
I went ahead and deselected “No Straw”, and now the app is happy. No more error message.
But not so fast, I encountered the same issue again when I selected “Yes Split Cup”. Sigh.
Well, my wife and I wanted our smoothies, so I stopped obsessing about the UX and proceeded to complete the order, but afterwards I couldn’t help but wonder — this UI looks pretty nice visually so we can safely assume a UX designer was involved. Furthermore, the engineer who implemented this UI clearly knew the user must select only one of the two options; otherwise, they would not have given us the error message in the red banner. So how did the UX designer and the engineer manage to make even the simplest task so difficult for the end user?
Design and Quality
We’ve all heard the statements, “Quality cannot be tested in. Quality must be designed in.”
I am in full agreement no amount of testing can bring quality into a poorly designed product. That’s why we, as an industry, have invested so much into UX design. However, even a well-designed product must be tested to ensure high quality before it goes out the door.
In the case study above, there could be myriad reasons why the UI is as clumsy as it is, including some subset of (but not limited to):
- The UX designer was involved in the visual design, but they did not have time to work on the interaction design.
- The UX designer did work on the interaction design and specified using single-select buttons, but they did not have time to check the final implementation.
- The engineer knew they needed single-select buttons, but they wrongly assumed they could use multi-select buttons instead by simply setting the maximum_selections_allowed parameter value to one. Judging from the UX we saw, I think this is the most likely scenario.
- There was no QE on the project.
- There was a QE, and the QE caught the issue and logged a bug, but the bug was closed as “Works As Designed”.
We could go on and on because there could be a dozen more reasons why the UI turned out the way it did. The same could be said about any defect in software — nobody ever intended for them to happen, but they do happen, and they probably happened through some confluence of bad assumptions, poor or missing designs, miscommunication, rushed implementation, etc.
Acknowledging that defects are inevitable in software, an important follow-up question would be: regardless of why it happened, how did such a glaring UX defect get released into production and into the hands of thousands of users, without getting caught by the product team?
By asking this question, we now shift our focus from design to quality.
Yes, design is essential to great UX’s, but because so much can and will go wrong between design and delivery, quality is just as essential.
UX Quality Challenges and How QE Can Help
Having led multiple large scale enterprise software UX projects during my career serving some of the most demanding users out there (e.g. chat agents in customer support centers who routinely support 3–4 customers simultaneously and must multi-task so efficiently in the software that each customer feels they are getting the agent’s full attention), I have seen my fair share of UX quality challenges. I have curated the top 4 below to share in this story.
Challenge #1: From “Happy Path” to Whole Product
We have all been there. In order to get a new product off the ground, we had to build the first end-to-end UX design mock-up or a working prototype which is used as a marketing tool internally and externally to tell our story and to create excitement for the project. For expediency, and to not let details get in the way of a good story, we focus all our attention on a tightly scoped set of use cases, and we show off how effortlessly the user can navigate through the software along tightly scripted “happy paths”.
This is a very important activity. It not only helps to drum up support for the project, these initial designs also serve as a lighthouse for the look-and-feel of the entire product. The initial set of “demo use cases” also helps to seed the product backlog of epics and user stories.
If the initial demo use cases covered 5% of the total number of use cases, the challenge comes when it’s time to build the remaining 95% of the product. In the best case, the product team has the continued support of their UX designer throughout the project, and the UX designer works in lockstep with the developers to make sure the developers always have the designs they need, when they need them. However, in many product teams, the UX designer may only have bandwidth to produce designs for a “representative subset” of the use cases, and the developers are then left to “fill in the blanks” from there, which may leave a lot of room for interpretation.
When evolving from the happy case to a whole product, there are additional factors that may impact the UX. The product team may be consuming components from other teams or integrating with pages developed by other teams, and the resulting UX may not be evident until all the components and pages are stable and the integrations have been completed. Since UX designers and developers often focus on their own area of the product, QE may be the only member of the product team to get a comprehensive view of the resulting UX across teams and across all use cases, including edge cases and error cases.
To complicate things further, modern “responsive UIs” that automatically adapt to different device form factors must render correctly across all supported browsers and devices, and QE are often the only ones doing the validation.
In addition, in order to setup every aspect of the software for testing, QE also has a pulse on the admin experience as well, which often doesn’t get the attention it deserves during design phase.
Taking together all of the above considerations, we quickly realize the following:
QE may be our only team member who truly has a comprehensive view of the UX we are delivering, and they are in a unique position to review the quality of the final UX before it is delivered to customers.
Surely, PM, Dev, and UX can, and should, schedule periodic reviews of the product UX to make sure there are no big surprises arising from integrations, and we could certainly conduct some cursory “sanity checks” of our UIs on various browsers and devices, but we will never achieve the breadth and depth of coverage QE covers as they test the product day in and day out.
At this point, I hope it is clear the opportunity is there for QE to contribute in a big way to UX quality, but we must equip them and empower them to do so. This involves the following:
- Equipping QE with intimate knowledge of the target end user(s) and how they work, so QE can be an effective advocate for the end user. (Unlike the Jamba Juice example, which everyone can understand because we are all smoothie consumers, we cannot assume the same for enterprise software.) This means involving QE in some of our user research activities, such as user interviews, user shadowing, and UX validations.
- Involving QE in UX design reviews so they know not only the spec, but the thinking behind the spec, so they can raise a flag when things are not working as intended during testing.
- Empowering QE to “speak up” when they have a concern about the UX. They should feel empowered to log their UX concerns as bugs, and “works as designed” can no longer used to dismiss valid UX concerns.
To a lot of product teams, this is nothing short of a culture change. You might be thinking, “There must be a reason we haven’t done this in the past. This is going to be hard.”
Actually, I have good news to share. I have done the above with QE on different projects at different companies, and never once have I been told I shouldn’t/couldn’t do it. Quite the contrary, QE is always extremely appreciative when they are invited to join in on user research customer visits and UX design reviews, and the product team is often impressed by the value QE brings with their “critical eye”.
While PM, Dev, and UX are all primarily focused on building the product, QE is always looking for ways to break the product, so their “critical eye” offers a very different and enlightening point of view.
To the QE’s I worked with, if you are reading this story, please feel free to chime in and share your experience in the comments.
Wow, this first point took more ink than anticipated. I promise to make the subsequent points short-and-sweet.
Challenge #2: Big Rocks vs Small Pebbles
It is one thing to identify “UX bugs” and log them. It is a totally different challenge to actually prioritize them and fix them. While QE can do the former, their efforts would be for naught if there is no resolve from the PM+Dev to do the latter.
This is often a challenge because many product development teams are eager to wrap up the current project, so they could move on to the next shiny new project. They are far less enthused to keep fixing a long list of UX bugs.
In some organization, this mentality is further reinforced by planning processes that prioritize “big rocks” (the shiny new initiatives) over “little pebbles” (things like small enhancements, technical debt, and bug fixes). The concept is borrowed from The 7 Habits of Highly Effective People, by Stephen Covey. The theory is that you can slot in small pebbles around big rocks, but not the other way around. While the goal of this approach is to ensure there is budget for large and impactful initiatives, which is a noble goal, but in practice it often puts UX quality projects at a disadvantage.
UX quality issues usually do not present themselves as big rocks, but as buckets of small pebbles.
In order for the efforts from QE to pay off, and for us to successfully deliver high quality UXs, we must find a way to prioritize these buckets of UX bugs alongside the big rocks and not treat them as individual pebbles that can be arbitrarily dismissed.
Challenge #3: Regressions
A lot of UX quality issues are due to regressions, where a software enhancement or bug fix inadvertently breaks an existing UI. Since this is impossible to catch without retesting the product frequently whenever changes are introduced, the only practical way to safeguard against regressions is via automated testing.
QE must be given sufficient time and resources to develop automated test suites for this purpose.
Challenge #4: Performance
Even the most efficient software UI design could become unusable in production due to slow performance. This is one of the most common gaps between customer expectation and reality. The initial demo always appears lightning fast because there is only one user on the demo system, but in production there may be tens of thousands of users using the system. With proper planning, QE can test the performance of the software under load to ensure acceptable performance, as well as model the performance profile of the software to ensure new systems will be equipped with sufficient server resources to handle peak usage patterns.
Note: Due to specialized skills and resources required, performance testing is often done by a centralized performance engineering/testing team. In this case, our QE can still help by acting as a liaison, helping them to define the most meaningful use cases and performance metrics for the product at hand.
Building the Four-Legged Chair
Having been on the front lines applying, and even evangelizing, the three-legged stool concept, I am very proud of the progress our industry has made to address the problem of poor UX design. Without the gargantuan investments we have made in UX design, we would not be where we are today. As we said earlier, quality cannot be tested in; it must first be designed in.
Our advancements in UX design, led by PM+Dev+UX , has put us in a position now to talk about the next step, which is UX quality.
Although change is hard, from my experience this particular change of recruiting QE to help ensure UX quality doesn’t need to be. As the PM leading a product team, this is a golden opportunity to assert our leadership to facilitate an important change that will benefit the team, the product, and our customers.
I shared my favorite UX quote in a previous story, but it is worth repeating here, especially with the Jamba Juice example fresh in our minds:
“It is easy to make it hard, and hard to make it easy.”
— Unknown
Let’s make it easy!
