Design X
… or there’s no limit to perfecting the design process
The human designer has always strived for systematization and creating a clear path of professional development. This has led to the growth of numerous courses, training programs, and competency matrices. Every self-respecting company should have such a model, otherwise, it’s very difficult to synchronize expectations and effectively develop new talent. The success is hard to overestimate: people eager to enter the profession bring their hard-earned money and exchange it for a clear growth plan. More mature designers have an adequate understanding of their level and can receive coherent feedback, instead of “Well, something’s not right, maybe grow for another year…” For leads, it’s become easier to track progress and argue for promotions with the higher-ups. Everyone’s happy.
But if such a model exists for designers themselves, why isn’t there one for products? Well, why not? In our company, we’ve had a direction called EngX (Engineering Excellence) for quite some time. It’s a special division that uses a special framework to audit so-called Product Delivery Pipelines, i.e., the development process. An auditor is embedded into the product team, who spends a week talking to everyone in the team, looking at productivity indicators, how development and testing processes are structured, how well everything is covered by tests, how developers interact with each other, whether there’s any knife-fighting at meetings, etc. During their stay with the team, our auditor makes little pencil marks in their notebook, and then generates a report, which is the end product of the service. It shows the maturity level of the development process, where its blind spots are, how much these spots cost the product each month, how much it costs to cover these spots, and what the final ROI will be. In the end, both the product and the team win, if not from specific actions, then from the awareness of their weaknesses.
I’ve described it and I’m already drooling! We in design “need this too”! Agree, we most often look at the product itself, trying to eliminate all interaction problems to deliver more and more value. This is logical, as the product brings in money. However, we spend little time on how exactly we deliver this value, and how we can do it more efficiently. Think about it, I’ll continue the story.
Realizing the importance of analyzing the process itself, we thought: “Why don’t we develop our own evaluation model? With graphs, recommendations, red spoilers and casting!” But we wouldn’t want to do this just because we can’t resist measuring something. We’d like to benefit everyone who will measure their process with our ruler, so that both the business gets its penny and designers find it easier to work.
Long story short: we developed such a framework, and if you’re itching to test it, scroll to the very end where it’s explained how it works and how to pass it. If even shorter — write to me here or on Telegram and we’ll agree on how to pull off this dark deed. And if someone wants to savor and learn what kind of framework it is and how it was created — read on.
What are we dealing with?
Before analyzing anything, you need to understand what this something is. In our case, there are a great many variations of the design process, but if you peel away the husk, they’re all the same under the hood. Like a Lexus — it’s the same Toyota, but for more money.
So, we took the three most popular models (Double Diamond, Lean UX, and the Stanford model) and tried to overlay them on each other, while also fitting the phases of actual delivery alongside. We ended up with this scheme:
After the great merger, as you could observe, we detailed who does what and how, to better understand how this can be evaluated. We ended up with a rather interesting model that can be used as a wall image of a spherical horse in a vacuum: you’re unlikely to find such a process, yet nothing prevents you from aspiring to it. The defragmentation was successful, now we needed to carve out a ruler.
Framework Development
Frankly, the idea of developing a maturity model for products didn’t only come to our heads. This cannonball had already been approached by Nielsen/Norman Group, McKinsey, Design Management Institute, and even InVision (RIP).
All of them fulfill the task I described above — to assess the maturity of the design approach in an organization or product team. And yet, they all answer this question differently: some from the perspective of integrating design into the overall delivery pipeline, some from the product design point of view, some from the angle of infogypsy consulting. But none of them aim to help people do their work more efficiently. In other words, our colleagues in this dangerous business see it as a tool to sell additional design services, not as a way to help optimize the work of designers themselves.
So we came to the conclusion that we couldn’t “steal like artists” — there was nothing to steal, and we’d have to carve our own Pinocchio. As a result of collective brain-straining efforts and analysis of empirical data, 13 categories, or dimensions if you will, were born, which we will consider:
1. Business stakeholders relations. Everything related to relationships with business stakeholders, their support, communication opportunities, trust, etc. This cross-section serves as a litmus test for the quality of collected requirements, client trust, and expectation alignment, which directly affects the final result. After all, no one wants to hear the phrase “This is not what we wanted!” during a presentation.
2. Product strategy. This section encompasses everything related to formulating a long-term product vision, a clear strategy for its implementation, and understanding how much the team shares the set course. Understanding the goal helps with competent planning and increasing the efficiency of the design process, while the absence of a reference point dooms the product to reactive design. Just going with the flow.
3. Opportunities creation. This category includes concepts such as team autonomy, defining business value and positioning, legacy level, and staff turnover. All these seemingly disparate indicators help determine how quickly a product can innovate, delivering additional value to users and business.
4. Decision making. Determines on what basis decisions are made, how they are recorded, and how often they change. Abandoning unprofitable strategies is a show of strength, but if decisions taken for implementation are constantly changing, it negatively affects both the inflating scope (inability to plan) and the effectiveness (not efficiency) of the design process, as work will be done in vain.
5. Data relations. Everything related to working with quantitative data: sources, influence, the role of the data analyst, etc. This cross-section gives an understanding of how much we can rely on quantitative data during the design process, and thus save resources on confirming hypotheses.
6. User relations. A category that determines how well the team knows its users and how they work with the product. Without understanding users, the design process becomes meaningless and starts to resemble roulette, where we place bets and try to guess whether users will like it or not.
7. Planning, prioritization. Here we have long-term and short-term planning, visibility of design scope for stakeholders, prioritization, and everything related to them. This slice gives an understanding of how viable the product is in the long term and how effectively the design team can distribute its efforts to make maximum impact in minimum time.
8. Design process. It smells like recursion, but this title refers to the applicability of advanced design process methodologies, as well as the process of working with innovations. A chaotic design process can also work, but that doesn’t mean it works at 100%. This plane gives an understanding of untapped potential that can be used for the benefit of the product.
9. Design delivery optimization. Here we have quite low-level things, like documentation, versioning, use of real data, and other seemingly small details. But you have no idea how much impact they can have on work stability, delays, and reworks.
10. Design system. The name speaks for itself — everything related to building and working on global and local design systems. It’s all quite obvious, as atomic design (in one form or another) has already become a necessary standard to ensure consistency of experience and speed up design.
11. Design evaluation. Here we have everything related to checking design solutions and evaluating their effectiveness. This includes moderated or unmoderated tests, work with product KPIs, etc. Everything that can minimize the risks of doing something wrong and wasting the budget.
12. Design team. Everything related to building work within the design team: ceremonies, processes, and communication. All these components are no less important than everything described above, as the team climate incredibly strongly affects its effectiveness.
13. Cross-functional collaboration. Similar to the previous point, but in relation to teams of other disciplines: BA, developers, testers, etc. For the same reasons: to assess whether synergy arises between teams and how effective they are.
As you can notice, we didn’t focus on specific research activities, user interactions, or project phases, as our main focus was on work efficiency, not its nature. Of course, within each category lie more detailed criteria, but I won’t describe them here.
Evaluations The first version of our framework included 13 categories, with 6–7 criteria in each. The ratings were universal for any of them, ranging from “nothing exists” to “exceeds all expectations”. Our desire for simplification and unification (read: “laziness”) played a cruel joke on us, but more on that later. According to the original plan, we wanted any designer, lead, or manager who filled out our evaluation sheet to receive a clear numerical rating value, indicating how pioneering or sucky their project was, and where the main areas for improvement lay. Thus, we would have burst into the product’s design process with both feet and immediately inflicted irreparable benefit. Armed with youthful maximalism and a hastily assembled template in Coda, we moved on to testing.
However, after testing on 30 teams, it turned out that the idea was only good on paper. Reality proved to be harsher. Here’s what we learned:
- Lack of details gives too much room for interpretation of criteria and ratings (it’s unclear what rating is for each criterion, what it means),
- The rating system works for creating an overall impression, but not for an objective assessment of risks and opportunities (We have 50 points out of 100, we’re somewhere in the middle, but so what? Why do we need 100?),
- A too subjective evaluation model gives free rein to the inner overachiever, ready to tear throats for an “excellent” grade. The fear that the assessment would be lower than internal expectations made people misinterpret questions and ignore the “bitter truth”.
But we weren’t the kind to give up so easily, so after a few sleepless weeks and a couple of truckloads of Red Bull, detailed descriptions for each answer option appeared in every question. All to eliminate ambiguity in interpretation and make the assessment more accurate. Now no one could say, “Well, I know what features I’ll be doing next week, so the product vision is at a level exceeding all expectations.” There was only one small thing left — to figure out how to show stakeholders how much money they’re losing in the sagging areas.
Here’s the translation, maintaining the original tone and style:
**Risk Analysis**
It’s quite a task, because in development you can focus on clear indicators of process efficiency, but in design, it’s not that simple. In development, we face quite specific tasks with defined success criteria. Like, here’s what we’re doing, here’s how it should work, here’s how long it will take. Variables can change, but the delivery pipeline can be improved to make the development process faster, less buggy, and more transparent. In design, however, we still need to understand whether this solution will be beneficial, or if it shouldn’t be implemented at all and money shouldn’t be spent. This is just one example. We operate with not-so-clear criteria, as both the size of initiatives and the resources spent on them can vary wildly. It’s clear as mud.
So instead of trying to express everything in money right away, we decided to introduce a layer of risks that can arise as a result of a (crappy) not-so-great process. Based on empirical data, we set the basic probability of their occurrence and built dependencies with criteria. Thus, we got a chain of “Bad ratings → Increased risks → Increased probability of losing money”.
For example, the lack of a formulated product vision entails the following risks:
- Missed Product-Market fit,
- Increased Time to market,
- Reworks,
- Budget deficit.
This can be converted into money using the number of FTEs (price of a workplace) involved in the product development iteration (nFTE) and the duration of this iteration (tITR).
- Increased Time to market: actual number of iterations needed to close the need × Duration of iteration (k×tITR).
- Reworks: actual number of iterations needed to close the need-1 × number of people involved in the venture ((k-1)×nFTE×FTE).
- Missed Product-Market fit: development time × FTE, because we made something that nobody needs (k×tITR×FTE×nFTE).
- Budget deficit, as a result of all of the above, can lead to the death of the product.
Thus, we don’t make false promises that your product team will start generating incredible millions of dollars a week after the audit, but instead give forecasts regarding current data and risks that this may lead to.
Did I lose you two paragraphs ago when I started hitting you in the face with some incomprehensible formulas? Let’s do it street-style. If a specific step in your design process isn’t up to par, it can lead to negative consequences… Or it might not. It’s all probabilities. But each negative event definitely has a price. For example, requirements changed → you can’t give the design to the team → the team smokes and waits, and money keeps dripping. And depending on how many people smoke and wait, and how long they smoke and wait, the amounts change. So we looked at the dependence of design process steps on all the nastiness that can arise and estimated the probabilities of big and small “floods”.
Yes, a shrewd reader will notice and reproach that we used Bayesian statistics (empirical data relevant to our context with some personal bias). But we don’t regret it. While this data doesn’t claim research accuracy, it gives an approximate idea of the scale of losses. In the future, if the framework survives, we will undoubtedly refine the parameters, but for now, we’re more interested in solving problems, not meticulously assessing the probability of their occurrence.
To avoid burdening stakeholders and teams with all this “risky” model, we took another step forward and identified three big and important factors to pay attention to in the end:
- Delays — how many extra iterations occur before achieving results.
- Productivity — how much work is underperformed per iteration.
- Quality — impact on product quality and viability.
All risks somehow boil down to these, so that later you don’t have to sort all the negativity into piles yourself.
Recommendations
For each project, product, team or organization, recommendations will differ depending on the domain, context, available opportunities, moon phase, and your stakeholder’s zodiac sign. There are many variables, so a silver bullet capable of finishing off all troubles and adversities in one fell swoop is not possible.
And yet, we’ve prepared a list of general recommendations relevant to raising the overall score. We don’t consider it as the ultimate truth, but rather as a basis for starting intellectual work.
No doubt that after the audit, you, the team, and stakeholders will want to gather, surround the evaluator (whoever takes on this role) with your opinion, and generate solutions on how to improve the indicators. This noble impulse should not be suppressed, but should be supported, because the path of selling through involvement is less thorny than constant butting heads: generated together — implement together.
From my side, I want to note that usually, after the first iteration of the audit, you’ll find more gaps than you can fix before the next meeting with the auditor, so you need to take a philosophical approach to them: prioritizing and adding to the backlog little by little.
Maturity Levels
A by-product of analyzing the results was defining the maturity levels of the design process on the product. We came up with something akin to a skill matrix for designers, but for projects.
Briefly, we identified four levels of maturity: reactive, collaborative, data-driven, and holistic. And again, we need to make a disclaimer that different maturity levels don’t necessarily reflect the “coolness” of the team. Small teams don’t need these cumbersome processes that will slow them down. For enterprises, these hullabaloos will be simply necessary for normal functioning. So don’t get upset if you didn’t get the level you wanted as a result of the assessment — maybe you don’t need it. Use common sense. But to make it easier to understand, let’s describe each of the levels.
Reactive design
Designers reactively follow input data and stakeholder opinions, without delving deeply into problems and their impact on the user or business.
The product simply functions based on managerial intuition and pursuit of process efficiency, which takes the top spot: how many tasks were closed and at what cost. Assessment of value and benefits is not considered.
Collaborative Design
Designers work in collaboration with stakeholders and the development team to achieve the best experience for their users. However, they use internal opinions about who the users are and what they want.
The product functions through collaborative discussions, prioritization, and predetermined processes. But real user and business needs are hidden.
Empowered Design
Designers work within the organization and know their users well thanks to constant collection of quantitative and qualitative research.
The product understands the value of design and considers designers as user advocates. The product area responds to user needs through hypothesis testing.
Holistic Design
Designers and stakeholders equally contribute to the product’s success, understanding the ultimate value not only for users, but also for the business itself and the entire company ecosystem.
The product functions with a clear vision and strategy, a clear understanding of the priorities of each function and its contribution to user value, business impact, and technical feasibility.
For each of the levels, we developed our own expectations for each of the criteria. You can even compare them with your results if you’re patient enough to go through the assessment (and for this, just contact me).
How does the review process work?
One. We use our developed framework to evaluate every little piece of your design process. But we can’t “drop the goose” everywhere at once, so we have to gather information piece by piece: first filling in the gaps from available information, and then clarifying as we go.
Two. We send out invitations to all “suspects” we’d like to talk to in order to clarify all uncertainties and testimonies.
Three. We move on to Measurements and fill in all the info we know about the project. This data is very necessary for an accurate forecast of the loss amounts we’ll calculate along the way.
Four. We move on to Assessment and start going through the entire process criterion by criterion in order. And so, we iteratively fill in all the gaps. Each interview takes about 40 minutes. And no, I haven’t lost my mind :)
Five. When everything is ready, we move on to preparing the Summary, where we give forecasts on losses, as well as an overall scoring with a list of recommendations.
Six. If it becomes interesting to know at what level of maturity the organization’s process is now, we do a comparative analysis of the Maturity level.
Seven. We discuss the results with the team and conduct a brainstorming session to prioritize improvement ideas. All that’s left is to follow the plan. The next check-up is in 6 months.
Remember to send part of the saved money to the charity of the article’s author (Just kidding).
Conclusion
What profit did we get from this giant exercise in our organization? In addition to the data collected during the pilot, our framework is just now turning into a separate service and is being integrated into EngX, which I wrote about at the very beginning. Thus, it can be used as a distinctive offering or as a module of one huge comprehensive audit. In short, we’re about to save everyone so much money that there’ll be enough for a whole “Thank you”.
However, it’s too early to pop the champagne, as the growing demand needs to be satisfied by someone. This leads us to the thought that we need to train a group of auditors capable of conducting such reviews in the required volume, and this is a task of a completely different level. So we’ll keep in touch with everyone who’s interested and share our developments.
As per good old tradition, I’m leaving here a link to my cozy Telegram channel. Happiness, health, and a love boat to you all. 🛳❤️
