Evidence-based design

What do we mean when we talk about Evidence-Based Design (EBD)?

Many agencies talk about designing using evidence or insight (or sometimes just the amorphous “data”) for better outcomes. Some — like us — use the term evidence-based design.

To say that basing design on something tangible makes for better outcomes isn’t really controversial (even if in reality there’s still a gap between good intentions and what happens in reality under a deadline).

At WAS we say Evidence-Based Design (capital E, capital B, capital D) is our core product. This is what we mean by that.

What do we mean by EBD?

When we talk about EBD, we’ve sometimes used the image of a pyramid, or a Maslow-esque hierarchy.

Evidence-based design hierarchy: Mindset, Approach, Toolkit

First comes having the right mindset. Then, a general approach. Finally, a toolkit for the day-to-day tasks.

This is a recognition that the real world is messy and no model will fit every situation. We don’t like to draw red lines too early. If our hearts and minds are in the right place, that’s a good start.

The EBD mindset

Evidence-based digital product design (EBD) means adopting a common mindset, one that takes the best of both the scientific method and design thinking.

EBD acknowledges intuition as a rich source of ideas to be tested, and complements it with the discipline and rigor of the scientific method.

It means prioritizing at every level. Balancing evidence and assumptions to make decisions, solve problems and predict outcomes. It means ensuring that we measure (or at least observe) the effect of our work to continuously improve our understanding and produce ever better designs.

It encourages action and a ‘build first’ mentality by providing confidence in good outcomes and making new ideas safe-to-fail. It means being comfortable with uncertainty.

It means using (or at least thinking in) the taxonomy of EBD: Problems, Questions, Goals, Outcomes, Ideas, Assumptions, Tests, Evidence… (more on this later).

The EBD approach

In circumstances where it still isn’t appropriate to adopt the operational tools of the whole framework, it’s still possible to approach a project or challenge as EBD.

There are a few principles to follow.*

1. Support every decision with evidence
2. Use the design process itself to generate evidence
3. Recognise there are multiple possible solutions to any problem
4. Keep it simple
5. Do the most important thing first
6. Start at the end

*NB. The framework is very much in permanent draft. There may be other principles we haven’t discovered yet, or some here that will at some point be retired. But this is our best map of the world at time of writing!

1. Support every decision with evidence

Decisions and ideas always have a justification — some are based on research and insight, others intuition and experience. These are all types of evidence.

Some evidence bases are stronger than others. Calling this out and making the justification explicit holds us to account and shows us where the gaps are.

Gaps aren’t always bad — it can provide more opportunity space to explore.

Where evidence is lacking, we call out the assumptions. It’s about being honest, but it’s also useful: we’re mapping our risks and writing our test plans as we go.

2. Use the design process itself to generate evidence

This principle will be familiar to anyone who likes a test and learn approach.

Much as research can be deductive (testing an existing theory) or inductive (observing in order to construct a theory), so can design. We might want to adopt an exploratory mode and lead with questions, or get our idea out in the world quickly and cheaply and see what happens.

The important thing here is to construct your tests in a way that generates evidence, rather than ‘data’.

Evidence is directional: evidence is evidence of something.

Data is just data. Ready to be put in any order to prove any point.

3. Recognise there are multiple solutions to any problem

Rejecting the problem-solution binary is important. Two designers given the same problem may come to entirely different solutions, and that’s good.

If we assume there is one “correct” way to solve any problem, we’ll drive ourselves to distraction chasing it. Or, assume we’ve found it and stop trying.

Design isn’t a contest or a race, but employing creative means to solve problems and create outcomes.

4. Keep it simple

Occam’s razor: the best solution is the one that carries the fewest assumptions.

Complicated solutions have a tendency to bake-in debt: design debt, research debt, technical debt. We want to solve problems, not create them.

This isn’t easy: it can take a lot of work to cut through the noise and find the simple solutions.

5. Do the most important thing first

In EBD, we always prioritise. On the basis that time and resources are always finite, our focus should always be on achieving our desired outcome.

Everything else gets carefully noted, logged and put away to solve another time.

This means starting the design with the core: the stuff you can’t take away without breaking the concept.

Usually this will require working out of sync with the user journey. The user will experience a login screen first, but this is probably the last thing you should design.

Why? Well, login flows are basically all the same. If we’re pushed for time, we can pick something off the shelf. Also, how can we effectively design a login screen until you know what the user is logging in to?

6. Start at the end

The last principle is also sort of the first. Start with the outcome you want to achieve, and work backwards.

This applies to the project, but also the week, the day, the task.

Start with what you want to get done, then do it.

This is crossing the river by feeling the stones. We can be indirect, leisurely, meandering. But we have to know the destination.

The EBD toolkit

If you want to go full EBD, there’s a toolkit for that.

EBD is a cyclical framework, comprising of four stages and two approaches:

Stages (the “What”)

• Problems
• Goals
• Ideas
• Tests

Approaches (the “How”)

1. Questions first
2. Ideas first

EBD is a cyclical framework, comprising of four stages and two approaches

As with the popular double diamond model, we seek to “design the right it” and also “design it right”.

But the advantage of a cyclical framework is the ability to start at any stage, provided you know the outcomes from the previous two stages. I think of this like a chain of custody.

Stages

The stages are what you will do.

Start each stage by reviewing your understanding of the previous two-stages. Evidence is used and important in every stage — remember EBD Principle #1: Support every decision with evidence.

Problems

Problems are things you want to change. They can be expressed as problem statements or as needs: user stories or job stories for example. Needs imply an inherent problem with the status quo.

What you need to know:

1. What evidence do we have for this problem and where was it discovered (the prior Test stage).
2. What Idea was being tested when the problem was found (the context).

Think about: Questions

When you know your problem, your first action should be to ask questions. A good problem statement ends with a ‘How might we…’ style question, and the Five Whys technique is famous. But the questions can be less formal and less structured than this. The important thing is to fully interrogate the problem.

Goals

A Goal is how you will measure success in your efforts to solve the Problem. A Goal has a target outcome, and at least one metric.

What you need to know:

1. What is the Problem to be solved?
2. Where and how was that Problem discovered in the first place (the Test).

Think about: Metrics

A good Goal is measurable rather than simply inspirational — it contains both the O and KR parts of an OKR to cite a similar framework). To be measurable, you need metrics. One of these metrics is designated the KPI (the one that really matters) and the others as health metrics to avoid unintended consequences.

Ideas

Aside from the obvious, Ideas are potential solutions to your problem and potential ways to meet your Goal.

What you need to know:

1. How will the success of your Idea be measured (the Goal)?
2. What is the Problem you’re trying to solve?

Think about: Assumptions

Ideas are prioritised based on the evidence behind them (and therefore confidence we have in them). No Idea is perfect though and there will undoubtedly be Assumptions being made. We document these now.

Tests

Whilst the term ‘tests’ might imply something highly controlled, the Tests stage is really all about making an Idea real. This could be highly controlled, or it could be something rough and ready like paper-prototyping or guerilla testing. By making it real, you generate evidence.

What you need to know:

1. The Idea being tested and its Assumptions.
2. How you will measure success of your Test (the Goal).

Think about: Hypotheses

Ideally a test should be based on a hypothesis — a testable version of your Idea with as many variables removed as possible. A perfect source of hypotheses to tests are the Assumptions you documented for your Idea, as these are everything propping up the Idea you don’t yet have evidence for.

Approaches

The EBD framework is expressed as a cycle so we can start at any point. The ‘approaches’ reflect how you would start given the stage and the ‘chain of evidence’ will always be different given the particular challenge and context of what’s gone before.

Part of the EBD mindset is being comfortable with uncertainty. This comes from making our investigations safe-to-fail, which in turn comes from using the toolkit effectively. The approaches allow us to flex the how as well as the what to best fit the situation.

Ideas first

Rather than starting with exploratory research to identify a new problem, we could start with a new idea. Provided we are clear about the Assumptions made for the Problem we are solving and the Goal to meet, this is allowed in EBD. We would proceed to develop our Idea and Test it, feeding the resulting evidence back into our design engine.

This is an example of the Ideas first approach, and a demonstration of how EBD brings design in line with the scientific method. This is deductive reasoning: starting with a theory or hypothesis and attempting to improve it through controlled experimentation.

Reimagining the double diamond to reflect this approach, it more closely resembles a sweet:

Rethinking the double diamond as a ‘sweet’

Questions first

In the other approach, questions first, we start without a theory and seek to build one through more open research and observation. This is inductive reasoning. This approach would often more closely resemble the traditional double-diamond model, though not exclusively.

Being able to start in any stage, and having choice of approach makes the EBD framework flexible and fast paced whilst maintaining a systematic and rigorous method.

Start at the end

EBD Principle #6 reminds us to always focus on the outcome we want to achieve and work backwards.

If the stages are the what and the approaches are the how, the outcome is the why.

The diagrams above are imperfect because they don’t include consideration of the outcome, which should in fact be the first step. The outcome sits above the cycle — it needs to be known and considered before starting, but is also the ‘north star’ to guide us and make sure our cycles are iterating towards the right end.

Decide what you need to do, then start, then work out the stages in between

There’s no tool for this. So we built one.

To make methods easily adopted and transmissible, it’s important to operationalise them as much as possible. It is not effective in the long run, nor scalable, to rely on individual heroics to follow a design methodology.

Project management tools available today aren’t designed specifically for this kind of digital product R&D. We recognised this gap and so we built our own.

ResearchSweet is a project management tool, designed from the ground up for digital product R&D. Instead of generic ‘tickets’, ResearchSweet captures and links Problems, Goals, Ideas and Evidence to visualise your knowledge graph and provide prioritised roadmaps based on the biggest problem, the greatest ROI and the maximum confidence.

Currently in closed beta, we are targeting a limited launch of ResearchSweet in 2022.