The Measurement Problem Part 1: Curiosity Killed Schrödinger’s Cat
The first measurement problem for agency teams is that a client cannot be involved in a project without adversely affecting or defining its outcome. Here I propose two preferable strategies to allow agencys to do their best work whilst managing risk for the client.
This is the first in a series of posts based on quantum physics as an extended metaphor for product development. I should caveat by saying I don’t understand quantum mechanics: no-one does. But I do find it fascinating and find it to be a wonderfully compelling and surprisingly comprehensive analogy, so why not?
A familiar scenario
I’ll start with an example scenario which I hope will be at least a little familiar to many of you.
You’ve just won a project with a new client to build a new digital product. Core to your pitch were your exceptional creds: you have a winning method and the case studies to prove it. Your proposition is simple: we’ll follow our process and you’ll get results.
You host a kickoff meeting and outline your plan. “Great,” says your client. “Your process sounds perfect. We have a few ideas for cool features ourselves though.”
This seems reasonable so you take some time to catalogue their ideas, but it’s clear there’s little evidence to back them up beyond anecdote and personal conviction. You add them to the backlog, filed under “if we have time”.
A couple of sprints in and you have your first concepts/wires/prototypes to demo. Your client seems underwhelmed. “Where are all our ideas?” they ask. You carefully explain the concept of Minimum Viable Product design, the importance of building core functionality first before moving on to more “out there” feature ideas. Everyone’s on the same page and you leave with approval to deploy.
A few hours later you get an email. “We’ve had another meeting and we’ve decided we really want our ideas in the first version. This project’s off-track, we think we should have daily check-ins with you. Unless you think twice daily would be more helpful?”
You rub your temples, throw out your plans and spend the next sprint trying to turn some pretty bad and/or vague ideas into something that might work. This is done between regular emails and meetings to track the progress you are no longer making.
Come the next demo, the client has a surprise. “We have a new product owner, and they have some great ideas too.” (You can see where this is going.)
Fast forward to the end of the project. It’s a mess. Nothing much was delivered except dozens of Frankenstein concept sketches.
“We’re out of time. You didn’t deliver.” says the client, “We’ll do it ourselves next time.”
Why hire an agency anyway?
If you’ve spent any time at all working for an agency I’m sure there will be something in the above scenario that gives you painful flashbacks or little shivers of recognition.
(If not then well done, it sounds like you’re having a lovely time.)
Client interference with project scope, approach and outcomes is a frequent pain for agency teams. It’s particularly problematic in situations where the agency has been brought in to assist an existing in-house team (understandably resentful), where the objectives are unclear or impossible (see also: Nice Round Number syndrome) or where a large number of stakeholders — often from separate and competing departments — must be kept continually informed and happy.
It’s worth remembering the reasons why companies hire agencies in the first place.
At the most basic level, it’s because agencies bring something to the table that the client company doesn’t have.
This might be skills-based: simply having a skillset (app development for example) that would require a disproportionately high cost for the client to hire a team themselves considering the potentially limited term of a development project.
It might be operational: perhaps spinning up a new business function would simply take too long when there are agency teams ready and waiting to go.
Or, as I’d like to focus on here, it could be to access the specialist experience agencies can offer. Agency teams spend all their time working on their given ‘thing’ for multiple clients. This means they will tend to be very good at that ‘thing’. So accessing that expertise is valuable to the client.
So what has any of this got to do with quantum physics?
The Observer Effect
The observer effect is a phenomenon in quantum mechanics whereby the act of observing an event changes the outcome of the event.
You’ll probably have heard of Schrödinger’s Cat. This famous thought experiment applies the observation effect to properties humans can easily understand (cats and boxes). So it goes, if you seal a (hypothetical) cat in a container with a contraption that will emit a deadly gas at any random time, the cat is in a ‘superposition’ — both alive and dead. That is until someone looks inside when the superposition is converted into an actual state.
Schrödinger’s Cat was always intended to convey the absurdity of the observation effect when applied to everyday objects in our physical realm (cats and boxes) so I personally find it a bit of a distraction from the real weirdness of the effect at a quantum level.
A better example is the real-life ‘double-slit’ experiment. In this experiment, electron particles are fired at a screen, with a barrier in the way containing two slits. The particle leaves a mark on the surface on impact, so we can see once it’s arrived. Like so:
Logic would dictate that the particle would pass through either one slit or the other (depending on the flow of the particle-wave at the point it hits the card) on its way to impacting the screen. We would be left with two ‘bars’ of impact pattern on the screen.
Right?
No.
In fact, we see a ‘wave’ pattern a bit like this:
This pattern is caused by ‘interference’ — the particle appears to pass through both slits simultaneously and “bump into” itself. This alters its course and causes it to land off-target to the left or right of where we expect.
However — and this is when it gets really weird — if we set up a little measurement device to track which slit the particle passes through, it will only pass through one. The ‘interference’ no longer occurs and this is what we see on the screen:
The observation doesn’t need to be human (we aren’t that important, cosmically). The effect occurs when computerised measuring techniques are used as much as a direct human observation.
In fact — if this situation needed to get any stranger — if all the observation data is deleted prior to a human seeing it, the effect disappears and we go back to the situation of uncertainty. This is simultaneously baffling, awe-inspiring and a little scary.
This is the measurement effect in action. In this case, by observing the electrons we resolve the superposition and affect the outcome.
So what can we learn from this?
Remember our fictional agency-client scenario from before? We can apply our quantum metaphor to this situation in a couple of ways:
- The strategic problem: By inserting themselves into the process, a client will affect the outcome of a project briefed to an agency and affect its measurable success.
- The creative problem: When you hire an agency, they will bring new and broad perspectives to a challenge. They will explore it from a range of angles to find new and interesting solutions to your brief. This is analogous to a state of unresolved ‘superposition’. To extend the metaphor, client involvement could collapse the superposition too early.
Why is this a problem?
Let’s consider our learnings from the previous section in reverse order and deal first with the ‘creative problem’ of early superposition collapse (whatever you think of the metaphor, it does sound dramatic doesn’t it?).
“Do not solve the problem that’s asked of you. It’s almost always the wrong problem.” — Don Norman
At We Are Systematic, as our name suggests we take a systematic approach to every project. Our core method is the double-diamond. What’s great about this model is the need to ideate: to go broad and think around a problem from all angles. This is done in two stages, first to discover the problem, then to discover the solution. The double-diamond works best when these ideation phases are sufficiently free, so we know we’ve explored all possible solutions that could occur at that time.
Now imagine any one of these scenarios occurs during the ideation phase:
- The client proposes a solution of their own
- The client expresses a preference for one solution or a specific pathway
- The client pushes us away from an area of thinking (“We tried that before”/”We can’t do that”/”That’s so-and-so’s department” etc)
- Or simply, the client sees the work in progress.
In all of these cases, the client opens the box and the cat is either alive or dead and thus changes the outcome in even subtle ways. In many of them, they actively nudge the project in a certain direction, even accidentally. This will affect the outcome.
“Observations not only disturb what has to be measured, they produce it… We compel [a quantum particle] to assume a definite position…we ourselves produce the results of measurements.” – Pascual Jordan, physicist”
This may not even be the client’s fault. An agency account manager may attempt to interpret the micro-expressions on a client’s face whilst they observe and intervene on their behalf (“I don’t think they liked that route…”/”They seemed to really like that one”).
In all cases, the genuinely free nature of the ideation is compromised. Even the most experienced designer will struggle to disassociate themselves from that knowledge and it will certainly have an effect on their ability to explore ideas without limitation.
Let’s turn to the strategic problem. When you hire an agency, this is in effect a form of experiment. You introduce new variables into your current process in the hope of creating new and better outcomes.
Agencies are selected for a project based on their skill set, past successes or ways of working. Hopefully, the client will be happy with the results and the partnership blossoms, but sometimes not and other variables are explored instead. Therein lies the nature of the experiment.
Now we’ve framed the client-agency partnership as an experiment, observation or involvement from the client becomes a clear strategic error.
If the client has had any unsafe involvement at all (we will come on to what safe involvement may look like), then the validity of the experiment is compromised. It will be impossible to attribute the success or failure of the project to the agency. We will never know whether the clients' input was materially improving or detrimental to the outcome. Leaving aside the products or deliverables of the project, the meta-outcome will be inconclusive.
This is bad business strategy. When you undertake any project, campaign or new initiative, it is vital to be able to attribute success or failure to your key variables. Otherwise, to quote a favourite sitcom of mine, “you’re just doing random crap.”
What does an effective agency partnership look like?
So, provided you accept my premise that the observation effect is detrimental to your agency strategy, what can you do about it? What is the right way to construct an agency partnership to avoid the measurement problem?
Here are two approaches:
Model 1: Safe-To-Fail
The essence of this approach is leaving your agency to it. Give them free rein. Wind them up and let them go. Then judge the results once the project is over.
This way you can be much more certain whether success or failure can be attributed to the agency themselves, rather than your own influence.
Giving a third party completely free reign with your product or brand can be a scary prospect, probably beyond the risk palette of most businesses. This is understandable. I’m sure many agencies would also sweat at the prospect of 100% accountability too.
To make sure both sides are comfortable with the approach, we must make the project Safe-To-Fail. It’s widely accepted in business that failure is good provided it leads to lessons learned. Failing fast, before too much cost is incurred, is even better.
When a project is Safe-To-Fail, it means we have put sufficient controls in place to maximise the learnings from any mishaps and prevent damage to the wider business.
Safe-To-Fail measures can include:
- Good briefing with clear expectations of deliverables and outcomes
- Defining some ‘health’ or ‘guardrail’ metrics that mean any disasters are detected early. (In the spirit of the model, the agency must be allowed to course-correct themselves unless an outer or ‘ripcord’ variance is breached)
- Timeboxing: setting an overall time limit on delivery but allowing the agency to spend that time as they see fit according to their process
- Limited scope: limit the project to a single campaign, product, feature, business line or similar to create a firewall between the experiment and the rest of the business. If the initial pilot is a success, these limits can be expanded.
- An agreed RACI, detailing under what circumstances and how often the agency will consult the client, what they will consult them for and what the agency needs in terms of detail and turnaround time. It’s important in this model to distinguish between an agency’s need to request decisions and information from the client and the client’s desire to be involved or track progress. In this sense, it’s very much ‘don’t call us, we’ll call you”
With this all agreed in advance, both parties can undertake the experiment safely, understanding the expectations of both parties. If a failure occurs prematurely, it’s safely within the confines of the project.
Otherwise, don’t open the box and check on the cat until the time is up!
Model 2: Unified team
In this second approach, a client representative joins the agency team as a product owner. This allows the agency to benefit from their business knowledge and get quick decisions when needed.
For this model to work, it has to be generally agreed that the product owner is joining the project team, not simply checking-in with it. They are responsible for fending off other stakeholders — no-one else is permitted to communicate with the project team.
For this approach to work, the product owner must see their role as a guard or filter for the clients' communications and requirements, not a conduit. The model fails if the product owner simply becomes a forwarder of emails or passes on new requirements or requests for status updates whenever they are made.
The product owner must be bestowed the power to say ‘no’ to even senior stakeholders. It must be agreed at the outset that they will only accept new requirements in the form of change controls or through agreed channels like regular product demos or sprint planning.
Conclusion
The observation effect tells us that you cannot observe (or worse still, contribute) to a situation without affecting or defining its outcome. In terms of agency-client relationships, this means a client can’t ever know if their partnership with a particular agency is provably successful unless their contact with the project is controlled. In fact, better outcomes can be expected if agencies are left to do their best work uninterrupted.
That said, most clients will be unhappy handing over the keys to their product or brand and cede all control to a third party. This paradox means they can never know the true efficacy of their agency ‘experiments’.
To get around this, I propose two potential solutions: ‘Safe-To-Fail’ projects, and/or unified teams. Adopting one or both of these strategies will ensure the outcome of projects can be judged entirely on the agency’s method or capability, allowing the business to make an informed decision whether to continue with their partnership or not. This is ultimately a much better business strategy than continuing with partnerships when there is no clear understanding of efficacy.
