Can Digital Medicine Build Evidence Less Like Prozac and More Like Pinterest?
Mental models are a powerful force.
When the emperor arrived on the streets in his “new clothes,” it wasn’t so much his foolishness that the tailor was taking advantage of as it was the emperor’s unshakable mental model of tailors, clothes, and status. The emperor had such a strong belief that clothes would emerge from the tailor’s process (measuring, sewing, fitting) that he allowed himself to be dressed in nothing, based on the power of that belief alone.
Indeed, mental models can be incredibly powerful and useful tools to align ideas and drive progress. But they can also be powerful blockers of vision.
At this moment in time, I wonder if digital health is being held back by the mental model of the phased clinical trial system for the creation of evidence. Fundamentally, it’s a fine system for molecule-based treatments like Prozac to go through, but you can’t do every job with the same tool. By focusing evidence creation on large phased and highly regulated trials, we’re missing huge opportunities to take advantage of the unique, fast-paced, continuous feedback cycles made possible in tech products.
In a blog post from 2017, an engineer at Pinterest wrote, “Every day we aim to have experiment results ready by 10 a.m. so we can make fast and well-grounded decisions. With more than 1,000 experiments running daily, crunching billions of records for more than 175 million Pinners, we need a reliable pipeline to support our growth…” My PhD thesis, on the other hand, had taken five years and included maybe half a dozen experiments.
For me, hearing about Pinterest’s experimental capabilities for the first time was revelatory. What they have is a rigorous protocol system for fully controlled (yes, controlled!), scalable, modifiable, real-time experiments. The protocol is iterative and accessible, too; there is a clear process for anyone on the product team to submit product experiments that they want to validate for ongoing development.
This means that when Pinterest makes a product change, it’s able to do so by implementing tiny changes already proven within small controlled experiments to lead to product growth and/or a better user experience.
From a design perspective this is very cool, and from a research perspective, it’s staggering. We are also called to wonder: what if all this evidence design and experimental power could be translated to a product that, instead of focusing on curating photos of artisanal cupcakes focused instead on, say, treating mental illness?
The core feature of the clinical trial system is that it ensures safety and mitigates risk. What the process doesn’t do well is enhance benefit. At the end of a trial all you know is, did black box molecule X do better than sugar pill Y? You know nothing about the mechanism of action or how to make it better next time. The model for Pinterest, on the other hand, is nearly exclusively about continuous improvements. The experiments are much smaller and shorter, the feedback cycle is tight, and the data is accessible for future thought and revisions.
The kind of evidence creation driving Pinterest is undoubtedly complex and the mapping to digital health products is not straightforward (increasing click traffic is a more direct goal than decreasing loneliness, for example). But the first thing we can shift is our mental model of how evidence can be generated.
How can we think more continuously about the evidence we create? How can we generate more, smaller improvements faster? and how can we ensure safety in this new way of working?