Policy A/B testing: our learning journey at Odyssey Momentum
Shortly, just like A/B testing website design mow, policymakers will test policies in smaller scale before introducing them generally.
At the weekend I had the honour to take part in Odyssey Momentum together with the following extraordinary team fellows:
- Janice Cessna data scientist
- György Farkas DAML developer
- Dani Jozsef software engineer
Our team participated in the Sovereign Nature track, whose goal it was to come up with a solution to protect biodiversity of the Sargasso Sea.
As winner was chosen by the jury the team CEVEN.protect. Here is the Odyssey announcement, here is CEVEN.protect’s website.
And here comes our story.
The “Monday morning quarterback” in me says that for me a hackathon is a place to try out risky ideas. (See also Ray Dalio’s quip: You have to bet against the consensus and be right to be successful in the markets.)
Risk taking out of despair
On Monday, it sounds as if choosing the risky way on Saturday was a straightforward decision. It wasn’t.
Our attitude was partly inspired by Odyssey founder, Singularity University faculty member and DutchChain owner Rutger van Zuidam’s quip, which we heard from him repeatedly: a hackathon (which was Odyssey called in the previous years, pivoted to be in 2020 as an “online mass collaboration arena”) is a place to “let the unexpected happen”.
A {hackathon, online collaboration arena} is a place to let the unexpected happen.
Even more predominantly that that, taking risk came out of despair on Friday, when we had to realize two things:
- The diversity of the views of our stakeholders about the actual challenge, its root causes, and the directions where we should look for a solution.
- The fact that we didn’t have a history in the domain at hand, and had no pre-packaged solution up our sleeves to pitch.
See our meme at the left side of our virtual team space below:
In hindsight, we took risk in two major ways:
- Teamwork strategy, and
- Focusing on learnings from — and about — the stakeholders at least as much as on what we were going to showcase to the jury.
So the details:
Risky teamwork strategy
Due to the short timeframe and what was perceived by us as a gap between the worldview of two groups of stakeholders, scientists and lawyers, we chose to work in parallel on three elements of our project, which are handled normally in sequence:
- Product concept
- Coding
- Communication
I was hoping that either a) there will be no mismatch in the end result, or b) the jury won’t notice it, or c) they will appreciate our bravery.
It was risky, but I treated it as an inevitable strategy described by Seth Godin in his highly cited blog post Buzzer management: if you want to win a quiz show, “You need to press the buzzer before you know the answer.”
If you want to win a quiz show, you need to press the buzzer before you know the answer.
Focusing on learning as much as on pitching
The second risky decision was that we stopped focusing on finding the best policy which can preserve biodiversity of the Sargasso Sea. Our stakeholders have been trying to find that for 15 years already. We won’t find it in three days.
Our only chance to make an impact in this domain seemed to be to get advanced to the next phase, conduct proper design sprints with the stakeholders, choose together with them the most promising policy candidates, prototype them, and iterate as many times as it’s necessary.
We have the right tool for that, the DAML smart contract platform, which we used for our hackathon prototype. (I don’t go into the details of why DAML is capable of that, I wrote about this extensively in my DAML Masterclass series, and it has been elaborated on in many blog posts written by the DAML people themselves, eg. The integral role of contract language in DLT interoperability.)
So during the jury round we wanted to convey the following message: if there is a technical solution to the problem of the gap between science and policy — and this is a big if — , we can deliver it.
This challenge — and opportunity — is much larger than Sargasso Sea: if we can bridge the gap between science and policy once, we can apply the solution to other, similarly big issues (of which I refuse to write a list, because it could seem insane to claim they are tractable).
If there is a technical solution to the problem of the gap between science and policy, we can deliver it.
In order to showcase our ability to prototype and implement any sensible policy candidate, we chose one, for which we had some data, put together a DAML contract flow and a React UI based on the DAML front end template, and presented it to the jury.
Our example application implemented a workflow to issue fishing quotas tied to honest catch reporting, and tracking those quotas.
I wouldn’t say that the jury was impressed by the specific use case. David Freestone, executive secretary of the challenge owner Sargasso Sea Commission commented: “That’s very ingenious, but you are not allowed to do that. Fishing quota issuing mechanisms are already in place.”
In order to convey a sense of the variety of policy approaches, I put here a list of the ideas which circulated during our discussions with the stakeholders:
- Incentivising catch reporting, which was actually demonstrated by the winnig team, CEVEN.project.
- Tracking fish movement with remote sensing technology.
- “From sea to table” traceability.
- Vessel movement restrictions.
- I have brought up during the discussions the topic of trading unused fishing quota by countries, which was evaluated by one of our stakeholders as “not a bad idea”, by another stakeholder “pure politics”.
Our pitch slide deck reflects our message best:
What’s next
A few days before Odyssey, I came to the idea, that policies can also be soft, meaning you don’t go to jail if you don’t obey, but policymakers try to influence you in a way which is called “choice architecture”. (The book Nudge: Improving Decisions about Health, Wealth, and Happiness elaborates on this idea, calling this approach “paternalistic libertarianism”. Some critics were not impressed by the arguments, the basic idea of choice architecture though seems plausible.)
And DAML’s rapid prototyping capabilities can help in exactly that aspect of policymaking which is most controversial: prototype testing of the efficiency of policies, be it “hard” or soft.
My vision of the future is that in a relatively short time frame this will be the norm: just like A/B testing of website design, policymakers will test policies in smaller scales before introducing them generally.
Just like A/B testing of website design, policymakers will test policies in smaller scales before introducing them generally.
As a first step towards this vision, I will propose to the Sargasso Sea stakeholders to continue working together, to do the aforementioned design sprints, choose the most promising policy candidates, and test them with DAML applications in short cycles.
