What Dominic Cummings thinks you should know… (Part 1)
14 days have now passed since Dominic Cummings (DC) released his blog, calling for “data scientists, project managers, policy experts, assorted weirdos…”. In the past week extensive ink has been spilled discussing the pro’s and con’s of DC’s most recent plan. For example, on Saturday, Rachel Coldicutt, describing herself as being late to the party, put out this great summary https://medium.com/swlh/inside-the-clubcard-panopticon-why-dominic-cummings-seeing-room-might-not-see-all-that-much-f940a48ae1cd.
I have to be the first to admit I am now more-than-late to the party — I am definitely now at the after party or simply here to take people home.
However, this blog takes a slightly different approach which is why I think there’s still value in writing it. While I will give my opinions along the way, the real aim of this blog is to discuss which bits of DC’s vision and research interests are worth knowing about.
This is going to be useful for three main reasons:
1) Firstly, DC’s blogging style isn’t the most accessible. I think the world needs a simpler interpretation of what he’s trying to say. Consider this blog a translation of DC’s circumlocution into something a bit more direct [overall, I’ve grouped everything DC has been saying into five main themes that we discuss over two blogs].
2) Secondly, whether or not you think it will work — it’s impossible to ignore what is going on at the core of government. Terms like “Seeing Rooms” and “Dynamic Land” have already received much wider attention than previously. Many of the other concepts could also achieve much wider prominence in government, research or Open Data circles, depending on what ultimately happens. Consider this blog your essential horizon-scanning guide.
3) Finally, if you actually intend to apply for a role with DC, you probably need a nice summary of the things he’s interested in to help you at interview. For those of you thinking of applying, you can consider this blog the York Notes of Dominic Cummings’s mind.
This two-part blog is structured around five key themes that emerge through the papers Dominic Cummings references. In part one we discuss:
- High Performance teams, whether it’s really enough to try and replicate the past.
- The degree to which ‘globally observable’ patterns are able to help us in policy.
In part two we dicuss:
- The limitations of AI and what elements of research are likely to be most relevant to policy-making and forecasting.
- Complex contagions — areas where they might be essential, but also the critical risks and ethics around this type of analysis.
- And finally, the vision for dynamic documents — Dynamic Land and Seeing Rooms.
Part 1
1. High performance teams - are we really better placed than ever to understand how they work and replicate them?
So let’s start with the basics — Have you heard of Derren Brown? Yes — Derren Brown the illusionist.
Derren Brown has an incredible illusion called “The System” recorded for TV which can be viewed here. In the illusion Derren Brown’s goal is to convince the world he has created a system to consistently win at horse racing.
His system is obviously amazing.
In the TV show, Derren is able to convince a member of the public to trust him — bet on the races using the system and consistently increase the size of their bets. We meet Khadisha and follow her as she takes part in 5 horse races using Derren’s system — she wins every time.
For race 6, the final race, Derren convinces her to take out her entire life savings and bet ‘the house’ on the final horse. What’s incredible about the programme is that we see everything — the highs, the lows, every aspiration, every moment of fear, until the final bet. But what happens next is mind-blowing… Before the final race, Derren reveals what he has done. You see the illusion wasn’t about having a system at all. Derren hadn’t been filming a single individual. In fact, the system relied on working with 7,776 (6^5) people. Bets are systematically placed on races with only 6 horses with individuals instructed to bet so that within the group overall there would always be a winner.
Derren’s real illusion was to convince the TV audience that he has a system by showing us an individual story. All along it was always guaranteed that there would be one such story and by following everyone on camera the team guaranteed they could create the illusion.
So what’s the relevance of this to Dominic Cummings? Well, Dominic Cummings is very interested in high performance teams and replicating what they do. Dominic Cummings is convinced there is a “huge amount of low hanging fruit” — literally “trillion dollar bills lying on the street” at the convergence of AI and the selection, education and training of people for high performance. He’s also convinced that the formula is there, cost would be low and in most cases the costs would be recouped “within weeks by stopping blunders”.
Now I have a lot of sympathy with what Dominic Cummings is trying to do. I do think he has a point about how we fund science — for example he states: “The Right tends to underestimate the role of patient taxpayer funding of basic science from which specific industries and products evolve, years or even decades later.” I also agree that new approaches to solving big problems focused around long-term support for high-performance teams are likely to be the best way to stimulate high-return scientific progress in a post-Brexit world. However, I think Dominic Cummings under-estimates the role of survivorship and context in his analysis and, as a result, overestimates the surety of the benefits and underestimates the costs and risks of doing this from inside government.
In truth, if we wanted to, we could dedicate a entire blog (and more) on what we could call “the conditions for high-performance.” We know the types of conditions that are likely to be necessary for the types of high-performance DC aspires to create. But are they sufficient? i.e. If the right conditions are in place does it naturally guarantee that we will get transformational success? This is an essential question since at the heart of DC’s pitch is the assertion that if he can build a high-performance team, success will follow. There are a couple of points to make:
Placing the right bet:
In his 2008 book Outliers, Malcolm Gladwell examines many areas of success but, in particular, he looks at the role of context. Compiling a list of the 75 richest people through history (adjusted for inflation) from Bill Gates to Cleopatra, Gladwell identifies that nearly 20 percent of them were born in the same country (the US) within a single 9-year window. Think about it — of all the wealthiest people that have ever been, 20% were born in a single country within 9 years of each other (the 1830s). Gladwell argues that the unique place and period of time you occupy contributes as much (if not more) to success than any other factor. Every decade produces roughly the same number of gifted children as did the 1830’s. The difference was that for this brief period of time the economic and political “stars” aligned in just the right way so that those who had the talent and could take advantage of the generational change that was occurring around them. Gladwell makes a similar argument for the billionaires of the tech boom citing 1955 (specifically between 1952 and 1958) as the optimum time to be born to have made it big in personal computing. Being born in 1955 would have ensured being the ideal age in 1975 to profit from the personal computer revolution.
DC is clearly interested in the personal computer revolution and replicating the success of that era in post-Brexit Briton. In one of his papers he walks through the history of PARC — the Xerox R&D centre in Palo Alto (you can read DC’s account here. PARC really does have an amazing pedigree of invention — over the 5 years from 1970–75, the PARC group created the first standalone PC; bitmap display; the modern GUI; object-oriented programming; word-processing; laser printing; the Ethernet networking system; and the overall ARPA vision. DC now feels as though there is a similar confluence of factors (prediction, data science & AI, communication and decision-making) that could achieve transformational levels of innovation on a par with the personal computer revolution. Obviously, as a technology innovator in the data science and prediction field, I don’t disagree that there is a sizeable opportunity, but with a ‘risk hat’ on I also think there is a good chance of a new AI-winter [later we go through all the areas where DC acknowledges today’s AI has severe limitations]. I you take the view that context plays as much of a role as the team, then it’s all about placing the right bet — it’s very difficult to say that DC has a better chance than anyone else of being right as to what that critical bet is likely to be.
Spreading the risk:
So this is where I come clean. If you were to ask me to name the most high-performing team I’ve ever worked for you might be shocked with the result. The truth is that I’d have to say “Lehman Brothers”. Yes, you heard me right — Lehman Brothers. The company single-handedly responsible for the destruction of the global economy. And I’m not alone. Every Lehman alumni I know agrees. Lehman Brothers was an amazing place to work and an incredibly high-performing team (while it lasted). We sought talent from the best universities around the world, we subjected them to rigorous tests, we gave them specialist training and we had systems that were highly optimised to achieve our goals. It just turns out, we were so optimised to focus on year-on-year profit growth, collectively we missed the longer-term risk-aversion that would have saved us.
No one ever asks me how Lehman Brothers created such a high-performing team — most people just jokingly ask if I was to blame for the Global Financial Crash [note — I was very much on the right side of the trade, just at the wrong institution]. The point I’m trying to make is that it’s very easy to view the successes and think we can learn from them — what we never consider is that high-performance teams also fail.
In this respect it’s just like Derren Brown’s illusion — we convince ourselves that there is something unique about the person who wins 5 races in a row without realising the number of losers who are following exactly the same system.
Why is this important? Well, frankly there is a risk that Dominic Cummings fails. If he does fail, then there are a lot of people who will want to highlight that fact. It could therefore be the last time we try. Therefore maybe, if we really do want to make a change, it’s better doing it outside of government with legal structures more willing to accept failure.
2. The patterns of human behaviour are predictable… sometimes
DC is deeply interested in how we improve prediction by utilising cutting-edge techniques from across different fields - from physics to weather forecasting to finance and epidemiology. He bombards the reader with a range of papers — some advocating one technique, others saying the techniques don’t work and others openly questioning whether scientific findings can really be true (in particular this). The desire to draw together all these strands is great.
However, I need to highlight a critical distinction between:
1) processes (e.g. maths or algorithms) that predict patterns; compared to
2) processes (maths or algorithms) that can predict paths or specific events.
Some of the papers DC cites are in this first group — they are concerned with predicting patterns. These types of approaches point towards some underlying structure but often fail to provide predictions that are specific. Let me give you an example — take video games. Let’s assume it’s 2050 and you rank the top selling computer games by total sales — I’m fairly confident, without doing any research, that I could tell you with a good degree of precision something about the shape of those sales — i.e. I could tell you what pattern those sales are likely to follow. But what I won’t be able to tell you is which video game, which team, which platform is the top ranked. In fact, I probably can’t tell you anything about the top 25. Patterns are interesting, and they point to some underlying dynamics in the way events (such as sales) occur — but most of the time people want to know specifics. Predicting specifics is much harder.
Power Laws
Some of the patterns that we know exist are truly exciting. They point to an underlying structure to the way we organise ourselves. This Ted Talk from Geoffrey West is a must watch if you want to know more:
Geoffrey West talks about how factors such as income, wealth, number of patents, number of creative people, crime rate, flu cases etc. all scale super-linearly (note there are positives and negatives in this list). Therefore for a given city size, it is possible to predict these outcomes with some regularity. In the New Science of Cities, Michael Batty lists 7 scaling laws for cities (for some slides see here) again reflecting the fact that we can be fairly confident regarding predictions of patterns in space.
There are a wide number of contexts in which similar relationships can be observed — below is a nice image from UCL showing the most important destinations in the network of international airports. Here the pattern of airport importance can be predicted and its distribution can be approximated by a power law.
Now, these facts are obviously of great interest to anyone trying to predict what’s going on. The problem is that most policymakers rarely care about patterns that are observed at the group level — instead they want to understand how to improve their city, their region.
For example if you take number of patents by city size in the UK, there are a couple of outliers (e.g. Cambridge). A policymaker in another city might want to know how to increase the number of patents per capita without increasing city size. If we take a purely mathematical, pattern-based view, then we might be inclined to argue that it’s impossible — if you believe that the pattern-based view is all there is, then you might ultimately conclude that individual policies have no impact at all — the only driver is city size.
Given that most policymakers are concerned with the impacts of specific policies (in their region) rather than patterns exhibited by groups of regions, it should come as no surprise that so far (having been modelling cities for 4 years), we’ve found limited practical application for these relationships. DC suggests readers review this paper — Scale-free networks are rare, Nature 2019 — which argues that most networks actually don’t follow power laws anyway and that the universality of these relationships remains controversial.
In short — it’s not clear how DC is planning to use these relationships but thus far we’ve found limited practical application for policymaking. Further, the paper he references suggests modelling is likely to be much more context-specific than the ‘power law’ narrative would have us believe making it much more difficult anyway.
Predicting War
There’s no doubt that patterns are interesting, but before launching into a deep dive of modelling, it’s important to ask what we’re trying to achieve. One of the things DC seems to be interested in is the prediction of international events. One example he gives is this paper which examines the frequency and severity of interstate wars. The paper claims that the frequency of wars follows a well-known statistical distribution (a Poisson distribution — see here) and that the severity of wars follows a power law.
This is certainly a curious observation. Does this mean that wars are random? How much influence do we really have to stop wars if they are just the result of underlying mathematical processes? If so, what does this mean for policy? Given that we can predict the probability of war over a particular time-frame how should we think about defence spending?
Now I’m not sure how DC is proposing to use this observation. But my recommendation is that he doesn’t and to explain why I’ll use a quick thought experiment. Imagine this paper wasn’t about wars at all, but was instead about US high-school shootings. What would it mean for policy if the frequency of high-school shootings followed a Poisson distribution (if you’re interested in whether they do or not see this article)?
The obvious danger here is that if something follows a statistical distribution we’re lured into reasoning that maybe it can’t be changed. But obviously, in the case of high-school shootings, the number of school shootings in the U.S. far exceeds that of several other high-income nations even after accounting for population size. The U.S. rates of lethal violence, mainly due to firearm homicides, also exceeds that of other economically comparable nations (see here) suggesting that the higher rate of school shootings is associated with the availability and accessibility of guns.
While the mathematical patterns in data are provocative, I’m simply not sure how relevant they should be to policymakers. To close the circle and return to the question of defence — if none of us have weapons then the probability of conflict approaches zero. Why should the Poisson relationship of the past provide any steer on our policies of the future?
Final notes
Finally, DC suggests a couple of further papers that are worth noting. Following on from the theme of predicting ‘bad stuff’, he suggests the following paper: Early warning signals for critical transitions in a thermoacoustic system. These types of techniques are likely to have more of a practical application — for example we used similar methods in financial services (pre-QE) to predict sell-offs in the stock market. In contrast to the above, spotting patterns like these can be useful (but are likely to be most useful for operators of real-time systems). Once again though there were limitations — for example in Financial services we were able to predict periods where the market would sell off but it was almost impossible to predict how long a sell off would last. A further challenge, once you go outside of settings such as financial services, is the task of compiling and maintaining relevant data for what you want to achieve. The effort here cannot be underestimated.
A further area DC is a fan of is the use of crowdsourcing to remove bias or highlight areas where data may be incorrect. To find out more about this, the following article provides a succinct overview: so you think you’re smarter than a CIA agent (we aim to talk more about crowdsourcing in a future white paper).
In part two, we’ll look at the limitations of AI, areas where AI could make a meaningful impact on policy, the ethical risks of DC’s plans and why he’s right about dynamic documents.
