Taming and training hunches: Data analytics and foresight
This article is part of Discovery Hub’s Data Test Kitchen series. We explore how data science and machine learning can help enhance foresight on emerging innovations and their social impact.
Ursula K. Le Guin, in her groundbreaking novel The Left Hand of Darkness, writes about the fictional Handdarata culture and their mythical art of foretelling. Its practitioners have “tamed and trained the hunch… the power of seeing (if only for a flash) everything at once: seeing whole”.
A different approach rooted in mathematics and calculations was envisioned by another science fiction writer Isaac Asimov in his Foundation series. He wrote about an imagined scientific discipline of psychohistory that could predict the future development of societies with a high degree of accuracy.
The practice of foresight has long been a fascination for many, from ancient prophets and science fiction writers to prediction market forecasters and corporate futurists. At its core, strategic foresight helps people and organisations to systematically think about the future in the face of uncertainty.
The future, however, is unknowable as it does not yet exist and thinking about it requires imagination. Perhaps the future’s lack of factuality is one of the reasons why it has often been debated whether studying the future is an art or science. This has been discussed at length, for example, by the American futurist Wendell Bell who concluded that futures studies is in fact an “action science” — it uses scientific thinking to improve decision making with respect to desirable and undesirable consequences. Nonetheless, the foresight and futures practice also includes methods such as speculative design and science fiction writing that can facilitate thought-provoking discussions about the future.
In the remainder of this article, I discuss how creativity and rigour can co-exist within strategic foresight, and explain how Nesta’s Discovery Hub harnesses data science to deepen our capacity to evaluate possible futures.
Quantitative foresight for social good
As Nesta focuses its efforts on achieving missions in health, early years and sustainability, our task at the Discovery Hub is to anticipate disruptive trends and help our organisation adapt. For us, the applied or “action” element of the foresight work is critical: We are endeavouring to support colleagues and partners to make better strategic choices in the here and now.
Data analytics can support this in various ways, quantitative foresight being a vast family of various methods. For example, we can measure trends around emerging technologies and innovations thus providing evidence of shifts in the external environment. By extrapolating from these trends we can devise future scenarios.
We can mine large datasets to surface signals of the future to come. We can visualise and simulate complex systems making the abstract more concrete.
These methods have been applied in many different contexts, from considering military strategies, to businesses and markets, to considering the direction of technological and social change. For example, recent forecasting of future energy costs (initially released just before the COP26 conference in 2021) demonstrated how a rapid transition scenario to green technologies would save trillions of dollars globally compared to the status quo. Another example leveraged agent-based simulations and a complex-systems approach to more realistically simulate the impact of automation on workers in the labour market.
Overall, quantitative foresight methods help us think about the possible outcomes, their likelihood of occurrence and the underlying factors driving these outcomes.
Combining the strengths of data analytics and foresight methods
By using data analytics, we aim to increase the rigour of our thinking about the future, while maintaining the creativity that futures methods are known for.
While data can provide evidence for or disprove our hunches, the deliberative and participatory nature of qualitative futures and foresight methods can infuse values, lived experience and judgement into the forecasting process.
Foresight can elevate quantitative modelling of trends and forecasts by considering the wider societal consequences of these trends, as well as by helping ponder the nonlinear effects and potential shocks that are difficult if not impossible to foresee in the data.
We can use foresight methods such as scenario modelling to help contextualise and make sense of quantitative predictions. Quantitative forecasts about narrow, specific questions can be situated within complex and compelling qualitative scenarios to create advanced early warning systems for policy makers.
Data science methods can also be used to amplify qualitative insights about the future. For example, Nesta’s work on the labour market demonstrates how expert judgements can be augmented by machine learning to produce estimates of future growth for hundreds of different jobs and skills.
Ultimately, data can provide suggestions of what might happen, but it won’t tell us what we should do about it. To get to the “so what’s” and to design proposals for action, qualitative methods and judgement is essential.
A systems-level view of trends: Innovation Sweet Spots
So what does this look like in practice? How are we now blending quantitative and qualitative methods to generate insights which can be applied in service of Nesta’s social missions?
Our experimental Innovation Sweet Spots programme is not unlike Ursula K. Le Guin’s vision of seeing the whole, as we aim to map innovation systems around social challenges. We analyse trends around innovations and technologies that can achieve social impact, by using large datasets on investment and public discourse. With this we wish to capture a multi-dimensional view of the socio-technical system in which innovation happens, as we recognise the important role that science and business as well as media, policy and social discourse plays in the development and adoption of new innovations.
By capturing a variety of signals, we can perform gap analysis and notice aspects in which certain innovations are excelling or conversely might be stalling or underperforming. This can support policymakers, funders, investors and anyone working on social challenges to better understand the innovation system as a whole, and direct resources where they can deliver maximum social impact.
Our pilot Innovation Sweet Spots project focussed on low carbon heating technologies such as heat pumps, geothermal and hydrogen heating, which are key to reducing household carbon emissions and enabling the UK reach its net zero goal. We found strong signals of research and development funding as well as an increasing public discourse about low carbon heating in the UK. Conversely, an apparent gap in private investment raised concerns about further scaling-up of these technologies. By discussing our findings with energy and investment experts, we highlighted some ways to overcome the key barriers to investment and adoption of these technologies.
Anticipating future developments
Besides examining historical trends, we also wish to combine quantitative and qualitative analysis to better anticipate future change.
Our next Innovation Sweet Spots project is on food technologies and innovations. In addition to measuring trends around innovations like lab-grown meat and dark kitchens, we are engaging industry experts through participatory methods such as the futures wheel and wiki surveys. This will help us consider the societal consequences of the trends and their impact on food environments and obesity.
We also consider future developments using more quantitative approaches. We have trialled an experimental machine learning approach that tries to learn patterns of historically successful startups and aims to predict future success of companies in sectors that are important for Nesta’s missions such as green technologies, food tech and early years education. The aim of this is not to replace human decision-making when making investments but rather to facilitate the development of policy to support growth and innovation, as low probabilities of predicted success could signal the need to better understand barriers to success in that space and introduce packages of grants or other stimuli.
We will report our findings from these projects in the near future.
Mining data for future signals
The seeds of the future can be found in the present, and data can facilitate surfacing those signs. We can leverage large datasets describing startup and research activity for horizon scanning, to find examples of cutting-edge innovation.
For example, in our collaboration with Nesta’s fairer start mission, we analysed over a thousand startups in the parenting and early years education space that are already bringing new products such as child speech recognition and smart parenting assistants to the market. We characterised the progress so far and envisioned some of the potential positive and negative consequences if these technologies continue to grow.
We are also interested in exploring how to facilitate discovery of such signals by visualising datasets related to innovations and technologies. An example of such a solution is our recent “map” of mobile apps for parents and children. We created it by data-scraping the Google Play Store and using automated visualisation algorithms to show the sprawling landscape of over a thousand apps that parents and children are navigating from preconception to preschool. While this was a demonstration of apps that are already on the market, we can use similar methods to visualise more forward-looking data related to development of technologies such as research project abstracts or patents.
Open source and open foresight
Combining data science with strategic foresight is still a relatively nascent practice with many avenues to explore. It is important that we keep experimenting and that it is done openly, backed by those focused on innovation for public good — not just profit. Perhaps this is one of the major differences between Discovery Hub’s data analytics work and the average corporation: that all our work is offered back to the commons — including the ‘how’ and the failures too.
In this work, we benefit from a close collaboration with Nesta’s Data Analytics practice, which also values working in the open and sharing as much as possible. We’re building on the years of experience and infrastructure developed by our colleagues working on mapping innovation and studying the labour market at Nesta. Moreover, our work of course depends on the vast and generous open source community without which it would simply not be possible.
We will continue trialling new methods that can help us understand the innovation environment, anticipate future developments and support better decision making. For example, we’re exploring the potential of causal inference to provide data-informed recommendations for actions (we will report the findings from our initial explorations in the near future). We are also curious about how network science can improve our understanding of the innovation system to pinpoint novel and promising innovation and collaboration areas. We’ll also consider the usefulness of participatory and data-informed approaches such as fuzzy cognitive mapping for creating scenarios and understanding complex systems.
As we progress in blending data with foresight, creativity and participation, we will write here at Discovery Hub’s Data Test Kitchen about our lessons learned. If you would like to discuss any of this, please reach out and I’ll be always happy to talk.
Huge thanks to Celia Hannon, Laurie Smith and Siobhan Chan for their helpful comments on this article.
