A Digital Evolution: Why Big Data needs Warm Data
The surging digital transformations that organisations are experiencing are a result of being swept through a strong digital current. Companies are racing into big data technologies, embracing tools like AI, whilst navigating through the tricky terrain of very real human issues such as data privacy and ethics. The liminal interface where humans meet technology is where we’re seeing the rubber meet the road.
Digital technology is now well and truly a part of our ecosystem, and to perceive it as merely a ‘tool’ in the story of our human evolution is to miss the many ways it’s shaping and shifting our society. If anything, our relationship with data and its excessiveness has brought into focus many questions on what role humans are playing in this evolutionary process — and what conditions are required to achieve the best possible outcomes.
We are still grappling with how big data and the information humans carry are intermingled — how one is influenced by the other, and what the resulting ‘information melting pot’ means for organisations and society. This may sound like some form of ‘digital alchemy’, but to really benefit from big data we need to examine these inter-relationships closely and observe how they are are evolving.
Of course, this is no easy task as humans hold vast amounts of information gathered during their lives through innumerable interactions with (and learnings from) their environment. These may be personal, subjective or coloured through cultural, social and economic perspectives across many contexts!
Yet, this information is invaluable in how we make sense of the world around us — and warrants closer study.
But how do we go about unlocking it?
The International Bateson Institute (IBI) has recently introduced the concept of ‘Warm data’, which is information about the interrelationships and interdependencies within complex systems across many different contexts. ‘Warm data labs’ are group processes where insights into warm data may be gleaned by working with information through ‘trans-contextual’ lenses.
The ‘magic’ in this process is that considering a complex issue across several contexts allows participants to become immersed in and have a better understanding of the complexities involved. Not only that, these processes allow for changes in perspectives as people explore the interdependencies within their multiple experiences.
Here is information that is not just unlocked, but evolving in real-time with new learnings — this is human agility!
What’s exciting about this, from a big data perspective, is the opportunity to explore the potential of warm data and how it may inform big data strategies for better outcomes — by providing the right conditions required to unlock valuable human information.
Let’s briefly look at two applications: digital transformation in organisations and data science.
Digital Transformation
Using big data for delivering better services is seeing many organisations undergo digital transformations for market survival, with worldwide spending expected to reach $2 trillion by 2020. What’s surprising in all this is that many companies receive poor returns on their investments, with Forbes estimating a waste of $900 billion of the $1.3 trillion spent on digital transformation in 2018 alone.
The challenges that companies face during such transitions go beyond just the technology adopted, and include supporting human capability, organisational culture and investment decisions etc. These are no strangers to any CDO, and many organisations are focusing on data strategies that aim to empower people with the right information to facilitate organisational transformation.
But where is the right information held?
Is it in the big data we’re collecting en masse? Or is it held within an organisation’s workforce? Or is it a mixture of these?
The warm data or the trans-contextual information at play here is dynamically changing and evolving — and within an organisation is held across the business silos, the experiences of the people working there, the interactions with stakeholders, the customer relationships, the judgements and assumptions made by data modellers etc.
While we’re figuring out how to improve the synergies between big data and the information that people within an organisation hold, recognising that we need a deeper understanding of the ecosystem across many contexts is critical to delivering new insights!
Warm data labs provide a process to uncover these, and a space to allow for deeper systemic patterns within the collective ‘human information’ held in an organisation to emerge.
But the effect is much deeper than that — warm data labs also provide the conditions needed for people to adapt to a changing work environment. While many may fear digital transformation as a threat to jobs, this process may not only help them adjust through the transition, it may uncover the invaluable knowledge assets they bring to the workplace that are usually overlooked.
Where once organisations tried to align their people with their digital transformations, warm data may allow for a ‘digital evolution’ — so that the transformation evolves through mutual learnings between people and technologies.
Data Science
The quest to derive meaningful insights from big data has seen huge investments in developing data science capabilities and progressing the field. The ‘big data ecosystem’ is indeed a complex one, and for all the efficiencies that more data and smarter algorithms can provide — there remains a gap between data-driven insights and delivering better solutions.
Organisations recognise that big data is not the ‘silver bullet’ — and are using different approaches for delivering better big data outcomes, such as using ‘more nuanced small data’, focusing on building teams of generalists rather than specialists, and finding better ways to communicate data-insights amongst others.
Yet, some of the biggest difficulties faced by data scientists today are about how the data and models they use relate to the business or societal issues at hand — understanding and defining the problem (or sub-problems) being solved, recognising the limitations of the data models including data biases, and how data insights may inform decision-making are just the tip of the iceberg!
At the crux of the issue, what most companies are reaching for are better solutions for running their businesses or solving problems — and this involves both data science and people. But it goes deeper than that — it also involves how data science is influenced by the information that people hold.
The idea that information held within our individual experiences might allow for a deeper understanding into value-creation and problem solving is not new in data science — with most data scientists recognising the need for domain knowledge and multidisciplinary skills. Indeed, some of Google’s recent approaches to forecasting involve having ‘humans-in-the-loop’, thereby integrating data science and business planning.
What is new here is the ‘extra maneuvering space’ that warm data labs offer for uncovering new and deeper human insights, by allowing for the problem to ‘freely move’ across many contexts. Sense-making is enriched through a trans-contextual understanding of the interdependencies within the problems we otherwise spend so much time de-coupling...
In a previous post, we explored the possibility that working with ‘warm data’ could complement more traditional ‘data-driven’ approaches, and help uncover new insights into complex problems. And there are plenty of opportunities in data analytics to do so — one has only to encounter missing data, sparse data, data anomalies or seasonality data to see the immediate potential.
How warm data may influence the creation of data science models, and the judgements and assumptions we make around them will play a large role in their integrity and effectivess in organisations and society.
Warm data lab group processes also support communication through ‘mutual learning’ within the organisation. Data visualisations may be more effective when developed within these conditions — rather than at the end of a data science project.
Working with people on complex problems in a trans-contextual setting not only allows for deeper insights into the complexities within the system, it also creates the conditions required for shifts in perceptions and strengthens the bonds within an organisation’s ecosystem.
This article is based on “The subtle art of data gathering with Warm Data”.
Phoensight is an international consultancy dedicated to supporting the interrelationships between people, public policy and technology, and is accredited by the International Bateson Institute to conduct Warm Data Labs.
