Embrace Complexity — Conclusion
Building Your Organisation's Knowledge Graph
“I have yet to see any problem, however complicated, which, when looked at in the right way, did not become still more complicated.” — Paul Anderson
This is the final article in a series (intro, data, cloud and AI) about embracing complexity. It has been a long journey and thank you for your attention if you have worked your way through the proceeding articles. If you are reading this for the first time then don’t worry, as this article can be treated as an executive summary.
Your Organisation’s Knowledge Graph
A powerful idea has been slowly building for many years now, originally known as the Semantic Web, and then later as Linked Data. This idea has finally come of age with the emergence of Knowledge Graphs. These technological networks allow an organisation to release the ‘free energy’ bound up in the chaotic jumble of databases and channel that energy directly into AI.
Built correctly, an organisational Knowledge Graph can combine the power of data, the cloud, and AI in one unified structure. If your organisation has more than a hundred separate databases or applications and you have not yet started building your Knowledge Graph then I urge you to initiate your programme immediately and without further delay!
To remind you, this series of articles has four main contentions:
- Networked Data. That network-shaped data can model complex structures including circular feedback loops and abstract models, and that networks (or graphs) make the connections between things as important as the things themselves.
- Networked Cloud. That the network-shaped cloud of connected computers means that ALL the important data in an organisation can be joined together regardless of where that information is stored. Moreover, it states that an organisation's Knowledge Graph is not just one big centralised database, but rather a distributed and interconnected ecosystem.
- Networked AI. That networked-shaped AI lets us make predictions about connections, loops and abstractions and embed the generated insights directly back as an integral part of the Knowledge Graph. And that this very active branch of machine learning is starting to outperform ‘traditional’ AI in complex tasks.
- Unified Network. And finally, it states that these three networks (data, cloud & AI) can be joined into one Knowledge Graph that has the powers of each component but is also more than just the sum of those parts.
To ground this in reality the series has outlined three practical tools:
- The Graph Adapter. Which sits on top of the existing databases, APIs and files in your organisation and converts 2D sets of tabular data into 3D graphs of data. The key intuition here is that the underlying databases, files and APIs do not need to change — the adapter just exposes a network-shaped layer on top of all other data structures.
- The Data Service. Which is a specialisation of an existing and well-established architectural pattern called a microservice (but where data itself is treated as a first-class citizen). An individual data service can use the graph adapters to publish graph fragments into the cloud using an HTTP server. Each data item is given a unique resolvable network address in the form of a URL. The data services (or Data Products) combine to form a peer-to-peer network (or Data Mesh).
- The Graph Neural Network. This allows each data service to mirror the passive graph-shaped data with an active graph-shaped machine learning model that gives each node the potential to also learn and predict. The data service publishes these learned node embeddings back into the network as pure data.
These tools combine to form a Knowledge Graph where each node in that graph is a data item, and it is a network address, and it is an artificial neuron in a neural network.
In a correctly constructed Knowledge Graph, each node can be data, cloud and AI, all at the same time.
Seeing your Organisation as a Whole
By mapping the network of all organisational data a Knowledge Graph lets us see our organisation for the complex system that it is.
Current change management programs are not fit for purpose as they are not equipped to understand the full complexity of large organisations. With a Knowledge Graph leaders finally have the instrumentation necessary to steer the ship. Changes can be modelled to take into account the way that everything connects together. How the feedback loops in the system drive and control change. With a Knowledge Graph leaders have a tool that is capable of providing a holistic and systemic view of how the organisation functions as a whole. They can work with the flow of the organisation's existing momentum.
“You can use the opportunities presented by a system’s momentum to guide it toward a good outcome — much as a judo expert uses the momentum of an opponent to achieve his or her own goals.” — Donella H. Meadows
A Knowledge Graph gives us the ability to connect ALL the critical data in the entire organisation together. This gives us access to the vast volumes of data that AI needs, and more than that, because it is graph shaped, it can also see the more nuanced curves, circles, and feedback loops that exist within that data.
Furthermore, a Knowledge Graph can be turned inwards and used to model and harness some of these feedback loops. Feedback loops around dataset publication and consumption can be used to establish an antientropic Internal Data Marketplace. A marketplace that will self-organise to clean and consolidate the fractured intellectual property that is currently scattered behind the organisation's firewall.
A Knowledge Graph can also be used to create a single unified Semantic Layer, where data is represented in business terms (like tracks and trains for a rail operator, or trades and risk for an investment bank). This makes the data available to all, not just the ‘high priests’ in IT.
Finally, a Knowledge Graph can be used to train proprietary AI models that feed off data that is refracted through the lens of your organisation's Semantic Layer. This idea is partly expressed in the concept of a ‘digital twin’ and it is how your AI can be trained to impact the bottom line within the particular niche that matters to you.
Seeing the Big Picture
“Systems thinking is a discipline for seeing wholes. It is a framework for seeing interrelationships rather than things, for seeing patterns of change rather than static snapshots…Today systems thinking is needed more than ever because we are becoming overwhelmed by complexity.” — Peter Senge
It is clear to see that the world is becoming increasingly complex and connected; there seems to be an unstoppable inevitability to the process of networkification and this process is rapidly accelerating.
A Knowledge Graph allows an organisation to internalise this network shape. In return, the internal network empowers that organisation to make wiser, more informed decisions. A Knowledge Graph gives an organisation the speed and agility necessary to thrive in an increasingly digitalised environment.
There are no silver bullets, the Knowledge Graph is not a product that you can buy off the shelf. It is a way of organising your data and algorithms in a unified network, and each organisation must bring its genius to this process. There is no magic ‘quick fix’. In order to succeed the entire organisation must engage with this complex task. It requires commitment, leadership, patience, dedication and the ability to collaborate at an organisational scale.
Moreover, the Knowledge Graph is a tool that can be used to see the whole, but embracing complexity is a mindset. In the industrial era, we looked down and understood the elements of systems in great detail and this approach has taken us very far indeed. Now in the technological era, we are beginning to supplement this analytical thinking with systemic thinking.
Systemic thinking sees the connections between the parts within our organisation but it also acknowledges that our own organisation is but one subsystem layered within a larger economic system, which in turn exists upon a beautiful (but fragile) blue and green sphere floating through space.
As a species, we sleepwalked through the last major social phase transitions. During the agricultural revolution, we came out of the forest and into the fields with promises of lands of milk and honey that too often turned into pestilence and peasanthood. The industrial era offered to take the weight of the plough off of our backs, only to replace it with smoke-belching factories filled with assembly workers.
Throughout history, technology has delivered amazing progress forwards, but as we move through our next phase transition. As we either let our data and AI collapse into the gravity wells created by the tech giants and superstates or, alternatively, as we each build our Knowledge Graphs. Perhaps we should take a moment to pause, and really pay attention. Eyes wide open.
The reinforcing feedback loop between complexity and the rate of change keeps on accelerating, but in what direction?
We would do well to ask ourselves some deep questions:
- What are we accelerating towards? To where are we heading?
- What do we want the information age to look like?
- Do we want to teach our AI to be ruthlessly efficient, or teach it to be kind and socially responsible? Perhaps we want a balance between the two? And if so where should the balancing point be?
I urge you to start building your Knowledge Graph now. However, we do not get to put genies back in their bottles. So, I also urge, that while you do so, you truly embrace complexity by endeavouring to always keep The Big Picture positioned firmly in the forefront of your mind.