Using data to fuel imagination

In 1967, when testifying in front of a Congress looking to assign blame for the tragic failure of the Apollo 1 spacecraft, Astronaut Frank Borman suggested that there was no single individual or group on which blame could be assigned.

The problem was a “failure of imagination” — the team working on the project had extensively worked through the challenges that they understood. It was a challenge that they didn’t expect which blindsided them. In retrospect, the failure of the Apollo 1 problem was predictable and in plain sight. But in the moment, it was either invisible or of insufficient concern to a large team of brilliant, well-intentioned, hard-working people.

Successful companies are constantly facing disruptive forces that they don’t see coming. Even the most thoughtful, sophisticated and focused leaders lack the omniscience to visualize these risks and to maneuver accordingly.

You’re already storing data — about your customers, operations, financial performance — which tells stories you’re not seeing. Stories of risks and opportunities which are currently unexplored by your smart, well-intentioned team.

Unlike the NASA team of 1967, today we’re equipped with powerful and inexpensive cloud computing and established machine learning techniques to ingest massive volumes of data and find patterns and trends invisible to the human eye. There’s simply no faster way to uncover the hidden factors influencing your business — and you’re likely already sitting on at least some of the data required.

It’s worth noting that these machine learning algorithms lack the power of the institutional knowledge possessed by an executive or business analyst. As such, they can detect fanciful, meaningless correlations. However, they’re also unencumbered by the constrained worldview and bias inherent in every human being. These algorithms are unaware that examining certain possibilities is simply “not the way things are done”. And as such, they can reveal new opportunities outside the prior boundaries of our imagination.

You may have read about how Google’s DeepMind AI software became the world champion of the ancient game Go — in part by simply never being exposed to “traditional wisdom” in the first place, effectively being left with only a rulebook and countless cycles for exploration. Its success was in finding new ways of playing the game that were outside human imagination because of centuries of precedent and the biases of collective experience. These biases exist within your organization. Even if “because that’s how we’ve always done it” is outlawed from your vocabulary, there are human instincts, preconceptions and experiences hiding certain opportunities and risks from your vision.

Unlike with Go, there’s no simple rulebook by which most businesses operate, so problems can’t be solved by an algorithm alone. By bringing machine learning capabilities together with thoughtful, considered analysis, new possibilities are very likely to present themselves. New understandings of customer behavior, marketplace trends, competitor behavior, or other factors.

In many organizations, data has historically been stored and analyzed in silos based on the ownership and function of that data, and for both practical and political purposes. Breaking down those silos and aggregating data to ready it for a broader-spectrum of analysis is a proven way to begin opening the doors to new possibilities. The same is true of looking for complementary third-party data sets that can surface new insights about your customers and the market or provide some other context which doesn’t exist within your first-party data alone.

Even if your existing data is chaotic and contaminated with errors, the same data mining techniques that can be used to discover hidden patterns and trends can be used to clean, correct and make sense of data that was previously destined to be written off.

There are few opportunities which provide a more direct path to expanding your team’s imagination than providing them with an expanded set of data and new insights surfaced by advanced analytical and machine learning techniques. Who knows where this journey will take you?