Business has seen an explosion in machine learning and cognitive data technologies over the past few years.
In the not too distant past, a PC came with a floppy disk with about 360 KB of storage. And not long before that, a 10 MB database was considered large. Most IT applications were built and run in-house to keep track of things. Small databases stored and tabulated employee information, kept track of inventory, and recorded debits and credits.
Most of these applications were mundane, but they freed workers from the mind-numbing drudgery of sifting through massive amounts of information.
As we track hits on websites, sell products online, engage with customers, and collect data from billions of telematic devices, applications and people alike are struggling to keep up.
There’s so much data, most of us are what vendors call “data rich and knowledge poor.”
And today, this might be truer than ever.
By 2020, there will be an estimated 6 billion smartphones and 50 billion telematic devices on our planet — a veritable tsunami. The world economic forum said that data would drive the Fourth Industrial Revolution.
Today, data volumes have swelled to a point we never could have predicted. But whether you go back 10 years or 30, there has always been more data than modern computer platforms could handle.
To me, “Big Data” is a marketing buzzword — a way for tech vendors to sell demos. Yes, data is big, but it always has been. What’s new is the temporal range of business intelligence.
We weren’t always able to use this data in the moment, but now we can. And in the future, we must.
Here’s how to prepare yourself for the revolution:
Don’t mine data — farm it.
So, you have more data than ever before. Great. Now how do you mine it?
In my opinion, you don’t mine data — you farm it. Farming is a sustainable venture, whereas mining depletes. And data doesn’t lose its value once it’s used. Rather, it can be used over and over, becoming more valuable each time.
In the past, we were simply trying to harness the data. But today, the more relevant question is how to convert that data into knowledge. The present challenge is sifting through this massive trove of data to find an element of predictive behavior that can then find its way into a real-time decision engine.
When handling big data, your process should always be two-fold. First, you derive the knowledge. Then, you make it actionable.
If you can’t use it to predict real-time behavior, you won’t be competitive in today’s market.
There’s still a place for long-term analysis.
In the old days, before spreadsheets, large Decision Support Systems (DDS) ran on very large and expensive mainframes. It was a slow, laborious process. A manufacturer might spend six months using DSS to assess the economics of building their next off-shore facility.
Today, that analysis could be done on a PC with little more than a spreadsheet.
Regardless of the amount of data, the decision process in the old days was relatively slow. The fact that it took six months to render a competent decision was often perfectly okay. It did not need to be made in an instant.
And there are situations today where that’s still the case.
Say you want to track down illicit money laundering. You gather banking wire transfer data from the past 5–10 years, then sift through this immense volume looking for patterns. After months of applying various statistical tests, you might find a bit of wheat amongst the rest of the chaff — a veritable needle in the haystack. You turn over the suspicious information to the FBI so they can root out the scofflaws.
Value is derived from the data, but it’s slow.
Or, consider using DNA from a cold case to track the perpetrator of a decade’s old crime. You retrieve the old DNA sample, then send it into a genetic web site. A few weeks later, you get the results online and match relatives to the perpetrator. Then you locate those relatives, find likely suspects, collect DNA from garbage cans, compare it with the original DNA and — voilá, you solve the case. It’s front page news.
Some intelligence still takes weeks and months to harvest from the data pool. Often referred to as “predictive analytics,” it can be a slow process. There is still a place for that, but it’s not the way of the future.
Real-time decision is the next frontier.
Understanding the temporal aspects elements of business intelligence is crucial to composing a strategy to extracting value from data.
In addition to less time-sensitive data projects, today, we also need decisions made in milliseconds. Not minutes, not hours, not weeks, not months, not years — but in real-time.
You might not even notice how frequently instantaneous business intelligence comes into play in your daily life.
When you open laptop in the morning to read the news, for example, a targeted ad lands on your webpage in less than a second. And when you use your mobile phone, your carrier is quickly able to verify it’s you. When you swipe a credit card, you get approval in mere seconds.
These actions seem simple from our end, but they’re made possible by highly sophisticated processes.
When you enter your credit card information to complete an online purchase, that action sends a few scant bits of information about you to a decision engine, which then swings into action to “decide” to approve or deny your transaction. False negatives and false positives are both bad outcomes. Has the card been reported lost or stolen? Is the transaction typical in terms of your merchant selection, dollar value, and shopping patterns? The receiving application maintains a profile of information and applies it to determine what should be done next.
And it’s got to happen in the blink of an eye, or customers will go elsewhere.
Our customer ThreatMetrix helps businesses prevent online transaction fraud, a job that requires making decisions in less than one second ThreatMetrix’s Senior Director of Engineering said in an interview that the longer a transaction takes on a website, the more card-abandonment issues there will be, which means higher costs for customers. “If there’s a spinning wheel or something for too long,” he said, “then people will abandon their transaction.”
Every time we use the internet, the receiving website has only milliseconds to decide what is the next best action — show you an ad, resume your last session, collect your data, approve your transaction, or any of a million other actions.
Fast or slow, there is an emerging spectrum as to how best to extract intelligence from data and then how best to apply it. Our modern computing platforms make all this possible in ways it never was before.
Today, a temporal consideration is a must.