Why Executives Must Care About Fast Data
Here’s an ugly truth: as you’re reading this article, a lot of the data in your organization is rapidly losing significant value. That’s because a lot of the value of data — for insight, opportunity and response — is highest at the moment it arrives but steadily decreases as data sits waiting to be used.
In short, data gets stale, and an organization that’s not able to act on data immediately is missing out. In a very real sense, a company is only as fast as its slowest data, and as a result, most enterprises that would call themselves data-driven really aren’t.
There are two sides to the world of big data: batch and fast. Batch is historical. Processing data in batches tells you what happened at some point in the past: a week ago, a month ago, last quarter. It provides historical context, making it possible to say, for example, “It appears that making this adjustment last quarter would have made us X% more efficient, so let’s do that going forward.” Batch-only has been the dominant approach in enterprises for some time, but the limitations of that approach have become apparent. In a batch world, the enterprise is reacting to data that is, by definition, old.
Fast data, in contrast, is all about now. It is about extracting insight and creating business value through analytics and processing on data in motion immediately, as it streams into the enterprise. That data can be from anywhere: customers, IT systems, IoT sensors, security infrastructure, logistics, manufacturing lines, field operations, etc. The gap between batch and fast data represents lost value for any business that doesn’t incorporate fast data solutions. The faster you can act, the more value you can extract.
Take the example of two companies: One uses data to adjust every quarter, while the other reacts immediately. Which one wins? The second company is the one that will be more responsive to customers, better able to adapt to the market, more efficient and as a result almost certainly more successful.
The first hypothetical company is fortunate that most companies and industries are still in that same batch boat. While enterprises have spent the past decade making massive data investments to beat competitors, the vast majority of that investment has been focused on data warehousing, data lakes and other systems for using slow batch data. Huge investments in the batch side of big data have had an interesting effect: They have made it increasingly hard to differentiate simply based on the amount of data you can collect and review. A decade ago, a relatively minor investment could yield huge competitive advantage. With all the investments made to date, today’s companies are investing huge sums to chase increasingly marginal returns. The ability to understand and act on data faster, in contrast, is how winning organizations will differentiate now and in the future.
Certainly, some industries and sectors are far ahead in embracing data in motion. In most financial services use cases, for example, things happen in real time, and being able to gather and act on data rapidly is not only the name of the game — it’s a prerequisite to establishing a competitive advantage. In fact, almost any online transaction across financial services, retail, social or streaming media, and travel and hospitality needs to harness fast data to be a great success.
Consider examples such as live interaction with customers to improve conversion and satisfaction, automated systems that rapidly adjust recommendations based on data analytics, or even the development of data-driven products that create new revenue opportunities. Each of these examples has been deployed in some measure by Amazon. Does Amazon keep winning in retail because it has a better understanding of each of the products it’s selling? Or is Amazon’s ability to harness fast data to drive customer satisfaction a critical factor? Executives from Borders or Blockbuster probably know the answer.
That said, the vast majority of companies are just beginning to consider fast data, meaning smaller investments can yield exponentially greater returns in terms of competitive advantage and differentiation. In fact, looking forward, fast data processing is becoming a requirement to take full advantage of some of technology’s most exciting advances.
Take machine learning and artificial intelligence (ML and AI) as an example. Most executives consider the application of ML and AI to automation, predictive analytics, security and other use cases to be potentially transformative for their industry, and R&D labs worldwide are working feverishly on developing and advancing their capabilities. Yet extracting value by putting historical models into production has become a major challenge.
Automated systems, for example, require the right input into the right model at exactly the right time combined with the ability to “learn” (i.e., adjust the model) as new data becomes available. It’s the ability to make rapid adjustments based on data analytics (the machine learning) that distinguishes them from the rudimentary robotic assembly lines of past decades. Fast data is all about that ability to gather, transform and react to data in motion, making it fundamental to machine learning and AI.
The industrial internet of things (IIoT) is another example. IIoT is predicated on the ability to gather and, more importantly, act on the tremendous volume and velocity of data from connected devices streaming into an organization. IIoT use cases such as adaptive maintenance, security monitoring, predictive repair and process optimization all require the immediacy of fast data.
This is actually just scratching the surface of the need for and value of new technology that’s not built for a batch world. And it’s not only technical advances but also the opportunity to explore entirely new business strategies or products that is exciting about the world of fast data.
This article was originally posted on Forbes.
