The Role of a Unified Platform in the Convergence of Big Data, Cloud, and IoT

September 13, 2017

Nitin Motgi is Founder and CTO of Cask, where he is responsible for developing the company’s long-term technology, driving company engineering initiatives and collaboration. Prior to Cask, Nitin was at Yahoo! working on a large-scale content optimization system externally known as C.O.R.E.

I am always puzzled that people think that “Big Data” is only about archiving massive amounts of data. The disruption in the market was not because companies could archive large amounts of data, but because the business value was realized by insights generated through analytics on the data aggregated. Proven by early adopters, the potential to create new business models opened doors to new business opportunities, improving customer experience and much more, and lead to the success of “Big Data”. These initial pockets of success lead to creating a new market, a new industry, and with those, a flourishing ecosystem of tools, technologies, and innovation all built around data. As Big Data became mainstream, new challenges emerged, mainly from data center operations and the skills-gap for building, managing and developing large-scale distributed systems. Managing large-scale distributed systems became more involved and expensive; the complexity and fear of losing the competitive edge drove businesses to find new solutions and offload to an alternative technology, the Cloud.

Mid-way through 2017, I have observed that big data analytics and cloud have become a top priority for most C-Level executives across industries. Harnessing insights from data is becoming more common, but, more importantly, adopting a cloud strategy to offload workloads (for cost efficiency) has also increased. Over the last 12 months, I haven’t met a CIO, CDO or CTO that isn’t driving or planning a cloud initiative. Leveraging the value and power of data and being able to extract value efficiently can give companies an edge in this competitive market.

The Internet of Things (IoT) is emerging and starting to transform how we live our lives, but the added convenience and increased efficiency comes at a massive cost. IoT is generating an unprecedented amount of data, which in turn puts a tremendous stress on the underlying infrastructure. As a result, companies are working to find ways to ease that pressure and solve the data problems. Cloud will play a major part in the solution, especially by making all connected devices work together across a common infrastructure.

Cloud offers a cost-effective delivery model for data analytics to organizations through a pay-as-you-use model. The cloud delivery model also has the potential to make your business agile, thereby reducing costs associated with self-managed data centers.

Convergence is inevitable across Big Data, Cloud, and IoT. We are seeing the beginning of this convergence already, and, in next 18 months, we will see this convergence accelerate. Businesses can’t choose from an infinite variety of vendors, technologies and OSS projects; they need simple choices. Simple usually means one to three options. Convergence can’t arrive fast enough to the world of data analytics and data-driven optimizations for businesses. There is just too much data, too many technology choices, too many vendors, too much left to interpretation, and too much-piecing things together in ad-hoc architectural solutions.

I recently read an interview on McKinsey Analytics with Ruben Sigala, chief analytics officer, Caesars Entertainment. He was providing insights into the challenges and opportunities around how to use data and analytics to inform strategic and operational decisions:

What we found challenging, and what I find in my discussions with a lot of my counterparts that is still a challenge, is finding the set of tools that enable organizations to efficiently generate value through the process. I hear about individual wins in certain applications, but having a more sort of cohesive ecosystem in which this is fully integrated is something that I think we are all struggling with, in part because it’s still very early days. Although we’ve been talking about it seems quite a bit over the past few years, the technology is still changing; the sources are still evolving.
– Ruben Sigala, CAO, Caesars Entertainment

I agree with Ruben Sigala’s sentiment and concern, and to add further, I believe it’s now the time for businesses to realize the inevitable and start investing in improving the efficiency of the data analytics and also looking at new ways to commoditize the underlying infrastructure. We have been long talking about infrastructure, and it’s time to start looking at the experience of developers and non-developers in a complex environment.

Anytime you have a problem to solve, you look for a framework that helps you address it. You use a framework because it provides the structure, the best practices, the knowledge and governance required to get to a solution efficiently. Along the same lines since the convergence of Big Data, Cloud, and IoT is inevitable, so is the framework that unifies building analytics solutions.

The foundation of a unified platform is a framework of abstraction and its associated runtime components. But the dynamics that embody big data — volume, variety, and velocity — also demand that a modern unified integration platform adapts well beyond data movement and operational data analytics, and comfortably accommodates ever-changing data formats in addition to the explosive growth of the amount of data. Giving technical users a consistent, unified work environment to develop integrated operational and analytic functions in a single data application platform helps to break-down traditional silos and achieving higher ROI. As developers and architects seek to employ more technologies to meet demands for data-driven applications, the complexity of disparate systems will only exacerbate the fundamental problem of building applications for big data.

Enterprises seeking to realize the full potential of data as the means to building data driven business need a single, cohesive system to address the challenges of today and prepare for tomorrow. These companies are using Cask Data Application Platform (CDAP) as it provides a cohesive set of tools and frameworks on top of a robust platform that allows you accelerate analytics solutions.

Learn more about Cask Data Application Platform (CDAP) and also download the latest release of CDAP 4.3 here and read about latest features here.

References

1. “How Companies are using big data and analytics”: http://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/how-companies-are-using-big-data-and-analytics
2. “The Holy Trinity of IoT, The Cloud, and Big Data“: http://dashboard.net/2016/09/09/holy-trinity-iot-cloud-big-data/
3. “The Continuum: Big Data, Cloud, and Internet of Things“: https://www.ibm.com/blogs/internet-of-things/big-data-cloud-iot/
4. “How IoT, big data analytics, and cloud continue to be high priorities for developers“: https://www.cloudcomputing-news.net/news/2016/jun/27/internet-of-things-machine-learning-robotics-are-high-priorities-for-developers-in-2016/