CDAP 3.0 — From Zero to App in 5 minutes

May 5, 2015

Nitin Motgi is Founder and CTO of Cask, where he is responsible for developing the company’s long-term technology, driving company engineering initiatives and collaboration. Prior to Cask, Nitin was at Yahoo! working on a large-scale content optimization system externally known as C.O.R.E.

The Cask Data Application Platform (CDAP) was created with the intent of empowering all developers to build data applications. It was, is and always will be a developer platform — a platform with the mission to provide developers with simple access to power technology. CDAP has proven to significantly lower the barriers to building Hadoop applications, simplify complex tasks on distributed systems and provide a level of abstraction that makes it easier to build next generation applications.

CDAP is over three years old now, and it has come so far in that time that it’s hard to recognize it from the first release. During this time it has been put through rigorous testing in many enterprises and is in production helping our customers and developers everyday. Toward the end of last year we released CDAP (f.k.a Continuuity Reactor) as open source under the Apache 2.0 license to extend the reach of CDAP and help make our platform available to every developer who wants to build data applications on Hadoop.

CDAP 3.0 is a significant expansion of capabilities and usability and represents a major product milestone for Cask. It started a year ago when Jon (my co-founder and Cask CEO) and I were chatting in front of our office having a conversation about how we could further simplify CDAP. We envisioned a code-free user experience — one where developers wouldn’t have to write any code to perform common tasks and could easily deploy reusable application patterns. They would be able to configure, run and manage these applications directly through the UI, harnessing the power of Hadoop without diving into the complexity of individual projects or building custom applications. An extensible plugin framework would allow developers and our community to build new patterns and make them available in the UI.

That’s when we had an “aha” moment and together coined the term “Zero App”. We both fell in love with this idea and phrase immediately and then spoke with our existing customers and prospects about the idea. We received great support and encouragement from our community and so over the coming months we began the journey to build the Zero App experience into CDAP.

Zero App is a new way to experience Hadoop through CDAP and has two parts:

1. A programming interface (API) that allows developers to build their own reusable application patterns and domain specific plugins.
2. A user experience that allows other developers or non-developers to configure these templates and utilize the application patterns through the user interface, interactive shell or REST APIs, enabling them to solve business problems without writing any code.

Zero App defines a new component called Application Templates, reusable blueprints of common use-cases built using CDAP APIs. As the first two examples, CDAP 3.0 includes templates for batch and real-time ETL pipelines. Application Templates are built using the CDAP Application abstraction and are made up of one or more Programs. These Programs can be MapReduce, Workflow, Spark and others, and then can be specialized with per-template plugins. For example, the ETL templates expose plugins for extending sources, transformations and sinks. Another example in our road map is the Personalization Application Template that exposes plugins for feature generation, modeling and scoring.

Zero App also defines Adapters. Adapters are specializations of Application Templates with configuration…but with zero-code. For example, the ETL Batch template can be specialized through configurations to have an RDBMS as its source, projections as its transformation and the OLAP Cube Dataset as a Sink. This would allow a developer to easily configure a pipeline that builds an OLAP Cube on HBase from data in an existing relational database.

We have put a lot of time and thought into building the next generation of CDAP. I’m incredibly excited today as we are releasing CDAP 3.0 with the Zero App experience to the community and the world. Please take a few minutes to download our software and try it out. Let us know what you think and come join our community of developers.

While we’re excited about our progress, there is a lot of work in front of us to deliver on our mission to bring developers simple access to powerful technology.