Announcing CDAP 6.1.1 Release

Mo Eseifan
Feb 3, 2020 · 2 min read

On behalf of the Cask Data Application Platform (CDAP) community, it is my pleasure to announce the release of CDAP version 6.1.1. This release includes a new, intuitive UI for Field Level Lineage (FLL), and extends FLL support to more plugins. It also introduces a new failure validation framework for plugins for early detection of configuration errors in pipelines. Additionally, you can now connect to your data in Azure Data Lake Storage (ADLS) and transform it visually in Wrangler.

Field Level Lineage

The new field level lineage feature allows users to better visualize the journey of individual fields in datasets. The updated design also makes relationships between fields easier to see by connecting fields in the UI, and users can analyze relationships and transformations by selecting specific fields. For more information on Field Level Lineage, see this blog post series: part 1, part 2.

Example of Field Level Lineage visualization

Pipeline Improvements

This release also introduces a new way for validating plugin configurations that allows us to detect and highlight errors within a pipeline, so that you can detect configuration errors sooner and build pipelines more efficiently. All the configuration fields with errors are highlighted once the user opens the plugin properties and a helpful error message is also displayed below the field . This greatly reduces the amount of time spent configuring and validating pipelines. For more information on the validation framework, please see this blog post.

Example of validation in the Joiner plugin

This release also adds new widgets so users get better controls to configure their pipelines more intuitively. These include ConfigurationGroup — a wrapper component that can hold other widgets. It allows for easier visual grouping of related fields. Additionally, developers can now use dynamic filters to define custom conditions for when certain fields should be displayed/hidden depending on the contents of other fields within the ConfigurationGroup. For example, show the ZipCode field if the value of the Country field equals “USA”.

Platform Enhancements

This release contains performance improvements to several areas across the platform. The most notable improvements can be seen on the pipeline list page and from within the Studio. These improvements have significantly improved the performance of the UI under heavy load.

Download CDAP 6.1.1 today and take it for a spin! Also consider helping us develop the platform by reaching out to the community with any comments, feedback, suggestions, or improvements or by creating and following JIRA issues and submitting pull requests.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store