Published in


Announcing CDAP 6.1.1 Release

On behalf of the Cask Data Application Platform (CDAP) community, it is my pleasure to announce the release of CDAP version 6.1.1. This release includes a new, intuitive UI for Field Level Lineage (FLL), and extends FLL support to more plugins. It also introduces a new failure validation framework for plugins for early detection of configuration errors in pipelines. Additionally, you can now connect to your data in Azure Data Lake Storage (ADLS) and transform it visually in Wrangler.

Field Level Lineage

The new field level lineage feature allows users to better visualize the journey of individual fields in datasets. The updated design also makes relationships between fields easier to see by connecting fields in the UI, and users can analyze relationships and transformations by selecting specific fields. For more information on Field Level Lineage, see this blog post series: part 1, part 2.

Example of Field Level Lineage visualization

Pipeline Improvements

This release also introduces a new way for validating plugin configurations that allows us to detect and highlight errors within a pipeline, so that you can detect configuration errors sooner and build pipelines more efficiently. All the configuration fields with errors are highlighted once the user opens the plugin properties and a helpful error message is also displayed below the field . This greatly reduces the amount of time spent configuring and validating pipelines. For more information on the validation framework, please see this blog post.

Example of validation in the Joiner plugin

This release also adds new widgets so users get better controls to configure their pipelines more intuitively. These include ConfigurationGroup — a wrapper component that can hold other widgets. It allows for easier visual grouping of related fields. Additionally, developers can now use dynamic filters to define custom conditions for when certain fields should be displayed/hidden depending on the contents of other fields within the ConfigurationGroup. For example, show the ZipCode field if the value of the Country field equals “USA”.

Platform Enhancements

This release contains performance improvements to several areas across the platform. The most notable improvements can be seen on the pipeline list page and from within the Studio. These improvements have significantly improved the performance of the UI under heavy load.

Download CDAP 6.1.1 today and take it for a spin! Also consider helping us develop the platform by reaching out to the community with any comments, feedback, suggestions, or improvements or by creating and following JIRA issues and submitting pull requests.




CDAP is a 100% open-source framework for build data analytics applications

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mo Eseifan

Mo Eseifan

More from Medium

Best Way to Deploy Sci-kit Learn model in dot Net Application

Computational Photography Series — Photographic Mosaic

Connect ESP32 to Qubitro with MicroPython

Warewulf v4 Sandboxing Environment on KVM using Vagrant