Announcing CDAP 6.1.1 Release

Mo Eseifan
Feb 3 · 2 min read

On behalf of the Cask Data Application Platform (CDAP) community, it is my pleasure to announce the release of CDAP version 6.1.1. This release includes a new, intuitive UI for Field Level Lineage (FLL), and extends FLL support to more plugins. It also introduces a new failure validation framework for plugins for early detection of configuration errors in pipelines. Additionally, you can now connect to your data in Azure Data Lake Storage (ADLS) and transform it visually in Wrangler.

Field Level Lineage

The new field level lineage feature allows users to better visualize the journey of individual fields in datasets. The updated design also makes relationships between fields easier to see by connecting fields in the UI, and users can analyze relationships and transformations by selecting specific fields. For more information on Field Level Lineage, see this blog post series: part 1, part 2.

Example of Field Level Lineage visualization

Pipeline Improvements

This release also introduces a new way for validating plugin configurations that allows us to detect and highlight errors within a pipeline, so that you can detect configuration errors sooner and build pipelines more efficiently. All the configuration fields with errors are highlighted once the user opens the plugin properties and a helpful error message is also displayed below the field . This greatly reduces the amount of time spent configuring and validating pipelines. For more information on the validation framework, please see this blog post.

Example of validation in the Joiner plugin

This release also adds new widgets so users get better controls to configure their pipelines more intuitively. These include ConfigurationGroup — a wrapper component that can hold other widgets. It allows for easier visual grouping of related fields. Additionally, developers can now use dynamic filters to define custom conditions for when certain fields should be displayed/hidden depending on the contents of other fields within the ConfigurationGroup. For example, show the ZipCode field if the value of the Country field equals “USA”.

Platform Enhancements

This release contains performance improvements to several areas across the platform. The most notable improvements can be seen on the pipeline list page and from within the Studio. These improvements have significantly improved the performance of the UI under heavy load.

Download CDAP 6.1.1 today and take it for a spin! Also consider helping us develop the platform by reaching out to the community with any comments, feedback, suggestions, or improvements or by creating and following JIRA issues and submitting pull requests.

cdapio

Mo Eseifan

Written by

cdapio

cdapio

CDAP is a 100% open-source framework for build data analytics applications

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade