Advanced CDAP Directives

Tony Hajdari
Nov 18, 2019 · 6 min read
Photo by Shahadat Rahman on Unsplash

CDAP provides a number of ways to process data in a pipeline, and one of the most flexible ways to build advanced data processing logic is through the use of directives with JEXL expressions in Wrangler. Transformation logic can be created in CDAP studio directly where a number of CDAP plugins are used to source, transform, and sink data by building a directed acyclic graph (DAG) of processing logic, and each of the transformations is done in sequential phases via purpose built transform plugins.

Wrangler is itself a transform plugin, but you can think of it as the Swiss Army Knife of plugins in CDAP, as it acts more like a plugin of plugins. Wrangler has the ability to not only use the default directives that come with CDAP, but you can also include your own User Defined Directives (UDDs) to add custom processing logic.

In this article I’ll show you how to use some of the more advanced features of directives to harness the power of Wrangler to perform transformations with fewer directives. Three of the most powerful and utilitarian of these directives are: set-column, filter-rows-on, and sent-to-error.

JEXL

Previously I mentioned an obscure acronym. So, what is JEXL you ask? The Apache JEXL page defines it as follows:

“JEXL is a library intended to facilitate the implementation of dynamic and scripting features in applications and frameworks written in Java.”

Let’s see what Wrangler directives look like with JEXL.

set-column :movie_source “imdb”.toUpperCase()

So what does this directive do? To better understand how these directives use expressions let’s unpack the directive syntax fist.

The syntax for this transform is broken down as follows:

<directive> <column> <expression>

The set-column directive can create a new column in a dataset, or set the column value to the result of an expression execution. It takes a column name as the first parameter and an expression as the second parameter. In this example we are creating a new column named movie_source, and then setting a value for each row in that column to the string “imdb.” The expression can take data as input from a column or it can be set statically. Where JEXL comes into the picture is in the use of the Java String function toUpperCase(). By adding the function in the expression we are instructing the directive to apply the uppercase function from the Java String library to every row where the string is encountered. In this case we implicitly set the string to a static value of imdb, but we could have just as easily uppercased any column value from the dataset.

For example, the following directive would create a new column, movie_source, with all the values retrieved from the source_of_data column, and strings within that column would all be converted to uppercase:

set-column :movie_source source_of_data.toUpperCase()

Examples

Perhaps it’s a bit early to grasp the power of expressions at this stage, so let’s use a few more examples to illustrate.

filter-rows-on condition-true country !~ ‘US’
filter-rows-on condition-true (country !~ ‘US’ && hourly_wage > 12)

set-column Name concat(first_name, “ “, last_name)
set-column :Date exp:{new(“java.util.Date”)}
set-column Adult age > 21 ? ‘yes’ : ‘no’
set-column Salary hourly_wage*40*50
set-column Salary_Increase var x; x = math:ceil(FLOAT(hourly_wage)); x + 1
set-column :foo “lorum ipsum yada yada”. toUpperCase()
set-column :c1 string:upperCase(“Make this all uppercase!”)
set-column :bar foo.length()
set-column :c2 string:length(c1)
set-column :moo foo.substring(1)
set-column :c3 string:substring(c1, -10)
set-column :baz exp:{var s = “meh”; bas = s}
set-column :baz exp:{var s = foo.substring(1); baz = s}
set-column :blah exp:{var s1 = foo.substring(1); var s2 = “OO”; if(s1.compareTo(s2)){“foo”} else {“bar”};}
set-column :blah1 exp:{var s1 = foo.substring(1); var s2 = “OO”; s1.compareTo(s2) ? “foo” : “bar”;}
set-column :c4 var x; x = math:ceil(22.5); x
set-column :c5 var x; x = math:floor(22.5); x * math:random()
set-column :c6 math:random()

send-to-error Name == null
send-to-error Age.isEmpty()
send-to-error Age < 1 || Age > 130
send-to-error !date:isDate(DOB)

As you can see in the example directives, you can perform calculations within expressions, as well as create variables, assign values, perform logic operations, use Date, Math, and String functions from Java, and the list goes on.

Emulated Join

One of the benefits of being able to use conditional operators within a directive is that you can emulate a dimensional table for joining data. To illustrate I will use a small movie dataset with MPAA ratings for movies and emulate a dimension table in the directive. For all intents and purposes this will be an in-memory table that will be able to perform super fast joins. Of course this technique is not meant to replace more scalable and maintainable alternatives but for smallish dimension tables, like MPAA ratings with descriptions, or state names with their abbreviations would be a good use case.

The sample data will contain the movies with their assigned MPAA rating abbreviation. Our directive will contain the description of the rating. We’ll check the sample dataset for the column with the MPAA rating and create a new column that provides the description for the rating as well.

The CSV file containing the sample data used in this article can be found here.

In CSV format our dimension table would look something like this:

G, General audiences — All ages admitted
PG, Parental guidance suggested — Some material may not be suitable for children.
PG-13, Parents strongly cautioned — Some material may be inappropriate for children under 13.
R, Restricted — Under 17 requires accompanying parent or adult guardian.
NC-17, No one 17 and under admitted.

We turn that data into a directive:

set-column rating MPAA_Rating == “G” ? “General audiences — All ages admitted” : “Not Rated” && MPAA_Rating == “PG” ? “Parental guidance suggested — Some material may not be suitable for children.” : “Not Rated” && MPAA_Rating == “PG-13” ? “Parents strongly cautioned — Some material may be inappropriate for children under 13.” : “Not Rated” && MPAA_Rating == “R” ? “Under 17 requires accompanying parent or adult guardian.” : “Not Rated” && MPAA_Rating == “NC-17” ? “No one 17 and under admitted.” : “Not Rated”

As you can see this directive uses the ternary conditional operator and performs 5 evaluations. Because these evaluations take place in a single directive they perform much faster than if separate directive would be called five times per record in the dataset.

Conclusion

Directives are used to transform data within the data preparation UI — Wrangler — which provides immediate feedback to the user as the transformations are applied. Once a pipeline is published the transformations run in parallel across any number of nodes in a cluster. What makes Wrangler even more powerful is that you can chain Wrangler plugins together to create units of transformation, so the output of one Wrangler step becomes the input of another Wrangler step in a data pipeline.

By leveraging the power of the expressions in directives you can come up with creative ways of solving data engineering problems.

cdapio

CDAP is a 100% open-source framework for build data…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store