Welcome to the second article in this four part series. In the first article I discussed some of the concepts related to continuous integration and testing. …

Welcome to my latest series on continuous integration of data pipelines with Cloud Data Fusion (CDF) and/or CDAP. …

To really become a Ninja with Wrangler Directives you have to get to know all the functions that Wrangler supports. In this article I’m going to list out all the Wrangler functions with a short description of what each one does. At the time of this writing the Wrangler code…

In a previous article I showed you how to get started with plugin development in CDAP. But, wouldn’t it be nice if we can attach a debugger to a deployed plugin and see how it performs inside of a pipeline with actual data? I’m going to show you how to…

By now you probably know that you run CDAP pipelines on Cloud Data Fusion (CDF), but did you know that you can also control your CDF instance remotely using the rest API. In this blog I will walk you through the process of deploying and starting a CDAP pipeline on…

One of my favorite features of CDAP is that the extensibility of the platform allows you to add new functionality yourself. …

CDAP provides a number of ways to process data in a pipeline, and one of the most flexible ways to build advanced data processing logic is through the use of directives with JEXL expressions in Wrangler. Transformation logic can be created in CDAP studio directly where a number of CDAP…

Part 3 of 3 — DB Data Pipeline — In part 2 of this series we set up and tested connectivity to the Cloud SQL databases for MySQL and Postgres. In this final article I’ll walk you through he process of loading data to Cloud SQL and using both MySQL and Postgres database instances to build a pipeline where…

Part 1 of 3 — How to integrate CDAP with Google CloudSQL This is part one of a three part series that will walk you through the following concepts: Configure Google CloudSQL with two databases; MySQL and Postgres. Build the custom JDBC drivers to connect to the CloudSQL databases with CDAP. Load databases with sample data, and create a pipeline that joins…