Azure Data Factory Data Flow

During several years of my life I built Data Pipelines using ETL Tools, then I moved to Azure and I started using ADF.

The Gap in Azure Data Factory (ADF) v1 and v2

The first thing I noticed using Azure Data Factory service was the lack of some important features; most of ETL Tools (like Pentaho, Informatica, etc) come with a lot of out of the box funcionalites that helps you speed up the process of building a data pipeline. I figured out that most of the typical funcionalites were missing in ADF and in order to overcome this GAP it is necessary to code more then usual or use MS SSIS together with ADF.

This lack of features is something that Microsoft is aware of, and since the begining of ADF v1 and v2 the ADF PM Team has been working to make significants improvements in the product to make the product more suitable to ETL/ELT Data Pipelines.

Azure Data Factory (ADF) Data Flow announcement

Recently Microsoft ADF PM team released about Azure Data Factory Data Flow and made some important improvements in the product.

In order to try this preview it is necessary to request permission from Microsoft using the link http://aka.ms/dataflowpreview

You will find the screen below where you will have to request the permission.

After you are authorized to use you have to create the services using Azure Portal, as you can see below:

Quick demo: Example of Data Pipeline created using Data Flow.

The video below will give you an introduction about ADF Data Flow.

More videos can be found here.

Important related links:
https://www.jamesserra.com/archive/2018/12/azure-data-factory-data-flow/
https://sqlplayer.net/2018/12/azure-data-factory-v2-and-its-available-components-in-data-flows/
https://mssqldude.wordpress.com/2019/01/23/adf-data-flows-databricks-notebook-etl-vs-adf-visual-data-flow/
https://github.com/kromerm/adfdataflowdocs