How to integrate Azure Data Factory, Azure Functions & Logic Apps the easy way?
In this article, we are going to see an easy way to use the 3 Azure Services together. We will show this with a case study.
Case Study:
Existing on-premise system has the following features:
— Data Movement and Data transformation
— Image resizing
— Manifest file creation with E-tag, Record Count and File name
— Customized Email Functionality
We would like to move this solution to Azure Cloud. How will we do that? All the above requirements can not be met easily at the moment using Azure Data Factory service only. However using few other Azure services these requirements can be met with ease.
Solution:
The above on premise solution can be migrated to the New Data Integration system on Azure Cloud using the following Azure services
— Data Movement and Data transformation can be handled by Azure Data Factory
— Image resizing can be done using Azure Functions
— Manifest file creation with Etag, Record Count and File name also handled by Azure Functions
— Customized Email Functionality for which Azure Logic Apps can be used
For the end to end solution all of the below Azure services are required:
1. Azure Data Factory
2. Azure Functions
3. Azure Logic Apps
4. Azure Key Vault
5. Azure Storage Account
6. Azure SQL Server.
- Azure Data Factory(ADF):
A cloud-based data integration service that orchestrates and automates the movement and transformation of data. It is like a conductor or orchestrator who does not do any things itself but everything managed under it.
— Azure Platform as a Service (PaaS)
— It is Serverless.
— Only pay for what you have use.
— Low-code and no-code Solutions.
2. Azure Functions
Azure Functions is a service for easily running small pieces of code or a function in the cloud. Azure Functions can be written in Programming languages such as c#, Java, Python, etc. It is such a powerful service which help you to do a specific task in your Data Integration Solution, IOT Integration Solution or Application Integration Solution.
— Event driven
— Serverless and PaaS
— Scalable
— Lightweight
How can we integrate Azure Functions with ADF?
Once you have created ADF and Azure Functions App, you can add functions in Azure Functions App. so now you have Function URL and Function key.
Azure Data Factory has an Azure Function activity. In Setting tab of Azure Function activity, you can set above Function’s URL to call Azure Function within ADF via Azure Function Linked Services. This is a no code low code solutions
3. Azure Logic Apps
A cloud-based integration service for creating and running automated workflows that integrate your apps, data, services, and systems.
How Azure Logic Apps integrate with ADF?
Once you have created ADF and Azure Logic Apps, you can add Triggers and actions in your Logic Apps. By Now you will have Logic Apps URL.
Azure Data Factory has a web activity. In setting tab of Web Activity, you need to set above Logic Apps’s URL to call Logic Apps. This is a no code low code solutions
4. Azure Key Vault — This service is mainly used for security and its a central place to store secrets.
5. Azure Blob Storage — This service is used to store source data files.
6. Azure SQL Server — This service stores final output created as sink or destination
Now lets see high level design of the actual solution. There are two main pipeline to drive required result.
- High level Architecture design for creating a manifest file :
1. The Azure blob storage container named file-in has a list of csv files. This input file can be of other format as well e.g. JSON, XML.
2. ADF reads file from above container and applies Data transformation and loads the output to Azure SQL table.
3. The ADF pipeline creates a manifest file using Azure Functions. The Manifest file gives summary of the input csv files e.g. File Name, E-tag, total number of records, last modified datetime. This functionality is not feasible using ADF service only and requires the use of Azure Functions(Code in C#, Python, Java etc)
4. Finally, the ADF pipeline sends a customized email using Azure logic Apps indicating whether the pipeline execution is a success or failure.
2. High level Architecture design for image resizing:
1. Azure Blob Storage Container named file-in has a list of jpg image files which are to be resized and a csv file having metadata information of these images.
2. The ADF pipeline first reads the csv file from the container. Each record of the csv file corresponds to 1 jpg file and has its metadata information (filename, etc.). The ADF pipeline reads each record of the file and moves the corresponding jpg file to the input container. Once all the jpg files are moved the csv file is loaded to the Azure SQL table.
3. Once the jpg files lands in the input container the Azure function will be triggered automatically and resize the image and store it to the output container.
4. Finally, the ADF pipeline sends a customized success or failure report in an email using Azure logic Apps.
GitHub Code Repo — alpaBuddhabhatti/PASSSUMMIT2022 (github.com)
YouTube Video Link — https://www.youtube.com/watch?v=l-pbdTkHR5U&t=1s
Conclusion:
In order to make your data driven solution easy, above services need to work together. Also it is easy to integrate them together. If your requirement is not feasible directly in ADF then you can use Azure Functions or Azure Logic Apps.
E. g you need to call a Rest API which generates a complex XML Output then you need to read and process that xml file. However, writing an xml Sink is not possible using ADF so you can use Azure Functions or Azure Logic Apps to write xml file to Azure Storage and then you can read and process xml file using ADF which is no code or low code solution. However for same purpose you can use Azure Functions which is Code first technique.
