Copy file from On premise to Azure blob using ADF pipeline

In this article, I have explained the step by step instruction on how to copy file from On premise to Azure blob using Azure Data Factory pipeline.

Step 1: Install Self-Hosted Integration Runtime in On prem

The integration runtime (IR) is the compute infrastructure that Azure Data Factory and Synapse pipelines use to provide data-integration capabilities across different network environments.

Download and install Self-Hosted Integration Runtime in On prem system from the below Microsoft website

https://www.microsoft.com/en-us/download/details.aspx?id=39717

Step 2: Create Self-Hosted Integration Runtime in Azure

1. On the home page of the Azure Data Factory UI, select the Manage tab from the leftmost pane.

2. Select Integration runtimes on the left pane, and then select +New.

3. On the Integration runtime setup page, select Azure, Self-Hosted, and then select Continue.

4. On the following page, select Self-Hosted to create a Self-Hosted IR, and then select Continue.

5. Enter a name for your IR, and select Create.

6. Copy and paste the authentication key in the Download integration runtime in On prem.

Step 3:Create Linked Service for the Source File system

1. Browse to the Manage tab in your Azure Data Factory and select Linked Services, then select New
2. Search for file and select the File System connector.
3. Configure the service details, test the connection, and create the new linked service.

Step 4:Create a DataSet for the source Linked Service

A dataset is a named view of data that simply points or references the data you want to use in your activities as inputs and outputs

To create a Source dataset with the Azure Data Factory Studio, select the Author tab , and then the plus sign icon, to choose Dataset.

Select the Linked Service that we created in Step 3.

Step 5: Create Linked Service for the Target Storage Account container

1. Browse to the Manage tab in your Azure Data Factory and select Linked Services, then select New
2. Search for file and select the Azure Blob Storage connector.
3. Configure the service details, test the connection, and create the new linked service.

Step 6: Create a DataSet for the target Linked Service

To create a Target dataset with the Azure Data Factory Studio, select the Author tab , and then the plus sign icon, to choose Dataset.

Select the Linked Service that we created in Step 5.

Step 7: Create the pipeline

Create the pipeline and import Copy Activity. Copy activity is to copy data from data stores located on-premises to cloud Azure Blob storage.

Video explanation of the above steps