Dynamic Column in Azure Data Factory Flow

Based on the requirement, we need load source data into Landing Table, using Azure Data Factory.

source file and database table

primary key is deviceId, day 1’s value need mapping to column Mon. day 2’s value need mapping to column Tue, and so on.

my data flow is:

link source to the txt file.

Add one parameter: ColumnName, calculate the column name in pipeline (by filename or other logic) and pass it to data flow.

in the Select1, add a rule-based mapping column, “locate(‘data’, lower(name)) != 0” then set the column as “$ColumnName”

then add DerivedColumn1, to setup all calculate columns.

finally, we got the result like this:

we can sink the data into the database.

--

--

--

Recommended from Medium

How in MediatR we can have events (Notifications) async and completely real Parallel

How in MediatR we can have events (Notifications) async and completely real Parallel

How to fix Internal Server Error for WordPress

A Heaping Helping of Heaps

Fixing Dockerfile image build consistency

The Internet’s Universal Process Sharing Platform, Minerva, Raises $1.1M Seed Round

Mobile App Deep Linking and how to configure its attribution of traffic sources

Why You Should Stop Representing Age As a Number in Your Code

7 Dying Programming Languages to Avoid in 2021

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Sen Zhang

Sen Zhang

More from Medium

Depositing Data In Azure

Azure Data Factory Control of a Virtual Machine

Azure Data Factory: Keep your secrets in Azure Key Vault

The Azure DevOps Rant #1