Tales from the Fabric — Chapter 1
The exodus of a pipeline
Prelude
Microsoft Fabric it’s been in GA for more than 2 months. If you’re a data professional and you don’t know what Microsoft Fabric is, you should definitely go to check it out.
Going beyond the hype generated by a new tool, digging deep inside of it it’s mandatory to propose good solutions to customer needs. In other words, Microsoft Fabric it’s not a cure-all, we need to study it, finding pros and cons, strengths and weaknesses as any other tool.
“Power is nothing without control”
The best way to study something is to try it, and the best way to try something is on real world scenarios, therefore we decided to start POCs on Microsoft Fabric.
The very first could be called “Migrating from Azure Synapse Analytics to Microsoft Fabric”. This customer has a pretty solid set of Synapse based applications, most of them orchestrate ETL/ELT with Synapse Pipelines, store data in Dedicated SQL Pools and present “massaged star-schema” to Power BI.
Even if Microsoft Fabric contains pieces of Synapse (and much more), at the time of writing there isn’t an automated migration tool from one to another.
The aim of this POC is to understand if a migration is feasible, which could be the impact in terms of effort and money and which are the pitfalls to avoid.
This series of short posts will be a collection of what we discovered during this POC.
The following index will be updated any time a new chapter is released.
- Chapter 1: The exodus of a pipeline (this post)
- Chapter 2: The downfall of the hierarchy… will it raise again?
- Chapter 3: The Lord of the Lakes
- Chapter 4: Breaking down the External Table walls
Please, be aware that Microsoft Fabric is a very mutable product, and details in this posts may change as time goes by.
I have to rewrite my pipelines
As you may know Microsoft Fabric and Azure Synapse incorporate an engine to orchestrate activities for our ETL/ELT workloads in fact they both have something called “Pipelines”.
Both Fabric and Synapse Pipelines are based on Azure Data Factory technology but you’ll easy find out that there’s no way to migrate pipelines from Synapse to Fabric.
If you’re an experienced ADF or Synapse Pipeline developer you may think to create all the pipelines “empty” and to recreate the flow copying the JSON definition of the pipeline. It could be a great idea to cut manual work, but you’ll quick realize that, as of now, Fabric pipelines’ JSON view is read-only.
Another approach could have been using APIs to create pipeline based on a JSON definition but, as of now, pipelines aren’t included in the list of supported items that can be created via APIs.
If there’s some brilliant approach that I’m missing I’d be more than happy to discuss about it, so feel free to comment on that.
Update 2024–01–30
Two brilliant guys came to me with an interesting approach that sounded promising.
In other words what Greg and Martijn proposed to create empty pipelines in the Fabric Workspace, connect it to git repo and this should have given the JSON code of the pipelines. Cloning the repo on PC we would have tried editing it manually using VSCode or whatever, pasting the JSON code from ADF/Synapse. After that we would have pushed back to the source and synced the Workspace with git.
Long story short even if it’s a brilliant workaround pipelines are not a supported item in git yet.
Update 2024–07–17
TLDR: the Edit JSON in Fabric pipelines is here, read more about it here.
Conclusions
If you’re trying to embrace Microsoft Fabric, and you want to migrate ADF and/or Synapse Pipelines into Fabric you are obliged to draw your pipelines manually from scratch.
See you on the next chapter talking about pipelines organization.
