Tales from the Fabric — Chapter 2
The downfall of the hierarchy… will it raise again?
Prelude
In this blog post series we’re telling tales from a POC with the aim to understand if a migration from Synapse to Fabric is feasible, which could be the impact in terms of effort and money and which are the pitfalls to avoid.
The following index will be updated any time a new chapter is released.
- Chapter 1: The exodus of a pipeline
- Chapter 2: The downfall of the hierarchy… will it raise again (this post)
- Chapter 3: The Lord of the Lakes
- Chapter 4: Breaking down the External Table walls
And then there was chaos
In our ADF/Synapse projects we always try to apply the “Divide et Impera” approach, making pipelines “atomic” and orchestrating them through a set of “Masters”.
Benefits of this approach are out of scope of this post but the drawback is you’ll easily get “factories” with dozens of pipelines, therefore we apply the “Keep it clean” approach adhering to meaningful naming convention and organizing pipelines in folders.
Pipelines are developed and deployed directly in a Fabric Workspace that has no grouping/hierarchizing feature yet for pipelines (something automatic exists for Warehouses and Lakehouses to organize default semantic models and SQL enpoints).
This means that as of now, all your pipelines will be in the same “parent folder” (i.e. the workspace) and you can’t organize them any further, therefore it’s mandatory to implement a strict and clear naming convention policy.
Good news
Looking to the Fabric Release Plan it seems the day we can organize objects in Fabric Workspaces is close (Q1 2024) and this could rule the chaos.
Update 2024–04–17
Folders are now available in Public Preview with some limitations, for example Dataflows Gen2 are not supported as well as Git integration.
With this feature we could reproduce the pipeline organization we had in Synapse.
Conclusions
Having many pipelines could be easy if you try to keep them simple and atomic. At the time of writing we can’t organize pipelines in Fabric Workspaces as we were used to in ADF/Synapse but it seems this feature will be rolled out in Preview in Q1 2024.
See you on the next chapter.
