The complementary strengths of Databricks plus Microsoft Fabric
A powerful partnership for modern data platforms
By Phil Charles and Phil Hetzel
As a result of the mainstream emergence of generative AI (GenAI) in 2023, organizations have turned to analytics teams and leaders to be accountable for new competitive advantages. Many of these teams find themselves in unfamiliar territory; what was once a team responsible for internal reporting is now looked upon as a driver of potential revenue. The new responsibilities demand enterprise maturation. Instead of analytics teams attempting to fulfill every need of their business stakeholders centrally, they should consider how to distribute analytical responsibility out to their peers in the business. After all, the business stakeholders will be closest to the problems they are trying to solve.
Databricks and Microsoft Fabric are two solutions that work well together to allow analytics teams to empower business stakeholders. In this article, we’ll share our point of view on how Databricks plus Fabric complement each other to provide a holistic, modern data platform enabling quick access to data, unlimited scalability for both storage and compute, business-enriched context, and easier self-service.
Let’s set the stage through an illustrative use case:
Contoso, a widget manufacturer, has recently modernized its data platform from on-premises relational databases to a lakehouse architecture on Azure to serve and operationalize a wider scope of analytics use cases for the business.
To realize the value of its data, Contoso has added IoT censors onto its manufacturing machines to capture real-time data about the efficiency and productivity of its plants. In addition, it is capturing product feedback from its website, while analyzing its customer service calls to identify opportunities to support better customer journeys.
Contoso modernized its data platform to capture these new forms of data, including customer service recordings and IoT data. The diversity of data sources (and variety of business use cases) demanded a flexible and scalable solution. Contoso selected Azure Databricks because of its ability to cover the widest range of potential technical and functional requirements demanded of an analytics platform. The analytics team at Contoso uses Azure Databricks to ingest data (structured, semi-structured, and unstructured), stage the data in the medallion architecture, and create business-ready, curated datasets for consumption. Contoso’s data scientists are creating and serving production-grade machine learning models through MLflow and Model Serving, taking care to use the built-in monitoring and versioning capabilities. Contoso is also entering the GenAI space with Azure Databricks, releasing its first Retrieval-Augmented Generation (RAG) use case employing the out-of-the-box foundation models and Mosaic AI Vector Search.
The Contoso centralized analytics team creates valuable outputs from its analytics platform as quickly as possible; however, Contoso business stakeholders find locating the correct data challenging. When business stakeholders do find the correct data, they are unsure about the nature, lineage, and definitions of the dataset. The uncertainty hinders business stakeholders from self-serving and creating their own analytical products. As a result, requests continue to pour into Contoso’s centralized analytics team, which struggles under the workload.
The challenge described above is a common use case where Databricks and Microsoft Fabric fit well together.
With the general availability release of Microsoft Fabric in November 2023, companies of all sizes are excited about its features but struggle to understand how it fits into an enterprise architecture, especially if they have adopted Databricks as their primary data platform.
Azure Databricks is a unified, open analytics platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale. With the release of Databricks Unity Catalog, it now offers centralized data governance features to manage and govern Databricks data assets. After the medallion architecture is fully built out in Azure Databricks, from ingest to a consumable semantic layer, Unity Catalog as the governance layer enables the true value creation you can get from your data.
These features include data lineage, metadata management, centralized security management, AI-generated enrichment features, and data classification. These features offer high value for IT professionals, data engineers, data scientists, and technical data analysts who are comfortable curating and querying data with PySpark, SQL, R, or Scala (i.e., coders).
When focusing on the needs of the business, Azure Databricks still lacks low code/no code features to enable business self-service. But Unity Catalog makes this easy to do as a downstream capability, and this is where Fabric really shines.
Microsoft Fabric, a unified data platform complementing Azure Databricks, excels in delivering user-friendly analytics for business end users. For organizations that have invested in and adopted Azure Databricks as their centralized data platform, integrating Microsoft Fabric can enhance business operations. It facilitates the final steps in data analysis and reporting with low-code features, a centralized semantic layer, and self-service data discovery.
Addressing Contoso’s operational challenges, the combination of Microsoft Fabric and Azure Databricks allows business stakeholders to independently manage their data discovery and reporting. Fabric’s low-code capabilities enable any business user to enhance datasets or apply business logic using external files. It offers a robust semantic layer with semantic models (formerly known as Power BI datasets), allowing business users to create centralized datasets and dynamic business measures for Power BI reports. The AI-assisted development features, referred to as “Copilots,” help various user personas to quickly develop reporting assets using natural language GenAI. Additionally, Microsoft Fabric integrates seamlessly with Microsoft Purview to offer data glossaries.
The trend of federating analytics to business stakeholders should be encouraging to the C-suite. In a cost-conscious environment, leaders should be avoiding the age-old churn of requesting data products from a centralized team, waiting for the request to rise to the top of the queue, and then giving feedback when the requirements inevitably have changed.
Slalom has been working with clients through use cases like Contoso’s to help realize value from their data assets in a governed way. This includes offering workshops to learn more about Microsoft Fabric and Azure Databricks, and how they can perform better together. Take a moment to learn more about Slalom and our expertise with Microsoft and Databricks. We’d love to chat about how we can help you today.
Slalom is a next-generation professional services company creating value at the intersection of business, technology, and humanity. Learn more and reach out today.
