Real-Time Analytics Using Microsoft Fabric
Microsoft recently revealed their newest end-to-end analytics tool called Microsoft Fabric. Its a tool that does all and the cherry on the cake is — its serverless. Essentially, with Fabric, you are only focusing on solving business problems and not worrying about maintenance overhead. Fabric comes with different experiences aimed at different functions of analytical pipelines. In this blog, I will be focusing on the Real-Time Analytics mainly along with Data Science and Data Engineering experiences.
For this introductory blog, you will be creating a streaming pipeline which will get sample data from a source, ingest it and store to a Lakehouse destination. Further, you will be processing the data in a notebook and storing it in delta format which is supported by Microsoft Fabric.
Pre-requisite: For this blog, you need to have a lakehouse already created in Microsoft Fabric.
1. Login to Microsoft Fabric at https://app.fabric.microsoft.com/
2. Once logged in, in the bottom left you will see a Power BI icon. Click on the icon and select Real-Time Analytics experience.
3. Now click on the Eventstream icon in the top pane under New banner.
4. As shown in the below figure, give your eventstream a name and click create. Note: It might take few seconds to load.
5. You will see a screen like the figure below and click on New Source.
6. Give your source a name and select Yellow taxi from the sample data as shown below and click create.
7. Next, click on New Destination and give it a name. Choose the workspace you want to work in and the lakehouse which you have already created. For delta table, you can give it a new name and leave the data format as json. Click Create.
8. Once both source and destination are created, it will start streaming and ingesting.
9. If you would like to preview the data, you can select the source or destination icon and click on Data Preview as shown below:
10. Next, click on the Real-Time Analytics icon in the bottom left and select Data Science experience. Here you should see the table name you gave in step 7.
11. Click on the ellipses next to the table you created and choose Load data -> Spark. It will create a cell populated with some code as show below. Hit Run next to the cell and it will display the top 20 rows of the data that you just streamed directly to your lakehouse. You can also visualize your data by selecting Chart view instead of Table.
12. Finally, if you would like, you can do some more processing on your data and save it in the lakehouse. In the figure below, I have only selected two columns from the original table and saved it in delta format in my lakehouse.
Note: In the last step, I have only selected two columns from the original table. This type of processing can also be done directly within the Real-Time analytics experience when you create a stream. This step is just to show you that it is possible in Data Science experience.
Today, you learned how easily you can stream, ingest and store your data using Microsoft Fabric and also perform further processing on it. Microsoft Fabric has really made it easy to do analytics on your data.
