Data Visualization using Databricks and PowerBI
In my previous post, I have explained the workflow of processing real-time data from sensors using Azure databricks and Event Hubs.
This post focuses on the data visualization for a given streaming datasets that can be easily on-boarded from databricks hive tables to PoweBI dashboard for drawing various visualization insights.
“Data Visualization is the biggest abstraction behind complex data analysis rules”.
Databricks has an inbuilt data visualization tool which can easily plot, map and bring popular KPIs and insights to the table. But when it comes to rich data visualization techniques, PoweBI can be a great tool that can be integrated with databricks tables.
- To get started within few steps, we need to first create a connection url.
Open the datatbricks cluster>Advanced settings>JDBC/ODBC connection
To create the connection url, use the HTTP path to copy the workspace id followed by the secret. sql/protocolv1/o/<workspaceid>/XXXX-XXXXX-
The bold part and the region should be pasted in the below connection url in this format.
https://<your_region>.azuredatabricks.net:443/sql/protocolv1/o/<workspaceid>/XXXX-XXXXX-XXX
2. Generate new token for Login password
Click on User Settings>Generate new token and note down the value. This token value will be later used as token password while logging into PowerBI
3. Open PoweBI desktop and search for spark in Get Data.
4. Copy paste the connection url which is created in step 1.
5. Set the username as token and Password as token id which was created in step 2
6. Finally, connection with databricks table is done and now, you will be able to access all the hive tables through PowerBI.
At the very beginning of this post I have created a Wind turbine dashboard out of the available data in databricks table.
