Creating data pipeline for streaming data using Azure Stream Analytics.

Call flow for stream pipeline will be as follows.

Services that we are going to use in Azure: — -

Ø Create an Azure IoT hub service — To receive streaming data from our simulator.

Ø Device for IoT data — Azure Raspberry Pi IOT web simulator (by this, we will get data per second).

Ø Azure Streaming Analytics — To process the streaming data (query will trigger it once input and output destination configured).

Ø Azure Storage — Blob storage as destination.

Ø Analyze the output data — We can also connect with Power BI.

1. We will create IOT Hub first.

2.Create Azure s stream analytics job.

Please note it says input, output missing and query not compiling as that need to be done.

3. Let us create final storage in blob where final live processed JSON data will be saved.

4. For live data, it has to come from some device or client, so I am using a Raspberry Pi IOT web simulator for live streaming data.

https://azure-samples.github.io/raspberry-pi-web-simulator/)

5. To connect to this device , we first add IOT device in IOT Hub and use connection string from that device which we will put in online raspberry simulator configuration.

6. Create IOT device in IOT hub.

7. Copy primary string..

8. Put that string on raspberry device connection string.

We will not run it as stream analytics still not configured with input, output and query.

9. Let us create input, output and query in stream analytics.

10. Now create output.

As we have created container in blob, all data auto populated when created output. Also, by default it pull all data including select blob/ADLS2 from your subscription.

11. Input and output created so need to create query.

When clicked query then found it was auto populated with command including input and output destination which we configured.

12. Started streaming by run it and now executed the query also.

14.

Now we can see streaming output data in blob storage.

Can see file size increasing.

15.

Above output data can be sent to power BI application if one has access.

We will delete resource group so no further cost incurred and created resource will get released.