Getting Started with GCP IOT — Integrating GCP IOT with Pub/Sub , Dataflow, BigQuery, Looker Studio
In this article you will be get an idea how to connect IOT devices to GCP services and generate valuable insights from visualization graphs using Looker Studio. Here, we are integrating GCP IOT with Pub/Sub which is fetching all the data from the IOT devices and then sending to Dataflow which is processing IOT data through its pipelines and then we are using Bigquery to manage and analyze IOT data and to get visualization insights we are using Looker studio resource.
Step by Step Process
Enable Cloud Pub/Sub and Google Cloud IOT API to use those resources for this usecase.
Search IOT Core in Google Cloud Console and you will land into this page
On another tab open Cloud Pub/Sub Topic resource
Create a topic , add a topic id and put all configurations as default and click create topic
In the IOT Core , create a registry. Add Registry ID, region where you want to add it and select the topic which you created previously . This topic will receive all the messages from the IOT Core
On the advanced options, you can choose MQTT and click create.
Then create a device in IOT Core. Associate your IOT device with GCP Cloud .Add Device ID , “allow” device communication
To create Certificates for device, go to Cloud Shell and paste this command
$ openssl req -x509 -newkey rsa:2048 -keyout rsa_private.pem -nodes \
-out rsa_cert.pem -subj "/CN=unused"
Two certificates are created one public one (rsa_cert.pem) and one is private one (rsa_private.pem)
Add the certificate content of rsa_cert.pem into Authentication (Public key value) and format as RS256_X509 and create.
Then create subscription in Cloud Pub/Sub topic
Create a subscription using this command
$gcloud pubsub subscriptions create \
projects/PROJECT_ID/subscriptions/my-subscription \
--topic=projects/PROJECT_ID/topics/my-device-events
Now you can the subscription
Now you will be able to see the IOT device is configured correctly and communication is Allowed.
Then you clone my GitHub Repository where publisher.py code is present which is generating some data as speed , pressure, oilLevel, engineUpTime.
$ git clone https://github.com/NanditaSahu03/gcp-iot-demo-1/
Then do “npm install” to install Node.js dependencies
Run this command to connect a virtual device to Cloud IoT Core using the MQTT bridge, substituting your project ID.
node publisher.js \
mqttDeviceDemo \
- projectId=PROJECT_ID\
- cloudRegion=REGION \
- registryId=iot-registry \
- deviceId=iot-device \
- privateKeyFile=rsa_private.pem \
- numMessages=25 \
- algorithm=RS256
Where mqttDeviceDemo is the method name in the publisher.py file and add cloud region, registryId and deviceId and privatekeyfile. We are sending 25 messages to topic.
We can see the messages when the subscriber is pulling it.
Then go to Dataflow resource and enable the API
Create Job from Template and fill the details — Job Name, Region, choose Dataflow Template as “Pub/Sub Topic to BigQuery” and add topic details and output as BigQuery table and add temporary GCS bucket location and click create.
Create a GCS Bucket and create a folder “temp” in it
Create a dataset in BigQuery
Add ProjectID, DatasetID and location and click create dataset
Then create table
Add project id , table name and add field name and type as per the values we are getting from Dataflow.
Send the IOT data from virtual device. See the Dataflow Job steps
We can see the data in BigQuery Table here
Then go to Looker Studio and connect to data — Choose BigQuery
Then choose custom query , then paste the query from BigQuery here and order by timestamp in ascending order ( Here we are printing all the columns coming from IOT virtual device )
Then choose Line chart rather than tabular format data to get better visibility
Then add date range dimension and dimension as timestamp (Date and Time) and choose metrics as speed, pressure, deviceName, oilLevel, engineUpTime.And in style add “Show points” for each metrics to get more insight.
Then click “Add” and you will be able to see the Chart formed.
Link for the Looker Studio Chart : Click Here
To use GitHub Publisher Code : Click Here
Buy me a coffee :) ← — — If you like my articles