Getting Started with GCP IOT — Integrating GCP IOT with Pub/Sub , Dataflow, BigQuery, Looker Studio

6 min readDec 15, 2022

In this article you will be get an idea how to connect IOT devices to GCP services and generate valuable insights from visualization graphs using Looker Studio. Here, we are integrating GCP IOT with Pub/Sub which is fetching all the data from the IOT devices and then sending to Dataflow which is processing IOT data through its pipelines and then we are using Bigquery to manage and analyze IOT data and to get visualization insights we are using Looker studio resource.

Step by Step Process

Enable Cloud Pub/Sub and Google Cloud IOT API to use those resources for this usecase.

Search IOT Core in Google Cloud Console and you will land into this page

On another tab open Cloud Pub/Sub Topic resource

Create a topic , add a topic id and put all configurations as default and click create topic

In the IOT Core , create a registry. Add Registry ID, region where you want to add it and select the topic which you created previously . This topic will receive all the messages from the IOT Core

On the advanced options, you can choose MQTT and click create.

Then create a device in IOT Core. Associate your IOT device with GCP Cloud .Add Device ID , “allow” device communication

To create Certificates for device, go to Cloud Shell and paste this command

$ openssl req -x509 -newkey rsa:2048 -keyout rsa_private.pem -nodes \
-out rsa_cert.pem -subj "/CN=unused"

Two certificates are created one public one (rsa_cert.pem) and one is private one (rsa_private.pem)

Add the certificate content of rsa_cert.pem into Authentication (Public key value) and format as RS256_X509 and create.

Then create subscription in Cloud Pub/Sub topic

Create a subscription using this command

$gcloud pubsub subscriptions create \
projects/PROJECT_ID/subscriptions/my-subscription \
--topic=projects/PROJECT_ID/topics/my-device-events

Now you can the subscription

Now you will be able to see the IOT device is configured correctly and communication is Allowed.

Then you clone my GitHub Repository where publisher.py code is present which is generating some data as speed , pressure, oilLevel, engineUpTime.

$ git clone https://github.com/NanditaSahu03/gcp-iot-demo-1/

Then do “npm install” to install Node.js dependencies

Run this command to connect a virtual device to Cloud IoT Core using the MQTT bridge, substituting your project ID.

node publisher.js \
mqttDeviceDemo \
 - projectId=PROJECT_ID\
 - cloudRegion=REGION \
 - registryId=iot-registry \
 - deviceId=iot-device \
 - privateKeyFile=rsa_private.pem \
 - numMessages=25 \
 - algorithm=RS256

Where mqttDeviceDemo is the method name in the publisher.py file and add cloud region, registryId and deviceId and privatekeyfile. We are sending 25 messages to topic.

We can see the messages when the subscriber is pulling it.

Then go to Dataflow resource and enable the API

Create Job from Template and fill the details — Job Name, Region, choose Dataflow Template as “Pub/Sub Topic to BigQuery” and add topic details and output as BigQuery table and add temporary GCS bucket location and click create.

Create a GCS Bucket and create a folder “temp” in it

Create a dataset in BigQuery

Add ProjectID, DatasetID and location and click create dataset

Then create table

Add project id , table name and add field name and type as per the values we are getting from Dataflow.

Send the IOT data from virtual device. See the Dataflow Job steps

We can see the data in BigQuery Table here

Then go to Looker Studio and connect to data — Choose BigQuery

Then choose custom query , then paste the query from BigQuery here and order by timestamp in ascending order ( Here we are printing all the columns coming from IOT virtual device )

Then choose Line chart rather than tabular format data to get better visibility

Then add date range dimension and dimension as timestamp (Date and Time) and choose metrics as speed, pressure, deviceName, oilLevel, engineUpTime.And in style add “Show points” for each metrics to get more insight.