IoT in Google Cloud Platform
In this post, we’ll simulate a collection of sensors data measuring device’s temperature. Data, once collected, will be accessible by users to:
- Store device’s temperature in real-time.
- Monitor the device’s temperature in real time.
- Perform analysis to extract insights.
- Get alerts on Mobile / Email.
The following components are represented on the diagram:
- Temperature sensors are simulated by running IoT python client on Docker container.
- The sensors send temperature data to an IoT Core Registry running on GCP through MQTT protocol.
- The IoT Core Registry publishes it into a Cloud Pub/Sub Topic.
- A Streaming Dataflow pipeline is capturing the temperature data in real time by subscribing to and reading from the PubSub topic.
- Temperature data is pushed into BigQuery for analytics purposes.
- The Data Studio can be created on BigQuery shows interactive dashboards.
- Cloud Function is also subscribed to the Cloud Pub/Sub Topic.
- The Cloud Function saves the temperature data to Cloud Firestore for real time querying.
- Firestore database maintains threshold for each device in Config table.
- If Temperature of the devices goes above or below threshold, Cloud function send message to another Pub/Sub Topic (Alert).
- This Alert has multiple subscribers: One subscriber is sending email to the user. Another subscribers can be created to send alert to WebApplication or sending message back to IoT core to set temperature on the device (Not implemented).
- Temperature is displayed in realtime on Web Application in Firebase (Not implemented).
- All components are logging data to Stackdriver.
Deploy this IoT app using Terraform
In order to complete this guide, you’ll need to install the following tools
- Terraform: This guide uses Terraform to deploy resources.
- Git: Git is used to clone the example code and trigger new deployments.
- GCP: You will need a GCP account with billing enabled.
- Python: Python is installed to run a script.
Create GCP Project
Create a GCP project for this tutorial.
Select Firestore mode
- Go to Firestore
- Select Native Mode
- Select a Location (e.g. United States)
- Click on “Create Database”
Create Service Account
- Create Service Account.
- Assign the roles: Editor
- Download the key and renamed it as terraform-key.json
Clone the Repository
Clone the following repository containing the sample code, then switch to the
$ git clone https://github.com/vikramshinde12/iot-on-gcp.git
Next, copy the
terrform.tfvars.example file to
terraform.tfvars. You will need to replace the value of the
In this tutorial, I have used sendgrid api to send the Email Alert. You can leave blank if you do not want to send email alert but just get function triggered.
Then, copy the service account key terraform-key.json to this folder.
Execute following command to set Google Credentials.
$ export GOOGLE_CLOUD_KEYFILE_JSON=terraform-key.json
Execute Terraform scripts
First initialize the Terraform.
$ terraform init
$ terraform plan
Now, apply the changed to the GCP platform.
$ terraform apply
This terraform has created components mentioned in Components section along with following:
- Created RS256 Keys (rsa public key and rsa private key)
- Registered the devices in the Cloud IoT core registry using rsa public key.
- Created threshold values for each devices in Cloud Firestore
- Created docker containers as simulated devices which is sending simulated temperature sensor data to Cloud IoT.
Once everything up and running, the simulated devices on docker containers will start sending random data to IoT. This data will be loaded into both Firestore and BigQuery for analysis.
If the temperature is above threshold limit set in the Firestore, it generates alert to Pub/Sub which has one subscriber, Cloud Function, at the moment. This cloud function sends email alert to the receiver mentioned in the config value.
Next in pipeline…
Following are the option that can be done with this data
- Create Web app or Mobile app in Firebase on Firestore.
- Trigger IoT device to automatically set temperature.
- Create Dashboard in Data Studio to get real-time insight.
First, permanently delete the resources created by Terraform
$ terraform destroy
Next, Stop the docker containers created
Finally, delete the project created for this tutorial
$ gcloud projects delete [project_id]
This way we have achieved End-2-End processing of IoT data in Google Cloud Platform.