Monitoring data from IoT devices with Golang, Google Cloud Platform and Grafana

How to set up a serveless monitoring environment on Google Cloud Platform for IoT devices.

In this article, we’ll be setting up step by step, a monitoring environment for IoT devices with an Arduino based board (ESP32), Grafana, and Google Cloud Platform tools. We’ll be using other tools like Docker and Prometheus, you don’t need to know how to use them, but I highly recommend that you know what they are and what they do.

First, let’s take a look into our infrastructure:

We’re using the built-in temperature sensor from ESP32 to sent data every 5 seconds to a pub/sub service in GCP. This data will be processed by a Go service and sent to Prometheus, and we’re using Grafana to visualize this data.

Grafana is a visualization tool. Prometheus is a data source for Grafana that collects the data in time series and displays in a way that Grafana understands. Since Prometheus can’t collect data directly to pub/sub, we need a third service to send it to Prometheus.

I made a Github repository with all the codes used and instructions in this article, fell free to check it out.

leozz37/iot-monitoring-gcp-grafana

So let’s get our hands dirty!

Getting started with Google Cloud Platform

On Google Cloud, we’ll be using Core IoT to manage our devices, pub/sub as messaging system and Google Run to host our containers.

First, let’s set up our project. You’ll need a Google account and a credit card, but don’t worry you won’t be charged for anything (if you don’t do some heavy work), your free trial lasts for 3 months and you have US$300 to spend in any Google Cloud service. But you can always keep an eye on your billing board to not have any surprises on your credit card.

I’ll use the Google Cloud CLI to set up our environment but feel free to use the web console. Install it for your operational system.

Installing Google Cloud SDK | Cloud SDK Documentation

To make things easier, you can export this environment variables and just paste the commands from this tutorial (choose your own names):

export PROJECT_ID=
export REGION=
export TOPIC_ID=
export SUBSCRIPTION=
export REGISTRY=
export DEVICE_ID=
export USER_NAME=
export IMAGE_NAME=
export SERVICE_NAME=

The export command should look like this:

export PROJECT_ID=temperature-grafana

To start, log in with your Google account on CLI, create a project, and select the project created. Open a terminal and type the following commands:

$ gcloud auth login

$ gcloud projects create $PROJECT_ID

$ gcloud config set project $PROJECT_ID

You can check your project dashboard, and if everything goes well, you should see your project there.

Now let's enable pub/sub and IoT Core services in our project. But before that, you'll need to enable the billing into your project. To do that, run the following command and continue to the browser and link a profile:

$ open "https://console.cloud.google.com/billing/linkedaccount?project=$PROJECT_ID"

$ gcloud services enable cloudiot.googleapis.com pubsub.googleapis.com

We also need to permit IoT Core to publish into pub/sub service. Since IoT Core is responsible for our devices, and they don’t need to subscribe to any topic, we’re giving them just the publishing role.

$ gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:cloud-iot@system.gserviceaccount.com \
--role=roles/pubsub.publisher

Choose a region here. I’m using us-central1, but pick the one that better suits you. We also need a pub/sub topic, a subscription, and a registry.

On MQTT, pub/sub works like an Instagram/Twitter hashtag, where you can publish a post using a hashtag and who is following (or subscribed) to that hashtag, we'll see your post. The same works for MQTT, but the hashtag is the topic, the photo is the message and the people following that topic is the subscription.

A registry is like a bucket for our IoT devices. It allows us to group devices and set properties that they all share, such as connection protocol, data storage location, and Cloud pub/sub topics.

Follow these commands:

$ gcloud pubsub topics create $TOPIC_ID

$ gcloud pubsub subscriptions create --topic $TOPIC_ID $SUBSCRIPTION

$ gcloud iot registries create $REGISTRY \
--region=$REGION \
--event-notification-config=topic=temperature-topic \
--enable-mqtt-config --enable-http-config

You can check your registries, and if everything goes well, you should see your registry with your topic and subscription there.

Setting up ESP32

We’ll be using the Espressif micro-controller ESP32 for its WiFi and a built-in temperature sensor. Also, I’m using the Arduino IDE, so make sure you have it installed and set up for ESP32 usage, if you need some help you can follow this tutorial:

Installing ESP32 in Arduino IDE (Windows, Mac OS X, Linux) | Random Nerd Tutorials

We need to generate an Elliptic Curve (EC) ES256 private/public key pair for our device authentication. Make sure to generate them into a “safe place”:

$ openssl ecparam -genkey -name prime256v1 -noout -out ec_private.pem
$ openssl ec -in ec_private.pem -pubout -out ec_public.pem

Now we have to register our device into Core IoT, so run the following commands:

$ gcloud iot devices create $DEVICE_ID \
  --region=$REGION \
  --registry=$REGISTRY \
  --public-key="path=./ec_public.pem,type=es256"

Install “Google Cloud IoT Core JWT” and lwmMQTT from Joel Garhwller libraries on your Arduino IDE. They’re responsible for connecting, authenticating, and sending messages to GCP.

Now let's use the library code example for ESP32-lwmqtt:

On ciotc_config.h, set your WiFi network and credentials:

// Wifi network details
const char *ssid = "SSID";
const char *password = "PASSWORD";
// Cloud iot details
const char *project_id = "project-id";
const char *location = "us-central1";
const char *registry_id = "my-registry";
const char *device_id = "my-esp32-device";

To get your private_key_str, run the following command at the same directory where you saved your public/private keys and paste the "priv" it into the code:

$ openssl ec -in ec_private.pem -noout -text

PS: The key length should be 32 pairs of hex digits. If your private key is bigger, remove the “00:” and if its smaller add “00:”. It should look like this:

You’ll need to set up your root_cert as well. Do the same steps as previously:

$ openssl s_client -showcerts -connect mqtt.googleapis.com:8883

It should look something like this:

If you're curious, I'm using Dracula Theme from Zeno Rocha.

On Esp32-lwmqtt.ino file, let's do some changes to get the ESP32 temperature. This is how our code looks like:

Now plug in your ESP32 into the USB and upload the sketch. Open the console, set it to 115200 baud rate and the logs should look like this:

Now if you check your pub/sub topics, you may see something like this:

Golang Metrics Converter

We need someone subscribed to the topic where our device is publishing and send those messages to Prometheus. So let’s create a service with Golang and put it into a container. First, we need a service-account and give it the pub/sub subscribe role:

$ gcloud iam service-accounts create $USER_NAME
$ gcloud projects add-iam-policy-binding $PROJECT_ID \
--member=serviceAccount:$USER_NAME@$PROJECT_ID.iam.gserviceaccount.com \
--role=roles/pubsub.subscriber

There are several ways to our service authenticate into GCP, so let’s use the JSON file method. So let’s generate a file with credentials for the service account we just created and export its path to an environment variable called GOOGLE_APPLICATION_CREDENTIALS:

$ gcloud iam service-accounts keys create resources/service-account-key.json --iam-account $USER_NAME@$PROJECT_ID.iam.gserviceaccount.com

$ export GOOGLE_APPLICATION_CREDENTIALS=$CREDENTIAL_PATH

The export command should look like this:

export GOOGLE_APPLICATION_CREDENTIALS=/home/leo/resources/service-account-key.json

Now let's create our Golang service:

And a Dockerfile for it:

Google Cloud Run

Cloud Run it's an awesome fully managed compute platform for deploying and scaling containerized applications quickly and securely. If you’re new to it, check this video

First, we need to enable Cloud Build, Cloud Run, and Container Registry in our project:

$ gcloud services enable cloudbuild.googleapis.com run.googleapis.com containerregistry.googleapis.com

Now let’s build and push our Golang service Docker image to the Cloud Build:

$ gcloud builds submit --tag gcr.io/$PROJECT_ID/$IMAGE_NAME

To deploy it, run this command:

$ gcloud run deploy $SERVICE_NAME --image gcr.io/$PROJECT_ID/$IMAGE_NAME --region us-central1 --platform managed --allow-unauthenticated --port 2112

GCP will generate an URL for your container, copy it. You can get it on your terminal or accessing your project on your project Cloud Run page.

Now paste it on your prometheus.yml file, on your targets (remove the https://):

Since we can't deploy existing images from Docker Hub to Cloud Run, we need to make a custom Docker image for Prometheus and Grafana, then deploy them. Let's deploy a Prometheus container:

Building and submitting to production:

$ gcloud builds submit --tag gcr.io/$PROJECT_ID/prometheus .

$ gcloud run deploy prometheus --image gcr.io/$PROJECT_ID/prometheus --region $REGION --platform managed --allow-unauthenticated --port 9090

Make sure to save the URL generated. Same thing for Grafana:

Building and submitting to production:

$ gcloud builds submit --tag gcr.io/$PROJECT_ID/grafana .
$ gcloud run deploy grafana --image gcr.io/$PROJECT_ID/grafana --region $REGION --platform managed --allow-unauthenticated --port 3000

Now you can access your Grafana board through the generated URL. You can log in with the admin login (default user: admin, pass: admin, make sure to change that).

Now we have to set up Grafana to listen to our Prometheus. After logging in, go to "Data Source" on the right menu bar, click on "Add data source" and select Prometheus.

On the Prometheus data source page, paste the URL to your Prometheus instance on the HTTP > URL and hit "save & test".

On my Github repository, there’s a JSON file that will import a Grafana Dashboard. Feel free to use it or create your own. If you looking into creating your dashboards, Grafana uses PromQL for querying metrics data, take a look into its documentation for more information.

leozz37/iot-monitoring-gcp-grafana

To import my dashboard, go to the right menu bar, then to Create and Import. Paste the JSON content into the text box and hit load.

Select Prometheus as your data source and boom, you should a dashboard like this one:

Now plug your ESP32 on the USB and you should see the graph going up and down!

And that’s it. You can monitor data from IoT devices anywhere in the world. If you have some trouble setting this up, feel free to contact me on my email or connect with me on my LinkedIn and Github.