Configuration and State in Google Cloud IoT Core
How to work with devices online and offline in IoT Core
The Google Cloud IoT Core Service is being discontinued and migrated to partner companies. Please find more info here. This article was about to be published when they broke the news. I kept it in the drawer for a while, then I decided to distribute it as a publicly accessible item. It could still be useful to people implementing Google’s partners’ solutions on IoT Core-compatible services.
About Digital Twins and “Shadows”
IoT devices are often operated through their virtual representations in the Cloud. The most common reasons to work with such virtualized devices, rather their physical self are:
- Running simulations and backtesting to predict their behavior or validate models.
- Access the last known device state or file a request for it to execute a command or change its state as soon as possible.
The first case listed above is commonly referred to as a “Digital Twin”.
It is meant to store a history of states and telemetry data from a device (or a fleet) and their operating conditions, so this data can be used later to build prediction models or simulate the possible outcomes under different operating conditions. As Wikipedia puts it:
A digital twin is a virtual representation that serves as the real-time digital counterpart of a physical object or process.
The second and very common case is what AWS refers to as a “Device Shadow”: a reliable way to interact with a remote device in full read-write mode, regardless of its connection status.
There are many possible reasons for an IoT device to go offline: the mobile or Wi-Fi network coverage may be bad in its current location, the battery may be depleted, the user is power-cycling it, a problem in the network…
so it’s convenient to ensure business continuity and a smooth user experience through the ability to read the latest known device state data from the Cloud and “leave a message” for the device, that will be delivered as soon as it comes back online.
The first use case (“Digital Twin”) would require a longer discussion, so the scope of this article is restricted to the very mundane use of “device shadows”: ensuring smooth operations on devices.
“Shadows” and “Twins” in Google Cloud IoT Core
IoT Core does not provide any explicit “shadow” or “digital twin” functionality, but it offers a light-weight version of them with the built-in State and Configuration management of a device.
State
Devices connected to IoT Core are expected to report their current state.
As the official documentation puts it:
Device state updates are typically triggered by a change to the device [..] or a similar change from another external source, such as a firmware update.
To update a device’s state in the Cloud, the device needs to publish its state over either the MQTT bridge or the HTTP bridge. The state will be:
- intercepted and automatically stored in a limited history storage (the 10 most recent states per-device are retained, at the time of writing), this storage can be accessed via the API or the SDKs
- dispatched via Pub/Sub to any subscribers of the destination topic
If you don’t need to keep a long history of the device states, just stick with the built-in State storage. For any real-world project I would recommend to store a longer history in a data store: that’s valuable information, there is a lot to be learned by analyzing this data at scale and it can provide valuable insights to the Customer Support teams as well.
Configuration
Configurations are stored in the Cloud and:
- promptly pushed to devices connected to IoT Core via MQTT
- delivered over MQTT to devices every time they connect to IoT Core (provided that the device subscribes to the correct topic; if no new configurations were set, the latest existing version is pushed every time)
- can be pulled via the IoT Core HTTP API
the Cloud also tracks which versions of a configuration were actually delivered to a device and the timestamps of creation and delivery. A configuration that was not yet delivered has a “null” delivery timestamp.
A delivered configuration is not necessarily an applied configuration.
That’s why devices should always push their State to the Cloud.
As for the States, the 10 latest configurations versions are retained.
Comparing State and Configuration
To verify that a Configuration was applied, a device should report the actual values for the same keys of Configuration object in the State. So going back to the Virtual Smart Plug demo project:
- The State of the Smart Plug is:
{ "switch": "off" } - A new Configuration is sent:
{ "switch": "on" } - If the Configuration was successfully applied, the Smart Plug should now report a new State:
{ "switch": "on" }
For very simple devices, the same “schema” can be used for both the Configuration and the State.
In a real world project, the State usually contains more information than just the Configuration of a device. A smart plug is a rather simple device, but while its Configuration might be down to a simple on/off status of the switch, its State likely also contains the current used power for display in the device client applications (a mobile app or a Smart Home dashboard).
Google encourages the use of the same JSON schema for both the Configuration and the State. In fact, you select a State in the device dashboard and press the Revert button at the end of the State payload, the State will be sent back to the device as a Configuration.
Let’s try this on the Virtual Smart Plug.
Pick a State from the list and press the Revert button:
The list in the Configuration & State dashboard will change like this:
a new “pending” Configuration was created and is waiting to be delivered to the device.
If the device simulator is running (or the next time you start it) you will see this output:
and the Configuration & State will be updated accordingly:
If you use a hierarchical format for the State, such as JSON, the reported Configuration can be posted in a child node of the State. This still allows for an easy 1:1 comparison of the Configuration JSON against the child JSON node of the State (just not with the built-in dashboard tool). Configuration and State are stored and delivered as plain text on the Cloud, but any format that can be serialized to text can be used.
Since both Configuration and State enforce a 64KB limit on the payload size, I would recommend against any format with complex markup and closing tags, to save some bytes for the actual information.
Google itself uses JSON for Configuration and State in its documentation.
Commands vs. Configurations
Google Cloud IoT Core also allows to send commands to devices using MQTT.
At first, this might be a little confusing. What’s the difference between commands and configurations?
I use this simplified framework to make design choices:
- Any short-term action that needs immediate execution is a command.
Think of a physical, digital payment device for a dispensing machine: transactions need to be processed quickly. If it can’t connect to the servers, it should just return an error. Deferring a transaction might lead to unintended consequences, because the user might just walk away, thinking that the sale was canceled. - Long-term changes to the device status or behavior that can tolerate a deferred application are configurations.
Think of smart bulbs installed in a patio: a routine should turn the lights on at dusk (~6pm) and off before bedtime (~10pm). If the network is unavailable between 6pm and 6.15pm, you’d still want the lights to be turned on for the evening and some delay might be tolerated.
There are also some subtle technical differences between commands and configurations that are best described in the official Google documentation.
It’s also worth noting that devices using HTTP connections cannot receive commands (but they can request configurations in pull mode).
