Synchronicity with Twilio Sync

Photo by Gabriel Gusmao on Unsplash

This is part 2 of a series about how Dwelo software uses Twilio Sync as its IoT communications platform.

As discussed in Part 1 of this series, Dwelo uses Twilio Sync as its device state source-of-record. This post will dive into the details about how that state is synchronized across the Twilio cloud, the Dwelo cloud, user apps, and IoT devices.

At a glance, Dwelo stores per-device state in document structures in Twilio Sync. At any given point in time, this document should be the source of truth for what state devices are in. Therefore, each update to this state is a result of an environment triggered (temperature or humidity reading) or actor triggered (physically flipping a light switch or changing the heat setpoint through your app) update to the state. If you imagine all the individual deltas to the document, or state changes, as a time series of edits, you get a natural time series of your sensor readings and commands! We’re about to dive into the structural mechanics of representation and access for this process.

The Twilio Sync cloud maintains a map of key/value pairs representing each IoT device in the system, including the Dwelo hub, the Z-Wave devices in each apartment unit, and any third-party devices that we connect with. Every device in a Dwelo system has a unique id — device_uid — and each device has a corresponding Sync map in the Twilio cloud whose name is this unique id.

For example a device with a device_uidof “abc-123-efg” has a Sync map that looks like this.

"abc-123-efg": {
  key: value,
  key: value,
  key: value
}

In each key/value pair, the string key identifies which element of state is being described, and the JSON value contains the current state of that element.

For instance, a door lock has a key/value pair to represent its locked state called DoorLocked. This key/value pair becomes the source-of-record for this piece of device state in the system. The lock will change the value of this key/value pair when a physical change happens to the lock state, and clients will be quickly notified of those changes.

Keeping data in a dictionary lends itself well to the varied nature of state data. For instance, thermostats have more sensors associated with them, so the Sync map for a thermostat would have a longer list of keys than a door lock, such asThermostatHeatSetpoint and ThermostatMode. Notice that we take advantage of having our value field as JSON by keeping other metadata — such as timestamp — in the record.

"hij-456-klm": {
  "ThermostatHeatSetpoint": {
    "state": {
      "unit": "F",
      "value": 65
    },
    "timestamp": "2019-10-08T15:39:40Z"
  },
  "ThermostatMode": {
    "state": "Heat",
    "timestamp": "2019-10-07T22:09:04Z"
  }
} 

Each Sync map also makes use of a key/value pair called “Command” to represent the desired state for each device, as shown below. This will be explained later.

APIs

Sync maps are accessed by device_uid using either the Twilio REST API -

sync.twilio.com/v1/Services/<service_sid>/Maps/<device_uid>/

Or using an MQTT client with the device_uid as the topic name -

sync/maps/<device_uid>/

At Dwelo, our cloud uses the REST API, and our hubs use MQTT. Recently, Twilio deprecated the MQTT interface, but we will continue to describe our usage of it here for accuracy.

Only authorized clients may access the Sync map for a device. In the Dwelo system, permissions are restricted such that the only clients with access to a Sync map are the hub that the device is connected to, and users with appropriate permissions to the hub. More on permissions below.

A Complete Example

Sensor State

The Sync map for a door lock contains key/value pairs for each piece of state the lock manages, such as its battery level, and the state of the lock.

"BatteryLevel": {
  "state": 70,
  "timestamp": "2020–03–18T05:18:47Z"
},
"DoorLocked": {
  "state": false,
  "timestamp": "2020–03–23T19:51:11Z"
}

When the door lock reports a state change to the hub via Z-Wave, the hub then publishes the state change to Sync using its MQTT client.

Once updated, the Sync cloud broadcasts the updates to all subscribed clients in an update that looks like this.

{ 
  "MapUniqueName": "abc-123-efg", 
  "EventType": "map_item_updated", 
  "DateCreated": "2020–03–23T19:51:11Z", 
  "ItemKey": "DoorLocked", 
  "ItemRevision": "1bd3", 
  "ItemData": '{"state": false, "timestamp":"2020–03–23T19:51:11Z"}' }

Because Sync persists each map in its cloud, clients don’t have to be online to receive notifications. For clients that don’t maintain a persistent connection, state changes can be retrieved by having MQTT fetch the undelivered messages for the topics the client is subscribed to, or a client can fetch the Sync map from Twilio’s cloud using the REST API.

Commands

To send commands to a device, there are three special keys in each Sync map that are manipulated by the Dwelo cloud and the hub — Command, CommandActive, and CommandResult.

When the Dwelo cloud API receives a command from a client (iOS, Android, Web, etc), the API first verifies client permissions. It then looks up the Sync map for the target device and examines the Command value of the Sync map for the device. It then proceeds according to the following:

1. If there is a Command value already present, with a timestamp recent enough to be in the command timeout window, the API returns a 409 Conflict response to the client. The implication is that there can only be one active command within the command timeout window.
2. If there is a Command value present with a timestamp old enough to be outside of the command timeout window, the API can overwrite the value with a new command. The former command is old and can be discarded, regardless of what state it is in.
3. If there is no Command value present, the API can write the field with the new command.

For our door lock example, to change the state of the lock, the Dwelo cloud updates the Command key in the locks Sync map and sets the desired_state field to be the new desired state for the lock.

Once it is notified of the Sync map update, the Dwelo hub determines which Z-Wave command(s) are necessary to put the door lock into the desired state. When the hub is ready to send the Z-Wave command, it first sets a CommandActive key in the Sync map, and sets the timestamp to the current time. This lets clients know that the hub is trying to implement the desired state in the device.

"CommandActive": {
  "node_id": 7,
  "timestamp": "2020-03-23T17:00:00Z"
}

When the hub has finished executing the command, it changes the CommandResult key to reflect the overall success or failure of the command. Clients that are subscribed for changes in the Sync map will then update appropriately.

"CommandResult": {
  "failure_string": null,
  "node_id": 7,
  "result": true,
  "timestamp": "2020-03-23T17:00:01Z"
}

Photo by Free To Use Sounds on Unsplash

Do You Have Permission to Do That?

Twilio Sync offers several levels of security for controlling access to objects in the Sync cloud. At the lowest level is a system of fine-grained ACLs that can be explicitly applied to every Sync object. Dwelo enforces these ACLs on every Sync map created to prevent unauthorized attempts to view device state by a client who should not have access.

When an apartment unit is provisioned, a Dwelo hub is installed and assigned to the unit. As part of the provisioning process, a Sync map is created for the hub and the hub is granted permission to read and write to the Sync map. Then, as Z-Wave devices are added to the apartment, the hub is granted permission to read and write to the Sync maps for each device.

Doing this prevents other hubs from having access to Sync maps for Z-Wave devices in other apartments.

Twilio Sync for IoT

NOTE: as mentioned above, Sync for IoT is now deprecated. We will describe our usage of it here for accuracy.

Our hubs use an MQTT client to securely communicate with the Sync cloud. This connectivity is facilitated by Twilio’s Sync for IoT product.

Dwelo uses Sync for IoT to organize our hubs into fleets and deployments. We have separate fleets for our production, staging, QA, and development environments. Inside of each fleet are one or more deployments. These deployments keep hubs logically isolated from the hubs in other fleets and deployments.

Each hub in a fleet must register a certificate with Sync in order to connect to Sync. The X.509 certificates are created by our hubs along with a private key, and the cert is passed to our cloud during provisioning. Our cloud creates a new logical device in the current Sync deployment, and instructs Sync to bind the certificate to that device object. Once bound, the Dwelo hub can then connect directly to the Sync cloud using an MQTT client initialized with the certificate and private key (mTLS).

Upon connection with the Twilio Sync cloud, the hub subscribes to the Sync map object that represents the hub, as well as the topics for the Sync map objects of the Z-Wave devices that the hub controls.

When the hub wants to write the updated state for a device it controls, it simply publishes to the topic with the key name appended. For instance:

# Updated state to publish
device_uid = "abc-123-efg"
key = "DoorLocked"
value = {
  "state": True,
  "timestamp": "2020-03-22T06:30:00Z"
}
self.sync_client.publish(
  topic=f"sync/maps/{device_uid}/{key}", 
  payload=value
)

The MQTT protocol is designed to have low network overhead, which is important for hubs like ours that may be connected to the Internet with a cell modem.

MQTT also supports a clean_session flag which can be set to trueso that if the client disconnects from Sync but reconnects later the previous session is restored and the client will receive any pending updates. In our case we set this flag to false, since it is not necessary for our hub to replay outdated commands from the cloud.

In part 3 of this series, we’ll describe our architecture for handling updates to Sync maps in the Dwelo cloud.

If you missed part 1, you can read it here.