Smart Home Cloud Services with Google: Part 2
In the previous post of this series, we explored using Cloud IoT Core and Firebase to build a device cloud for smart home devices. We saw how Cloud IoT Core enables us to securely connect constrained devices to Google Cloud, while Firebase constructs a user framework around our device data. As a quick review, here is the cloud service architecture we discussed last time.
Now, let’s look at extending this cloud service to integrate with the Google Assistant through smart home Actions. This enables users to link their account through the Google Home app and control their devices through any Assistant-enabled surface.
If you are unfamiliar with Actions on Google or smart home Actions for the Google Assistant. I recommend reading IoT & Google Assistant part 1 and part 2 by my colleague, Dan Myers, as a starting point.
As a developer, this brings the power of the Home Graph to your devices and gives them context within the user’s home. This context is what enables users to make natural requests, like “What is the temperature in the hallway?”, instead of referring to the device by name.
To build a smart home Action, create a new project in the Actions console. We will add two new features to our device cloud service and configure them in our console project: account linking and intent fulfillment. Let’s start by taking a look at how to integrate with the account linking process.
You can find the sample code described in this post on GitHub.
Account linking
Users authorize the Google Assistant to access their devices through account linking. This process enables the user to sign in to the account they use with the device cloud and connect the device managed by that account to Google. The Actions on Google platform supports several different account linking flows, but only the OAuth 2.0 Authorization Code flow is supported for smart home Actions.
To configure OAuth account linking, you need to supply two endpoints in the Actions console: one for authorization and the other for token exchange. The authorization endpoint is a web UI where the user can authenticate and agree to link their account with the Google Assistant. It must return an authorization code that uniquely identifies the user.
Since we are using Firebase Authentication for the client apps, we can add a new Angular route to our web client for the user to sign-in and return their Firebase ID token as the authorization code once they authorize access:
export class LinkAccountComponent implements OnInit {
redirectUri: string;
state: string;
idToken: string;
constructor(private authService: AngularFireAuth,
private route: ActivatedRoute) { }ngOnInit() {
this.authService.idToken.subscribe((token) => {
this.idToken = token;
}); this.route.queryParamMap.subscribe((params) => {
this.redirectUri = params.get('redirect_uri');
this.state = params.get('state');
});
}linkAccount() {
const next = new URL(this.redirectUri);
next.searchParams.append('code', this.idToken);
next.searchParams.append('state', this.state);
window.location.href = next.toString();
}
}
Snippet from web/src/app/link.component.ts
Once the authorization flow is complete, the Google Assistant calls your token endpoint to exchange the authorization code for a persistent refresh token. This token does not expire and remains valid unless the user chooses to revoke device access or unlink their account.
Using the Firebase Admin SDK, we can validate and decode the ID token to obtain the UID of the user. If the token is valid, we can generate a refresh token and associate it with the user’s UID in Firestore. This enables us to look up the token again later for validating future requests.
async function handleAuthorizationCode(request, response) {
// Auth code is a Firebase ID token
const decodedToken = await auth.verifyIdToken(request.body.code);
// Verify UID exists in our database
const result = await auth.getUser(decodedToken.uid); // Encode the user info as a JWT
const refresh = jwt.sign({
sub: result.uid,
aud: client_id
}, secret);
// Register this refresh token for the given user
const userRef = firestore.doc(`users/${result.uid}`);
await userRef.set({ 'refresh_token': refresh }); ...
}
Snippet from functions/smart-home/token.js
Firebase Authentication is an identity provider, but not a complete OAuth solution. This means our device cloud service must augment Firebase by minting and verifying tokens used for access to user data. The example code uses the JWT standard to create a self-encoded token with the following JSON payload:
{
"sub": uid,
"aud": client_id
"iat": issued_at_time
}We are using the JWT.io library for Node.js for all operations related to generating and validating tokens in this example.
The Google Assistant uses this refresh token to request an access token that will authenticate requests for device data. Our example service validates the refresh token signature and checks to make sure it’s the refresh token we expect for that user.
async function handleRefreshToken(request, response) {
const refreshToken = request.body.refresh_token;
// Verify UID exists in our database
const decodedToken = jwt.verify(refreshToken, secret);
const result = await auth.getUser(decodedToken.sub); // Verify incoming token matches our stored refresh token
const userRef = firestore.doc(`users/${result.uid}`);
const user = await userRef.get();
const validToken = user.data().refresh_token;
if (validToken !== refreshToken) throw new Error(...); // Obtain a new access token
const access = jwt.sign({
sub: result.uid,
aud: client_id
}, secret, {
expiresIn: '1h'
}); ...
}
Snippet from functions/smart-home/token.js
OAuth access tokens should expire, which requires the Assistant service to periodically request a new one using the persistent refresh token. This enables the user to revoke their authorization if necessary. The example access tokens contain the same self-encoded payload as the refresh token, but they expire after one hour.
Intent fulfillment
With the authorization and token endpoints in place, we are ready to begin implementing the fulfillment logic for the user’s devices. In this post, we will focus on implementing each intent in the context of our device cloud example, but you can find additional details on these intents and how they work together in the documentation.
We can use the Actions on Google Client Library for Node.js, which handles parsing the fulfillment requests and provides individual callbacks to handle each intent.
const { smarthome } = require('actions-on-google');
const fulfillment = smarthome();/** SYNC Intent Handler */
fulfillment.onSync(async (body, headers) => {
...
});/** QUERY Intent Handler */
fulfillment.onQuery(async (body, headers) => {
...
});/** EXECUTE Intent Handler */
fulfillment.onExecute(async (body, headers) => {
...
});/** DISCONNECT Intent Handler */
fulfillment.onDisconnect(async (body, headers) => {
...
});
Snippet from functions/smart-home/fulfillment.js
At the beginning of each handler, we need to validate the access token provided with the request. Since the access tokens our application provides are formatted as a JWT that has the user’s UID encoded inside, we simply need to verify the JWT signature using our application’s secret to ensure the token came from us, and check that it has not expired. All of this is handled automatically by the verify() method of the JWT.io client library for Node.js.
const jwt = require('jsonwebtoken');/**
* Verify the request credentials provided by the caller.
* If successful, return UID encoded in the token.
*/
function validateCredentials(headers, jwt_secret) {
if (!headers.authorization ||
!headers.authorization.startsWith('Bearer ')) {
throw new Error('Request missing valid authorization');
} var token = headers.authorization.split('Bearer ')[1];
var decoded = jwt.verify(token, jwt_secret); return decoded.sub;
}
Snippet from functions/smart-home/fulfillment.js
If the provided token is valid, the method will return the UID, which we will need in the intent handlers to query the proper device data. Let’s examine how our device cloud can interact with each intent: SYNC, QUERY, EXECUTE, and DISCONNECT.
SYNC
The Google Assistant sends a SYNC intent after account linking succeeds to request the list of available devices. The response tells the Google Assistant which devices are owned by the given user and their capabilities (also known as traits) of each device. This includes an identifier to represent the user (agentUserId) and a unique id for each device.
For the device cloud sample project, this means returning the list of metadata for all devices where the user’s UID is set as the owner.
fulfillment.onSync(async (body, headers) => {
const userId = validateCredentials(headers);
// Return all devices registered to the requested user
const result = await firestore.collection('devices')
.where('owner', '==', userId).get();
const deviceList = [];
result.forEach(doc => {
const device = new Device(doc.id, doc.data());
deviceList.push(device.metadata);
}); return {
requestId: body.requestId,
payload: {
agentUserId: userId,
devices: deviceList
}
};
});
Snippet from functions/smart-home/fulfillment.js
The SYNC response only contains the device types and their capabilities; it does not report any device state. Below is an example device entry in the SYNC response payload for a light bulb and thermostat:
{
id: 'light-123abc',
type: 'action.devices.types.LIGHT',
traits: [
'action.devices.traits.OnOff',
'action.devices.traits.Brightness'
],
name: {
name: 'Kitchen Light'
},
willReportState: true
},
{
id: 'thermostat-123abc',
type: 'action.devices.types.THERMOSTAT',
traits: [
'action.devices.traits.TemperatureSetting'
],
attributes: {
availableThermostatModes: 'off,heat,cool',
thermostatTemperatureUnit: 'C'
},
name: {
name: 'Hallway Thermostat'
},
willReportState: true
}Request Sync
When users add or remove devices associated with their account, you should notify the Google Assistant through the Home Graph API via Request Sync. Without this feature in your service, users must unlink and relink their account to see changes or explicitly say “Hey Google, sync my devices”. Calling the request sync API triggers a new SYNC intent to allow your service to provide updated device information.
In our example, we can observe when a device node is added or removed in Firestore, and request a sync in each instance. The HomeGraph API will throw an error if that user has not linked their account, so we also need to verify that a persisted refresh token exists for the user in Firestore (created during account linking).
const { smarthome } = require('actions-on-google');
const homegraph = smarthome({
jwt: require('./service-account.json')
});/**
* Cloud Function: Request a sync with the Assistant HomeGraph
* on device add
*/
functions.firestore.document('devices/{device}').onCreate(
async (snapshot, context) => {
// Obtain the device owner UID
const userId = snapshot.data().owner;
const linked = await verifyAccountLink(userId);
if (linked) {
await homegraph.requestSync(userId);
}
});/**
* Cloud Function: Request a sync with the Assistant HomeGraph
* on device remove
*/
functions.firestore.document('devices/{device}').onDelete(
async (snapshot, context) => {
// Obtain the device owner UID
const userId = snapshot.data().owner;
const linked = await verifyAccountLink(userId);
if (linked) {
await homegraph.requestSync(userId);
}
});
Snippet from functions/smart-home/request-sync.js
QUERY
The QUERY intent asks for the current state of a specific set of devices (noted by their ids). A QUERY may be sent by the Google Assistant in response to a voice command (e.g. “What is the current temperature in the hallway?”) or to update the UI in the Google Home app.
fulfillment.onQuery(async (body, headers) => {
validateCredentials(headers);
// Return device state for the requested device ids
const deviceSet = {};
for (const target of body.inputs[0].payload.devices) {
const doc = await firestore.doc(`devices/${target.id}`).get();
const device = new Device(doc.id, doc.data());
deviceSet[device.id] = device.reportState;
} return {
requestId: body.requestId,
payload: {
devices: deviceSet
}
};
});
Snippet from functions/smart-home/fulfillment.js
The device cloud sample project stores this data in the state field for each device using an internal representation of the device attributes. Our QUERY handler converts these attributes to match the device state values required by the Assistant for each trait. Our light bulb and thermostat devices declared support for the following traits:
Below is an example of the device entries in the QUERY response returning the state for each supported trait:
{
'light-123abc': {
online: true,
on: true,
brightness: 100
},
'thermostat-123abc': {
online: true,
thermostatMode: 'heat',
thermostatTemperatureSetpoint: '20',
thermostatTemperatureAmbient: '17'
}
}EXECUTE
When the user issues a command (e.g. “Turn on the kitchen light”), your service receives an EXECUTE intent. This intent provides a distinct set of traits to be updated for a given set of device ids. This allows a single intent to update a group of traits or devices simultaneously.
Here, we update the contents of the device-configs document for each device, which triggers Cloud IoT Core to publish the configuration change. As we discussed in the previous post, the device will report its new state to Firestore in the devices collection after the change is processed successfully.
fulfillment.onExecute(async (body, headers) => {
validateCredentials(headers);
// Update the device configs for each requested id
const command = body.inputs[0].payload.commands[0];
// Apply the state update to each device
const update = Device.stateFromExecution(command.execution);
const batch = firestore.batch();
for (const target of command.devices) {
const configRef = firestore.doc(`device-configs/${target.id}`);
batch.update(configRef, update);
}
await batch.commit(); return {
requestId: body.requestId,
payload: {
commands: {
ids: command.devices.map(device => device.id),
status: 'PENDING'
}
}
};
});
Snippet from functions/smart-home/fulfillment.js
The EXECUTE response must return a status code indicating whether each command was successful in changing the device state. If the cloud service can synchronously verify that the command reached the device and updated it, then it returns a SUCCESS. If the device is unreachable, then the service can report OFFLINE or ERROR.
Since our commands are written to Firestore through one document and the result is sent asynchronously back through another, we report PENDING rather than reporting SUCCESS. This indicates that we expect the command to succeed, and we will report the state change when it arrives.
Report State
Integrate the Report State API into your service to proactively report changes in device state to the Google Assistant. This is necessary to publish the latest device information to the Home Graph, which enables Google to look up device state without sending additional QUERY intents to your service.
In our example, we can define a new cloud function that triggers on updates to the devices collection. Recall from the previous post that this is where state updates from Cloud IoT Core are published. The function takes the updated device state and forwards it to the Home Graph API.
const { smarthome } = require('actions-on-google');
const homegraph = smarthome({
jwt: require('./service-account.json')
});/**
* Cloud Function: Report device state changes to
* Assistant HomeGraph
*/
functions.firestore.document('devices/{device}').onUpdate(
async (change, context) => {
const deviceId = context.params.device;
const device = new Device(deviceId, change.after.data()); // Check if user has linked to Assistant
const linked = await verifyAccountLink(device.owner);
if (linked) {
// Send a state report
const report = {};
report[`${device.id}`] = device.reportState;
await homegraph.reportState({
requestId: uuid(),
agentUserId: device.owner,
payload: {
devices: {
states: report
}
}
});
}
});
Snippet from functions/smart-home/report-state.js
The format of the states reported for each device is the same as a QUERY response.
DISCONNECT
Your service receives a DISCONNECT intent if the user decides to unlink their account from the Google Assistant. The service should invalidate the credentials used to provide access to this user’s devices.
For our example, this means clearing out the stored refresh token we generated during the account linking process. This negates any future attempts to gain a new access token until the user links their account again.
fulfillment.onDisconnect(async (body, headers) => {
const userId = validateCredentials(headers); // Clear the user's current refresh token
const userRef = firestore.doc(`users/${userId}`);
await userRef.delete(); // Return empty body
return {};
});
Snippet from functions/smart-home/fulfillment.js
What’s next?
Congratulations! Now your user’s devices are accessible through the Google Assistant. Check out the following resources to go deeper and learn more about building smart home Actions for the Google Assistant:
You can also follow @ActionsOnGoogle on Twitter and connect with other smart home developers in our Reddit community.
