Open in app

Sign in

Write

Sign in

A complete guide to voice-controlled home automation using Google Assistant (For a couple of $)

Aurélien Esprit
24 min readNov 22, 2018

--

Phones, Google Home & Google Mini, Amazon Alexa and other voice-control system are entering more and more homes every day. All the home automation actors are now enrolling to get their agents certified so it can be integrated to these solutions.

Who never dreamed as a child to control things with their voice?

In this article, I will describe the technical implementation that is common to all commercial solutions, where the fun resides :)

I will not cover these commercial solutions that are very often black boxes with little to no interests in understanding home automation.

At the end of this article, you will have the fundamentals to:

  • Build an ‘IOT’ device relying on ESP8266 (electronic device with GPIO & WIFI)
  • Pilote this device from a Raspberry Pi automation server, using NodeJS
  • Obtain and deploy a public SSL certificate on you raspberry Pi automation server
  • Deploy a RESTful API on your Pi automation server, that is reachable from internet (and Google Assistant device)
  • Build a Dialogflow Agent connected to your automation server to pilote your IOT device with the voice from your mobile device or your Google Home!

Total Cost: Less than 6 Euros! (assuming you already have the raspberry)

Duration: 2–10 hours of fun, depending on your initial level.

This article is quiet a long read, but I tried as much as possible to cut it down into independent / self explaining chapters, each chapter being related to one element of the call flow so you can pick what you want.

Ready? Lets Go!

“Ok Google, open the garage door”.

First things first, the prerequisites

Some basics knowledge about:

  • Raspberry / Linux
  • Electronics & Arduino programming language
  • NodeJS (or whatever language you are confortable with to expose a RESTful server + an HTTP Client)

Material:

Since all android devices benefits from Google Assistant (you may have to install it from the store if not already installed), the choice of Google Assistant + Google Dialogflow is the easy path.

The Call Flow of our scenario:

Google assistant is an interface embedded in mobile phones and in the Google Home / Mini allowing user to interact with google solutions (voice or keyboard)

Google Dialogflow is an AI that can collect some human user intents (an intent can be seen as a request for some service) and format them in a machine language. You could see Dialogflow as a human to machine translator.

Dialogflow relies on a webhook, that is an API able to provide the service requested to Dialogflow. Dialogflow will call this service with the user intent formatted in a machine language. In commercial solutions, this webhook is Cloud based, but in our case, we will expose it on a cheap raspberry with NodeJS.

This webhook will actually pilot the connected objects in the house. In our use case, it is a combination of an ESP8266 and a relay. The ESP8266 is an very cheap, small print, WIFI capable electronic device with GPIOs. That means, you can interact with it via your home WIFI to trigger some switches (relay), collect temperature, humidity, contactor information…

To see a progression along the tutorial, we will start with the last step of the call flow and rewind it. Let’s start with the last part of the call Flow: Raspberry => ESP.

Building our ESP driven component

First, a little bit of electronic to connect the relay to the ESP. The ESP PIN controlling the relay is PIN marked as D1. The resistor is 4.7 kOhm. The transistor is a 2n2222 type. On the relay, the pin marked as Vcc is connected to a 3.3V PIN of the ESP, the Pin marked as ground is connected to an ESP Ground PIN (GND), and the IN pin is connected to the transistor as bellow.

Now, let’s move on with the programming part.

The ESP can expose a web server and listen for some HTTP requests. This way, the Raspberry (or a web browser) can send some orders for execution. I use Arduino IDE to program the ESP and ESP NodeMCU v1.3. This device is very user friendly since it provides a micro USB connector for easy flashing / powering during tests.

I assume you know how to declare additional boards such as ESP NodeMCU to the Arduino IDE. You might also have to install some libraries if the IDE screams at compilation time. Plenty of tutorial on the net will help on these topics.

What we want to do: Make some HTTP GET Requests on ESP to read state / open / close the relay switch.

To read the relay state: http://MyESP_IP:MyESP_PORT/relay/info

To close the relay: http://MyESP_IP:MyESP_PORT/relay/ON

To open the relay: http://MyESP_IP:MyESP_PORT/relay/OFF

Note about security: Security is the poor cousin of IOT. This is no big news, you read those articles about all those IOT devices turned into zombie bots by hackers. For a commercial solution, it is a shame. In our case, the ESP does not support yet HTTPS. However it handles basic authentication mechanism and we will implement it. The ESP is exposed on your own WIFI. There is a risk of ‘Man in the middle’ attack that could intercept the login + password to pilot your device, if an hacker could hack first your WIFI.

The step by step ESP code is commented bellow.

// Including the ESP8266 WiFi library & others required
#include <ESP8266WiFi.h>
// Inluding the ESP8266 WebServer library to expose a Webserver.
#include <ESP8266WebServer.h>
// To handle JSON objects 
#include <ArduinoJson.h>// Declare your wifi parameters, SSID and Password, to be replaced with your network details
const char* ssid = “mySSID”;
const char* password = “mySSIDPassword”;// Relay control related information
// The relay control PIN. This GPIO will be set to 0 or 1, activating or deactivating the relay switch. Take care the board PINOUT (= physical PIN Id to Logical Pin Id mapping) depends on the board, search it over internet
int relayPIN = 5;
// The relay State (0 or 1), to keep track of relay status 
int relaystate;
// Declare relay location, this will be used in HTTP answers sent back by ESP to Rasberry Server
const char* relayLocation=”\”porte garage\””;// Retrieve unic ID of the ESP
uint32_t device_id = ESP.getFlashChipId(); //device_id;// The server will be exposed with Basic Authentication mechanism. Here we declare the user login + pass chalenged by the ESP Webserver
const char* www_username = “myESPServerLogin”;
const char* www_password = “myESPServerPassword”;
// And during an HTTP Request, we need to know if user is authenticated so we use a boolean
boolean authenticated;// Declare the server Port
int serverPort = 80
// Start Web Server on port 80
ESP8266WebServer server(serverPort);

Now add the void setup function. To connect to WIFI, and start the web Server.

void setup() {
 // The PIN we used must be declared as input or output PIN. 
// Our relay PIN is an OUTPUT PIN: The signal is sent OUT of the ESP to the relay to trigger it.
 pinMode(relayPIN, OUTPUT);
// Initializing serial port for debugging purposes
 Serial.begin(115200);
 delay(10);// Connecting to WiFi network
 Serial.println();
 Serial.print(“Connecting to “);
 Serial.println(ssid);// Start WIFI connection
 WiFi.begin(ssid, password);while (WiFi.status() != WL_CONNECTED) {
 delay(500);
 Serial.print(“.”);
 }
 Serial.println(“”);
 Serial.println(“WiFi connected”);// Handling the HTTP GET REQUESTs
/ For each route, we associate a function that will be called if the request hit the route:
 server.on(“/info”,HTTP_GET, info); 
 server.on(“/relay/on”,HTTP_GET, relayON); 
 server.on(“/relay/off”,HTTP_GET, relayOFF); 
 
 // Starting the web server
 server.begin();
 Serial.println(“Web server running. Waiting for the ESP IP…”);
 delay(2000);
// Printing the ESP IP address
 Serial.println(WiFi.localIP());
 
}

Now we will write the functions related to request handling. On HTTP request, we want to check the Basic Authentication credentials are well provided:

void checkAuth(){
 authenticated=true;
 if (!server.authenticate(www_username, www_password)) {
 Serial.println(“authentication issue”);
 authenticated=false;
 return server.requestAuthentication();
 }
}

Function to handle information requests over /info:

void info(){
 checkAuth();
// If user not authorized, quit the function.
 if (!authenticated){return;}
 
// Build the answered JSON
 StaticJsonBuffer<300> jsonBuffer;
 char JSONmessageBuffer[300];
 JsonObject& result = jsonBuffer.createObject();;result[“id”]=device_id;
 result[“ip”]=WiFi.localIP().toString();
 result[“type”]=”relay”;
 result[“commands”]=”ON:run HTTP GET on URL /relay/on, OFF: run HTTP GET on URL /relay/off”;
 result[“location”]=relayLocation;
 result[“status”]=relaystate;
 result.printTo( JSONmessageBuffer, sizeof(JSONmessageBuffer ) );
 
// Finally, send the success HTTP Code 200 along with the JSON answer.
 server.send(200, “application/json”, JSONmessageBuffer)
}

Function to change relay state to targetState, and functions to open and close the relay:

void setRelay( int targetState ) { 
 checkAuth();
 if (!authenticated){return;}
// Change relay state
 digitalWrite(relayPIN, targetState);
 relaystate = targetState;
 
 StaticJsonBuffer<300> jsonBuffer;
 char JSONmessageBuffer[300];
 JsonObject& result = jsonBuffer.createObject();;result[“id”]=device_id;
 result[“ip”]=WiFi.localIP().toString();
 result[“type”]=”relay”;
 result[“commands”]=”ON:run HTTP GET on URL /relay/on, OFF: run HTTP GET on URL /relay/off”;
 result[“location”]=relayLocation;
 result[“status”]=relaystate;
 result.printTo( JSONmessageBuffer, sizeof(JSONmessageBuffer ) );
 result.printTo(Serial);
 Serial.println(“\n”);
 server.send(200, “application/json”, JSONmessageBuffer);
}// Functions to  close the relay = power the connected equipment
void relayON(){setRelay(1);};
// Function to open the relay = shut down the connected equipment
void relayOFF(){setRelay(0);};

Finally, the main loop:

// runs over and over again
void loop() {
// Listen for incoming HTTP requests:
 server.handleClient();
 }

Using the console in Arduino IDE when starting the ESP, you will retrieve it’s IP, in my case 192.168.1.177.

Once the code is uploaded on the ESP, just open your browser and open your ESP URL on a browser:

http://192.168.1.177:80/relay/ON

Browser will ask for the login and pass. Use the one defined in your ESP code (myESPServerLogin & myESPServerPassword)

This should turn on the relay (you should ear a ‘clic’ ), and the browser should display something similar to:

{
“id”: 1458280,
“ip”: “192.168.1.177”,
“type”: “relay”,
“commands”: “ON:run HTTP GET on URL /relay/on, OFF: run HTTP GET on URL /relay/off”,
“location”: “porte garage”,
“status”: 1
}

Congrats, you are now able to locally pilot your garage door!

Next step is to write your webhook code (on the raspberry), so that when the API is called by Dialogflow, it triggers appropriate command on the ESP.

Building our ‘automation’ service on the Raspberry: The client part

I do use NodeJS here. It is lightweight and having an RESTful API up and running is a matter of minutes. There are plenty of modules out there, and you just have to connect the dots to build a working code very quickly.

I’ll assume you know how to install NodeJS and its module manager, npm on your server (Raspberry Pi) ,and how to install a module with npm.

The NodeJS service is composed of two interfaces:

· One on which the API listens for HTTP requests sent by Dialogflow (Restful server). It will be built on ‘express’, The module to expose a Restful server over NodeJS.

· One on which the API makes HTTP calls to the ESP (REST Client). It will rely on ‘axios’ that is a nice asynchronous HTTP client module.

To be able to run tests while we build the code, we will start with the HTTP client code, then write the RESTful server, and then connect both of them.

The Axios Client Code: (ESPClient.js)

// Import the axios module
axios = require('axios')// Create an async function that takes as input required information to communicate with ESP and the order to send ( = target relay state)
async function ControlRelay(relayESPInfo,TargetState) {
// Declare an object result, that will contain some information about the result of the function.
var result = {}
try {
// username and password to address the ESP Webserver
var username = relayESPInfo.username;
var password = relayESPInfo.password;// Build URL to trigger corresponding to expected action
 var url= relayESPInfo.url + “/” + TargetState
// Shoot the request using axios, and await for the result.
 res = await axios.get(url,{auth: {username: username,password:  password}})
 result.success=true
 result.value=res
 return result
} catch (error) {
 result.success=false
 result.error=error
 return result
}}

You can now test ESPClient.js script by adding these lines at the end of your script.

relayESPInfo = {
“url”:”http://192.168.1.177/relay",
“username”:”myESPServerLogin”,
“password”:”myESPServerPassword",
“type”:”relay”
}
// Call the relay control function
ControlRelay(relayESPInfoGarage,1)

Running the script under a shell terminal with:

# node ESPClient.js

should turn on the relay. To build the Restful API of our automation server, we need first to take a look at dialogflow to understand the request it will send to our server.

Building our Dialogflow Agent

Dialogflow is the google IA developed to power voice bots. It basically extracts required information from human inputs to convert them to a machine understandable format. Of course, you have to define the scope of its understanding. It can only understand what it was trained for. You have to define an Agent. Let’s build our very basic agent to fit our need. If you want to develop a more complexe agent, you can take a look at my article about building a ‘complex’ chat bot.

Connect to https://console.dialogflow.com/api-client/ and connect using your google account. Note that only devices using this account will be able to pilot the device at the end of this tutorial (To get a step further, you will have to release your agent for Alpha / Beta test, allowing other users to use it).

Create a new project, selecting your project name & langage:

The interface:

Let’s take a look at the the left side panel, where you can see ‘intents’, ‘entities’, and ‘fulfillment’.

In our scenario, the intent is to ‘control the garage door’, more precisely to

open the garage door | close the garage door

So we will create the intent ‘control_door’. We will then go to ‘training phrases’. Here, we will enter a set of phrases (the more the better) based on which the IA will learn to understand the intent.

In these phrases, there are 2 important values: Open or Close = the action, and garage door = the target of the action.

These are called slots, and we have to define them and provide their possible values. Note that plenty of predefined types of slots are supported by default by Dialogflow, such as time, dateTime, duration etc… In our use case, we will create two new types of entities (= slots), one ‘action’ and one ‘location’.

Select ‘close’ word from left to right, and click ‘create new’. This will bring you to ‘entities’ tab.

Create your entity named action, covering the two actions you can expect from a door: open & close. You can add synonyms or disable them.

Now, go back to your control_door intent, and assign this type to your verbs. The action:action should now appear in the scroll-down menu.

Create the ‘location’ entity and assign it as well to relevant words in the training phrases.

You should end up with intent phrases looking like this:

In ‘action and parameters’, flag both entities as ‘required’. In ‘Responses’, set “you want to $action the $location”

At this step, let’s check the scenario is working. Using the simulator in the upper right corner, provide your intent: ‘shut the garage’. The IA will detect ‘shut’ and match it to entity action ‘close’, and will detect ‘garage’ and match it to entity location ‘garage door’ (because we defined garage as a synonyme of garage door). Using the response pattern you provided, the answer must be “you want to close the garage door”.

A successful call to our intent in the simulator

So far, the answer is provided by dialogflow itself. There is no communication from dialogflow to our automation server. To create this interaction, we must define a webhook, where dialogflow will send the parameters collected via an intent. This webhook is our automation server on the raspberry Pi.

Note about security: For some security reasons, Dialogflow will fail to call your API if:

  • It does not have an FQDN (an URL, not an IP)
  • It is not presenting a valid SSL certificate (Implying your webhook must be exposed over HTTPS). Google has taken security in hand, and what can seem painful is actually a Must (Banning HTTP website is part of the strategy).
  • You server port is not reachable from the outside (= the internet)

We will need to clear all these points to correctly expose our web server API. This will be covered in next chapter.

Prerequisites to exposing our HTTPS Server over the net

Obtaining a public URL for your server

When you send a letter to someone (do you still? ;) ), you provide the address and the name. Your postal service maps this information to a geographical location (receiver home) so they can actually deliver the message.

Well, on the internet, the postal service is a DNS service mapping URLs (user friendly addresses) to a IP (=your ‘home’ on the internet).

I used https://www.noip.com for this tutorial because it is free for basic needs.

a/ Create an account and create your no-IP hostname: For the tutorial, I took https://mydomotics.ddns.net/

Note: Your Internet address is automatically detected and the mapping is done once you create the hostname.

b/ If your IP is not static (some ISP provide dynamics IPs, so your IP changes from time to time), then your server needs to send it’s new IP to the DNS Service for update. To do so, you need to install a service on your OS.

For a Raspberry, get the DUC tool tar.gz and follow the instructions.

Once the make install step is done, the auto configuration client starts, provide requested information:

Auto configuration for Linux client of no-ip.com.
Please enter the login/email string for no-ip.com xxx@hotmail.com
Please enter the password for user ‘xxx@hotmail.com’ ********Only one host [mydomotics.ddns.net] is registered to this account.It will be used.
Please enter an update interval:[30]
Do you wish to run something at successful update?[N] (y/N) N

Your are done!

Redirecting the ports on your Box

For your server to be reachable from the internet, you need to redirect request received overs port 443 (HTTPS default port) on your ISP Box to the raspberry Pi over the port of your choice (8081 in my example bellow). This port is the one on which your automation server will listen.

To get the SSL certificate (next step), you need to redirect port 80 to your raspberry Pi on same port.

Depending on your ISP, you will find convenient tutorial on the net to this port forwarding.

Obtaining an SSL public certificate

Not long ago, having a public certificate for private use was costly. Fortunately for all the fab lovers, it is now possible to get a free public certificate using Let’s Encrypt. The certificate lasts 3 months, but can be automatically renewed using a python script in the Raspberry cron.

For the raspberry,Installing Certbot to get a Let’s Encrypt Certificate is the easiest way to go.

https://certbot.eff.org

Select ‘None of the above’ as software, and Debian 8 as System, and run the instructions from a terminal on the Pi:

pi@raspberrypi:/usr/local/src $ sudo wget https://dl.eff.org/certbot-autopi@raspberrypi:/usr/local/src $ sudo chmod a+x certbot-autopi@raspberrypi:/usr/local/src $ sudo ./certbot-auto certonly

Note: This last step will take some time on a Pi. If everything goes well, you should be asked some questions. Adapt the answers to your environment.

Installation succeeded.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator standalone, Installer None
Enter email address (used for urgent renewal and security notices) (Enter 'c' to cancel): xxx@hotmail.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Please read the Terms of Service at https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must agree in order to register with the ACME server at https://acme-v02.api.letsencrypt.org/directory 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(A)gree/(C)ancel: A
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing to share your email address with the Electronic Frontier
Foundation, a founding partner of the Let's Encrypt project and the non-profit organization that develops Certbot? We'd like to send you email about our work encrypting the web, EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: Y
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c' to cancel): mydomotics.ddns.netObtaining a new certificate
Performing the following challenges:
http-01 challenge for mydomotics.ddns.net
Waiting for verification...
Cleaning up challengesIMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/mydomotics.ddns.net/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/mydomotics.ddns.net/privkey.pem
Your cert will expire on 2019-02-15. To obtain a new or tweaked

Finally, move the certificate and key to the place of your choice, you will need to provide the path later. Also you may need later to set access rights to the cert & key for the Pi user (if you get this error later on when starting the server: Error: EACCES: permission denied, open ‘path_to_key/privkey.pem’)

Note: if you are the lucky owner of a Synology NAS, DSM software allow to get very easily (tutorials online):

  • dynamic DNS
  • lets Encrypt certificate
  • reverse reverse proxy to forward the requests received on the NAS to the Rasberry over HTTP (no certificate on the Pi needed anymore in that case, since the certificate is hold by the NAS).

Now that we are able to expose our Raspberry server over the internet with a public SSL certificate, lets finish the dialogflow Agent configuration.

Connecting our Dialogflow Agent with our automation Server (webhook)

Go back to the Dialogflow console of the project. On the left panel, select fulfillment, enable the webhook, and enter your public URL, a login and password used to authenticate on your automation server, than at the bottom, click save.

Note about security:

Dialogflow is currently not able to do some Oauth2 authentication with the webhook (Oauth2 is the standard nowadays for client to server). If you consider that Basic Auth over HTTPS is not enough, note that dialogflow allows to add some headers in the call, and you could implement some control on the additional headers on server side.

Now that the Dialogflow webhook is declared, you must associate it to some intent fulfillment. Open the ‘control_door’ intent, and at the bottom of the page, select ‘enable fulfillment’,

and “Enable webhook call for this intent”:

Now, once the required parameters of this intent will be collected by dialogflow, they will be sent over to the webhook in a POST request. The webhook will analyse the parameters, build adapted answer, and forward it to dialogflow that will relay it to user. You can also redirect the Default welcome intent to your webhook for it to provide the answer.

Customizing our Agent invocation

Finally, lets give a name to our agent. Note that this step is not mandatory. By default, with google Assistant, you will start your agent saying / writing:

“Talk to my test app”

However, if you can modify this invocation with the one of your choice

“Talk to my easy automation”

Under ‘Integrations’, click the Google Assistant / Integration Settings link.

On next screen, click ‘Manage Assistant APP’ in the lower right corner.

Click on ‘decide how your Action is invoked’.

Provide an invocation name for your agent and save.

Building our ‘automation’ service on the Raspberry: The RESTful API part

Using NodeJS again, we are going do use the Express Module dedicated to handling HTTP(s) requests, the reference to expose a REST API on NodeJS. Lets start with a basic server code, the hello World. Just to be sure when can reach it from the internet. We will call the script ‘server.js’

// Import https allowing to run express of HTTPS
var https = require('https');// Import the certificate and the private key to be used (provide the path were you stored previously generated cert & key)
var fs = require('fs');
var sslOptions = {
 key: fs.readFileSync('ssl/privkey.pem'),
 cert: fs.readFileSync('ssl/fullchain.pem')
};// Import express module, and create the server
var express = require("express")
expressApp = express()// Declare the listening port for the server
const httpsPort = 8081;// Finally, start the server
https.createServer(sslOptions, expressApp).listen(httpsPort );// And tel express how to handle GET requests over “/”, here just answering “Hello World!” and logging request.
expressApp.get('/', function (req, res) {
 console.log('Request received over /')
 res.send('Hello World!')
})

From your browser, hit the URL recorded in the Dynamic DNS service, eg your server FQDN, in my case https://mydomotics.ddns.net

You should get a “Hello World!” in the browser. And the certificate is trusted and therefore page is secured!

Now that we confirmed the server is exposed over HTTPS, on a public URL and with a public certificate, we have all the conditions for Dialogflow to successfully hit it.

let’s modify a bit previous code.

Note about security: So far, we obtained an SSL Server certificate that allow client to authenticate the server, and than to cipher client-server communications. We can add some basic auth mechanism on request. Bellow code includes it.

// Import bodyParser allowing to handle JSON objects, and basic Auth to check user authentication on HTTP Requests
var bodyParser = require('body-parser')
var basicAuth = require('express-basic-auth')// Configure the basic Auth parameters, login and pass required to address the HTTP Server. These are the one you declared in the dialogflow 'webhook' section.
const basic_auth_config = { users: { 'myDialogFlowUser': 'password’ }}

Let’s now write the DFapp that actually handles the requests received by Dialogflow and build the answers.

const {SignIn,dialogflow,Image,SimpleResponse} = require('actions-on-google');
const DFapp = dialogflow();
const languageCode = 'English — en';

As we saw previously, Dialogflow will collect some intents and forward them to the webhook for fulfillment. In the webhook, we need to analyse the received intents to build appropriate answer. For this purpose, we will use DFapp.intent constructor. This constructor allows us to retrieve the intent ‘slots’, the actual information extracted by Dialogflow. The default Welcome Intent is an ‘automatic intent’, it will be triggered when user invokes the agent from within google Assistant.

Building the answer is done using conv.ask, either passing a ‘text’ or a specific function.

// DEFAULT WELCOME...
DFapp.intent('Default Welcome Intent', (conv) => {
conv.ask("Hello, I am your home automation Bot. What can I do for you?");
})

At this step, in our minimalist scenario, the user will ask to either open or close the garage door. The intent ‘control_door’ will hit the webhook for fulfillment.

// control_door INTENT 
DFapp.intent('control_door', async (conv) => {   console.log("###########################################################")
console.log("Current INTENT: " + JSON.stringify(conv.intent, null, 4))
console.log("Collected conversation parameters: " + JSON.stringify(conv.parameters, null, 4))

if (!conv.parameters.action ){
 conv.close("Sorry, an issue occured")
}
if (conv.parameters.action=="open"){
 let actionResult = await ControlRelay(relayESPInfo,"on")
 if (actionResult.value){
  console.log(JSON.stringify(actionResult.value.data,null,4))
  conv.ask("Ok, I opened the door!")
 }
 else
 {
  conv.close("Sorry, an issue occured")
 }
}
if (conv.parameters.action=="close"){
 let actionResult = await ControlRelay(relayESPInfo,"off")
 if (actionResult.value){
  console.log(JSON.stringify(actionResult.value.data,null,4))
  conv.ask("Ok, I closed the door!")
 }
 else
 {
  conv.close("Sorry, an issue occured")
 } 
}
})

Note the console.log(JSON.stringify(conv.parameters, null, 4)). This is where the magic of Dialogflow relies. It contains all the ‘entities’, e.g action and location in our case, collected from the user’s input:

{
    "action": "open",
    "location": "garage door"
}

Also note the async on top of the function, meaning we will be able to call function, await for the result to return relevant answer with conv.ask. In our use case, the async function ControlRelay(relayESPInfo,targetState) is called with 2 arguments: Information about the relay to drive, and the state to set.

// Declare the relayESPInfo object (use the info from your ESP Code)
relayESPInfo = {
"url":"http://192.168.1.177",
"username":"myESPServerLogin",
"password":"myESPServerPassword",
"type":"relay"
}// Import axios used in next function
const axios = require('axios')// Create an async function that takes as input required information to communicate with ESP and the order to send ( = target relay state)
async function ControlRelay(relayESPInfo,targetState) {
// Declare an object result, that will contain some information about the result of the function.
let result = {}
try {
// username and password to address the ESP Webserver
let username = relayESPInfo.username;
let password = relayESPInfo.password;
// Build URL to trigger corresponding to expected action
 let url= relayESPInfo.url + "/relay/" + targetState
// Shoot the request using axios, and await for the result.
 res = await axios.get(url,{auth: {username: username,password:  password}})
 result.value=res
 return result
} catch (error) {
 result.error=error
 return result
}}

For some debug purpose (it will help to check that you actually receive the HTTP POST from Dialogflow), write a function that will intercept the request, do whatever you want on it, (here just print the POST Body content) and then forward the call to next consumer with the ‘next’.

function myDebugFunction(req,res,next){
 console.log(JSON.stringify(req.body,null,4));
 next()
}

Finally, declare expressApp and the middleware it will use, and start the server:

const expressApp = express()
.use(basicAuth(basic_auth_config))
.use(bodyParser.json(), myDebugFunction)
.use(DFapp)// Declare the listening port for the server
const httpsPort = 8081;// Finally, start the server
https.createServer(sslOptions, expressApp).listen(httpsPort );

What happens here is that if SSL authentication is successful, incoming messages are going through the expressApp. Messages are routed through basicAuth function first, if authorization is granted, they are passed to the Json parser, to our debug function, and finally to DFapp.

Full Code should look somehow like this:

// Import https allowing to run express of HTTPS
const https = require('https');
// Import the certificate and the private key to be used (provide the path were you stored previously generated cert & key)
var fs = require('fs');
var sslOptions = {
 key: fs.readFileSync('ssl/privkey.pem'),
 cert: fs.readFileSync('ssl/fullchain.pem')
};// Import bodyParser allowing to handle JSON objects, and basic Auth to check user authentication on HTTP Requests
var bodyParser = require('body-parser')
var basicAuth = require('express-basic-auth')// Configure the basic Auth parameters, login and pass required to address the HTTP Server. These are the one you declared in the dialogflow 'webhook' section.
var basic_auth_config = { users: { 'myDialogFlowUser': 'password' }}// Require actions on google module 
const {SignIn,dialogflow,Image,SimpleResponse} = require('actions-on-google');//Import the middleware in charge of dialogflow intents
const DFapp = dialogflow();
const languageCode = 'English — en';// DEFAULT WELCOME INTENT...
DFapp.intent('Default Welcome Intent', (conv) => {
conv.ask("Hello, I am your home automation Bot. What can I do for you?");
})// control_door INTENT 
DFapp.intent('control_door', async (conv) => {
    console.log("###########################################################")
    console.log("Current INTENT: " + JSON.stringify(conv.intent, null, 4))
    console.log("Collected conversation parameters: " + JSON.stringify(conv.parameters, null, 4))
 
 if (!conv.parameters.action ){conv.close("Sorry, an issue occured")}
 
 if (conv.parameters.action=="open"){
  let actionResult = await ControlRelay(relayESPInfo,"on")
  
  if (actionResult.value){
   console.log(JSON.stringify(actionResult.value.data,null,4))
   conv.ask("Ok, I opened the door!")
  }
  else
  {
   conv.close("Sorry, an issue occured")
  }
 }
 
 if (conv.parameters.action=="close"){
  let actionResult = await ControlRelay(relayESPInfo,"off")
  
  if (actionResult.value){
   console.log(JSON.stringify(actionResult.value.data,null,4))
   conv.ask("Ok, I closed the door!")
  }
  else
  {
   conv.close("Sorry, an issue occured")
  } 
 }})function myDebugFunction(req,res,next){
 console.log(JSON.stringify(req.body,null,4));
 next()
}// Import the axios module
axios = require('axios')// Declare ESP info
relayESPInfo = {
"url":"http://192.168.1.177",
"username":"myESPServerLogin",
"password":"myESPServerPassword",
"type":"relay"
}// Create an async function that takes as input required information to communicate with ESP and the target relay state)
async function ControlRelay(relayESPInfo,TargetState) {
// Declare an object result, that will contain some information about the result of the function.
let result = {}
try {
// username and password to address the ESP Webserver
let username = relayESPInfo.username;
let password = relayESPInfo.password;
// Build URL to trigger corresponding to expected action
 let url= relayESPInfo.url + "/relay/" + TargetState
// Shoot the request using axios, and await for the result.
 res = await axios.get(url,{auth: {username: username,password:  password}})
 result.value=res
 return result
} catch (error) {
 result.error=error
 return result
}}// Declare the express server
const express = require("express")
const expressApp = express()
.use(basicAuth(basic_auth_config))
.use(bodyParser.json(), myDebugFunction)
.use(DFapp)
// Declare the listening port for the server
const httpsPort = 8081;// Finally, start the server
https.createServer(sslOptions, expressApp).listen(httpsPort );

We are ready to start our server, using ‘node server.js’.

Finally, the End to End testing!

If you reached this point, congrats! We are ready to test live! Take your phone out of your pocket, fire up google assistant and invoke your agent! (you can also use the assistant simulator that will also provide some debug:

In Dialogflow upper right corner, launch the Google Assistant interface.

Engage conversation either by clicking / typing / speaking:

Enjoy opening and closing the relay (and therefore what ever you plugged on it!)

Debugging and next steps

About debugging:

I struggled with also the steps to actually have a first successful webhook call. Here is what I strongly recommend to debug:

  • Implement the ‘myDebugFunction’ express middleware and customize it to your need. Seeing what Dialogflow POSTs is very insightful.
  • Use the google Assistant Simulator until everything is 100% functional
  • Use its DEBUG and ERROR tabs (upper right on last screenshot)

some hints depending on the symptoms:

POST requests dont hit my webhook.

  • The call didn't reach your webhook: Check in your browser that you can get a reply for your URL and that the certificate is valid
  • The call didn't pass the basic auth. You can temporarily disengage the basic auth by removing .use(basicAuth(basic_auth_config))

POST requests hit my webhook but the simulator says: ‘ ‘easy automation’ left the conversation’

  • Check in your raspberry console the logs, and the simulator’s logs.
  • In case you still dont have any clue, implement more logs in the NodeJS code.

About next steps

Now that we discovered the basics, we could, reusing this same scenario, see other possibilities of Dialogflow, such as:

  • user Sign In to add an additional layer of security and a personalized user experience
  • Providing a rich user experience with the rich messages / buttons / cards etc… in google Assistant

we could build a much more complex IOT environnement for home automation, and develop the dialogflow agent to drive all these little things (And not rely on those commercial home automation servers cloud based, with frequent outages, that are doorways to hackers… )

We could unplug Google Assistant / Dialogflow and replace these with a classic Web App / Web Client, reusing the automation server REST API.

Or we could use Dialogflow and our webhook to do some totally different tasks, such as building a booking system (Rooms, cars, events… , check out my article about chatbots).

It’s up to you to imagine your best use case!

Conclusion:

The objective of this tutorial what to build and control some IOT device via the google assistant.

What we did:

  • we built our (insanely cheap!) IOT device
  • we programmed our IOT device using Arduino IDE and exposed it’s restful server
  • we built our home automation server on Raspberry using NodeJS, with 2 interfaces: One client to pilot the IOT device, one server to listen for external requests.
  • we built our Dialogflow agent to convert human requests into machine requests, so that our automation server could fulfill the request.
  • On the way, we learnt how to expose a server over the internet with its own FQDN, a public SSL certificate and how to secure its access with basic authentication.

I hope you enjoyed the journey as much as I did and that it gave you plenty of ideas for the next rainy week end.

Stay tuned for the next episode!

Nodejs
Dialogflow
Home Automation
Esp8266
Google Assistant

--

--

Aurélien Esprit

Written by Aurélien Esprit

10 Followers

Tech Enthusiast, innovator, enterprise solution designer

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams