How to create a custom private Google Home Action with API.AI and Google App Engine

Google Home Assistant [photo: thegadgetflow.com]

Google Home is a voice-activated speaker powered by the Google Assistant. It has already built-in actions but it is also possible to add customized actions.

This tutorial details step by step how to build an Assistant Action and how to keep it private (ie. you can use the Action in your own Google Home device, but it will be not publicly available).

The basic idea is to create an Application (webhook) running in a Google App Engine server that is triggered by an Action built with API.AI.

I assume that you know how to use a shell and how to write an application in one of the programming languages and environments supported by the Google App Engine (in this tutorial I started from a Node.js template). Also the links in this tutorial refer to documentation available for any platform but I only tested in my macOS Sierra.

STEP 1. Create API.AI and Google Cloud accounts. You must create the accounts with the same Google email that you use to control your Google Home.

• Create an account on API.AI;
• Create an account on Google Cloud;

STEP 2. Create an Assistant Action on API.AI. API.AI is a development tool for conversational end points. It allows to create a natural language interactions action for Google Home.

• Follow the official Google video to learn how to create an action: Actions on Google: Building Assistant Actions using API.AI .
  Achtung!! If you wish to keep your Assistant Action private stop the video at minute 13:50 and don’t click “DEPLOY” on API.AI console.
• Make note of the project ID that you have created in Google Cloud.

STEP 3. Code the behavior of the Assistant when triggered by an Intent. For example, what should the Assistant do when you ask “Do you have a recipe for a quick hot soup?”. API.AI recognize the words ‘quick’, ‘hot’ and ‘soup’ but we need an application that process those words and give us a recipe for a quick hot soup. This is called a webhook.
You can create a webhook in Node.js starting from a template.

• Create a folder in your workspace, I will refer to this folder as <my_project_dir>.
• Create a folder inside <my_project_dir>, I will refer to this folder as <template_dir>.
• Download the template zip from github: apiai-webhook-template-nodejs , unzip and move the files into the <template_dir>.
• Download and install Node.js
• Install the node dependencies in the local node_modules folder, open a terminal and run:

$ cd <template_dir>
$ npm install

• Open the ‘app.js’ file inside <template_dir> with your text editor or your favorite javascript IDE and code your webhook logic inside the function:

function responseHandler (assistant) {
// Complete your fulfillment logic and send a response
}

Note: When an Intent (described in your API.AI action) is triggered, a Json post is sent to the server where the application is listening. The Json contains the Entities that triggered your Intent as parameters and they can be retrieved by parsing the ‘req’ parameter in the app.post method.
To get an example of the Json sent to the server, go to API.AI, test your Intent in the right panel “Try it now” and then click the “SHOW JSON” button.

Example for STEP 3.

API.AI action (left) and the relative webhook (right)

The screenshot above shows an example of a API.API Action (left side) and a webhook (right side).
The Action in API.AI contains the Entity ‘command’ with ‘bed’, ‘tv’, ‘goodbye’ and ‘christmas’ as keywords (with relative synonyms).
Eg. if an Intent is triggered saying “It’s tv time!”, the Action recognizes the Entity ‘command’ with value ‘tv’ and sends a POST request to the webhook with a Json containing the parameter “command”: “tv”.

Then the webhook parses the Json in this way (line 35 right side):

var command = req.body.result.parameters.command;

and based on the value of the variable command, it sends a POST request.
In the ‘tv’ example, it sends a POST request like this:

request.post( “http://website.com/light_tv_on");

To know more about my light control project, go to my GitHub page: https://github.com/sarettasarda/project-smars

STEP 4. Test the Node.js application locally. Be sure that your application works as expected before deploying it to the server. Because the deployment process takes a few minutes, it’s quicker to debug the application when it runs locally.

• To launch your application, open a terminal and run:

$ cd <template_dir>
$ npm start

• If your application doesn’t have any bugs (and you didn’t modify the starting part of the server in the ‘app.js’ at the bottom of the file), you will see on the terminal: “App listening on port 8080” .
• To just simply trigger your application, open a second terminal and try to send a Json with curl to your running local webhook:

$ curl -X POST -H “Content-Type: application/json” http://localhost:8080

• If you want to test with the Json similar to the one sent from API.AI, you can test it sending the JSON body with curl:

$ curl -X POST -H “Content-Type: application/json” -d ‘[json_here]’ http://localhost:8080

[json_here] is the Json that you are expecting to receive from the API.AI action. As said in the previous step, you can get an example of the Json in API.AI project trying the Intent (on the right panel) and clicking the “SHOW JSON” button.

STEP 5. Deploy the project in the Google App Engine. If your webhook is running as expected you are now ready to upload it to the server. I will refer to <project_id> as the ID of the project that you created in STEP 2 while following the ‘Actions on Google: Building Assistant Actions using API.AI’ video tutorial.

• Download, install, and initialize the Google Cloud SDK in your machine following the docs or for Mac users running from command line:

$ curl https://sdk.cloud.google.com | bash
$ source ~/.bash_profile

• To deploy the application in the Google App Engine, run:

$ cd <template_dir>
$ gcloud config set project <project_id>
$ gcloud app deploy

• You should see a log output similar to the following example. Make note of the endpoint URL. You’ll need this in the next section to configure the webhook URL.

Updating service [default]…done.
Deployed service [default] to https://[project-id].appspot.com

You can read logs from the command line by running:
$ gcloud app logs read

To view your application in the web browser run:
$ gcloud app browse

• Your application is now live at ‘https://[project-id].appspot-preview.com’.
  The ‘-preview’ part of the URL is a redirection made by Google to preview your application, and it will be changed in ‘https://[project-id].appspot.com’ as soon as your application will be reviewed and accepted by the Google team if you decide to make it publicly available.
  In this tutorial we are implementing a private Action, so we will not publish the Action and we will only work with the ‘https://[project-id].appspot-preview.com’ address.

STEP 6. Connect your API.AI action to your webhook.

• Open the API.AI project.
• Open ‘Fulfillment’ window on the right panel.
• Copy the URL of your application ‘https://[project-id].appspot-preview.com’ to the URL field in API.AI.
• Save the Fulfillment.
• In any relevant Intent, enable the Fulfillment for the response: open ‘Intents’ window, open the intent that you want to connect, scroll down to ‘Fulfillment’ section, check “Use webhook”.
• Make sure all domains in the ‘Domains’ window are turned off.

Copy the URL of your application ‘https://[project-id].appspot-preview.com’ to the URL field in API.AI.

STEP 7. Test the Assistant Action.

• On API.AI open ‘Integrations’ window on the right panel.
• Enable ‘Action on Google’ and open the Settings
• Click “AUTHORIZE” button on the bottom right corner, then click “PREVIEW” button.
  Achtung!! Don’t click “DEPLOY” if you wish to keep your Assistant Action private!
• You can test the Action talking directly to your Google Home “Ok Google, talk to <your_action_name>” or in the API.AI project in the right panel “Try it now”.

Actions on Google Settings for my ‘smeagol’ project.

STEP 8. Set an infinite preview*. The action preview will last only 30 minutes. You can choose to deploy and public your Assistant Action (only if Big G likes your work) clicking “DEPLOY” button OR keep your action private by setting it as an infinite preview.
To keep your action private:

• Download Google gactions cli (gaction is a self-updating command line interface that lets you test your actions).
  Note: on Mac and Linux run “$chmod +x gactions” in a terminal to make the binary executable.
• Copy the executable inside <my_project_dir>.
• Initialize gaction, open a terminal and run:

$ cd <my_project_dir>
$ ./gaction init

• gaction creates a file named ‘action.json’ in <my_project_dir>, move that json file inside the <template_dir>.
• Open a browser developer console (for Chrome: click the three dots in the upper right corner -> More tools -> Developer Tools ).
• Click the ‘Network’ tab.
• Open your API.AI project and open the ‘Integrations’ window.
• Open “Actions on Google” settings (the Action on Google integration should be already on).
• Click “AUTHORIZE” button.
• Click “PREVIEW” button.
• In the browser developer console inspect the XHR response from:

https://console.api.ai/api/agent/googleassistant/preview

• Copy the content of the response and paste it into the file ‘action.json’ inside the <template_dir>.
• Save the file.
• To set your preview to last forever, open a terminal and run:

$ cd <my_project_dir>
$ ./gactions preview -action_package <template_dir>/action.json -preview_mins 999999999 -invocation_name [invocation_name_here]

This command will send the Action (‘action.json’ file, same as the the API.AI project) to your Google Home and will run it as a preview for 999999999 minutes (about 2000 years).

*thanks to Brady at ‘Make google actions development project preview persist longer’ for the infinite preview trick and video!

Thank you for following the tutorial!