How to implement local fulfillment for Google Assistant actions using Dialogflow

NB: this blog covers Actions on Google node.js deployments only, and presumes some basic prior knowledge of Actions on Google/ Dialogflow

Google Home Devices that run the Google Assistant

Hello, world!

Problem statement

I’ve been getting into Actions on Google quite a bit of late — enjoying myself thoroughly — with silly voice apps. My learning has been primarily designing small “party tricks” that I can amuse visitors with. I’ve been doing the same with Alexa too, but since I’m a lot more familiar with the Google Cloud Platform and Firebase especially, I have prototyped more on Google.

Actions and working with the Google Assistant often requires server side logic to handle the interaction with the desired level of personalisation or customization. This is called “fulfillment”.

The development cycle for fulfillment can be a bit tedious because you need to push your server code to the cloud each time, to see if it’s working. Every time we make changes that we want to test, we need to push code to Firebase and give it 30–60 seconds to propagate. It’s like web dev before hot reloads — ugh! So 2011.

Summary of the Architecture

I won’t go into the details of building Actions for the Assistant, as there is plenty out there to help you with that. But a high level overview of the system now will help you understand the problem statement that inspired this blog.

Here is a diagram from Google that explains how Actions/ Assistant interactions are processed.

A user’s interaction gets picked up by the device, converted to text that is parsed by the device, and passed to a DialogFlow agent which is a NLU engine. That then works out the “intent” of the user’s statement (“phrase” or “utterance”). That intent then gets matched to code that “fulfills” the intent of the user, then returns a response that is (hopefully) appropriate and acceptable.

If you have studied the picture, you will see that a request is made in the last column, which denotes Firebase. What it really denotes is a back-end server that has a HTTP webhook that receives ‘requests’ for fulfillment and then works out how to respond appropriately. Architecturally, the Dialogflow NLU agent sends a POST request to the server webhook, which then gets processed by the server using SDKs.

The back end server that processes requests and responses is easily hosted on Firebase Cloud Functions (of course, Google makes it easy to use their stack — that’s just good business!).

Also, a locally run server will generate a webhook endpoint like http://localhost:3000, which is not accessible to Google Assistant’s POST requests.

Solution Statement — local fulfillment dev server!

There are three steps to this solution:

1. Create the back-end fulfillment server as an Express server.
2. Handle Dialogflow requests and responses as an Express endpoint. The default Firebase Cloud Functions server users the Actions on Google Nodejs Client Library which has built-in functionality to receive HTTP POST requests from the Dialogflow agent. But that won’t work outside the Firebase environment (i.e. on our local development server). So we need to deploy our fulfillment server an an Express app.
3. use the Ngrok Node package to create a temporary HTTP endpoint that tunnels HTTP requests through to our local:3000 (or whatever port you use) server.

I will not go into the steps of setting up your basic code for a simple nodejs fulfillment server — the Dialogflow/ Actions on Google docs help you with all that. But I do provide here the snippets that show you how to convert that Dialogflow App into an Express App, and what your package.json needs to look like.

Let’s start with the basic setup for the Dialogflow node.js client. We will call it the Dialogflow App.

This is the app that ‘handles’ the intents. The above code is just scaffolding. Export the app, and import it in functions/index.js.

index.js is the entry point in our functions folder, which contains the cloud functions we push up to Firebase Cloud Functions. In this file we create the Express App, and import the DialogflowApp object, and then pass it in to the Express route that will receive the HTTP POST requests from Dialogflow. Note that we need the body-parser npm package as the HTTP requests are JSON.

The key parts of this code are that we create a POST route that takes, as the handler-callback, our DialogflowApp object. I created a GET route just to make quick browser GET requests to test the endpoint is working. But Dialogflow only uses the POST route.

Note that I’ve made two exports here. One is the Express app and the other is the dialogflow App itself. This creates to Firebase functions with two endpoints which are identified by the property attached to the exports object. One endpoint will be <……/fulfillmentExpressServer> and the other will be <……/dialogflowFirebaseFulfillment>.

I can use either of these HTTP endpoints for fulfillment, once I’m done developing locally and have pushed the final code up to Firebase Cloud Functions.

NGROK for the local development server tunneling

There is some funny looking code in there on line 26. On my Mac terminal, I use export IS_LOCAL_DEV=true before I start the server locally. That codeblock on line 26 basically starts the server listening locally, which is not needed when we push the code up to Cloud Functions — it is for the local server only.

Which brings me to the above configuration in package.json. In the scripts property, you can see one called tunnel. Note the port number 8000 there. That can be set to anything you like, but make sure that the code in index.js that sets the PORT constant (line 27 in the Gist) is the same port number.

Before you run the code, check the following:

1. You’ve set your environment variable and the code matches it — in my case I used IS_LOCAL_DEV=true
2. Your port numbers match in index.js and the tunnel script

Then you open two terminal windows and run the following:

npm run dev and npm run tunnel

In the terminal that ran the tunnel (ngrok) you will see something like this:

Note the two ‘Forwarding’ addresses. They’re identical, except one is https. Either works. That is your endpoint, assuming your POST route is / and not /<some path>. If you’ve added a path to the POST handler then you need to add that path to the Forwarding address.

Note that every time you run ngrok it generates a new, temporary URL for you. Therefore, you need to update your Dialogflow fulfillment webhook every time you run npm run tunnel.

And there you go. No more pushing every little change in code to Firebase Cloud Functions and waiting a minute or two before testing. Develop on the fly, and with nodemon, your server restarts and you can keep testing as you code!

I can be contacted on Twitter: @ZubinPratap

And if you are a code newbie, and find this overwhelming, that is natural. You’re jumping too far ahead. I made that mistake too, and it cost me too much.

If you want to learn to code, or are overwhelmed by it, I encourage you to take a look at my course on Udemy. It teaches you how to learn to code rather than how to code. Coding is easy, but learning to code is not. If you want a promo code go here.