Hands-on with Dialogflow & Google Assistant: Writing your first Conversation Agent

Image Reference : https://developers.google.com/actions/images/surfaces-hero.svg

Last Update : March 10, 2018 :

— Use Dialogflow across the article.
— Updated screenshots (if needed)

All Source code for the material is available here.

Feel free to use this material as part of your workshop. Let a thousand actions bloom!

You : “Ok Google, how do we write our first action for Google Assistant?”
Me : Let’s start

Prerequisite : Sign up for Google Cloud Platform (GCP)

In case you have an existing GCP Account, you are good to go. If not, you should sign up for the same. GCP gives you a 2-month and $200 credit to get started and that is sufficient to learn tons about the platform. In fact, there is a Free-tier on GCP that in my opinion is the most generous that you can come across. Sign up here.

Overview and Purpose

This hands-on lesson provides a step-by-step guide to creating a Feedback Agent using natural conversation. We shall use Dialogflow platform to write this Agent. We will learn how to use Google Actions Console to test and validate our Agents on multiple target surfaces like Google Home and a Phone. You can sign-in to Dialogflow with your Google account that you used to sign up for GCP.

Sample Conversation

Before we begin with Dialogflow, we can take a look at a sample conversation. It is important to model this, so that we have a reference conversation that we can start to mode.

The conversational flow that we shall come up with is given below, with a sample user, User1 and the Feedback Agent:

User initiates the conversation …

Feedback Agent: Hi! I am the Feedback Agent. Thank you for taking the time to give us feedback.
User1: I would like to leave some feedback.
Feedback Agent: Which city did you visit our Resort in?
User1: I went to your Mumbai resort.
Feedback Agent: How would you rate your experience on a scale of 1 to 5.
User1: I would rate it at 3.
Feedback Agent: Any specific comments or feedback, you would like to give us?
User1: The food was excellent. We found the entertainment activities to be a bit limited.
Feedback Agent: Thank You for your feedback. We appreciate you taking the time to give us feedback. Have a good day and hope you visit us again.

Note: The words/sentences that are highlighted in bold are the inputs that we are interested in capturing.

Building our Agent with Dialogflow

Follow the steps given below to build out the Feedback Agent.

Step 1 : Create the Agent

Sign in with your Dialogflow account. If this is your first time using Dialogflow, you will have no Agents configured. In the top-right, from the main menu, click on Create Agent as shown below:

Alternately, instead of the left navigation bar, you will also see in the middle section the following form controls, as shown below. Click on CREATE AGENT button.

This will bring up the Agent configuration screen, where you can define the Agent name, provide a useful description and more. The screen is shown below:

Provide the following information:

1. Name for the Agent i.e. FeedbackAgent (no spaces in the name)
2. In the Google Project section, leave it empty for a new Google Project to be generated for you or if you have signed up for Google Cloud Platform and have an existing project, select a specific Google project from the list shown.
3. Click on CREATE.

You will then be led to the Intents screen but we can skip that for now. We will first define a few entities.

Step 2 : Create the Entities

In this step, we are going to model our entities. If you look at the sample conversation that we defined, we had to capture the rating (on a scale of 1 to 5) and the location of the resort.

Now, for our example, we will consider that the hotel has resorts at the following locations:

• Mumbai
• Delhi
• Kolkata
• Bengaluru
• Chennai
• Hyderabad

So, what we are going to do now is to define our location entity with valid values. And we shall do some interesting things i.e. define synonyms, such that if someone says Bangalore or Bengaluru, it means the same and so on for other cities too.

Let us create the resort-location entity as instructed below.

Go to Entities from the main menu. Click on CREATE ENTITY button. Give a name to the entity resort-location and create the entries as shown below, ensuring that the Define synonyms checkbox is selected.

Click on the SAVE button.

Create another entity called rating as shown below:

Remember to click the SAVE button.

You should now have the following list of entities:

Step 3 : Create the Intents

Click on the Intents option in the main menu. You should see the following Intents by default:

Click on the Default Welcome Intent. By default, this has an event configured for it, named WELCOME. This is automatically activated for multiple integrations like Google Home, etc.

You will see a list of standard Responses at the bottom. The Agent will randomly pick one up to respond here. You can opt to modify this message if you want.

One of the things we can do now is to remove all the standard responses and add the following text message, which we discussed in the sample conversation flow.

“Hi! I am the Feedback Agent. Thank you for taking the time to give us feedback. “

The Responses section should look like this:

Make sure that you SAVE the intent.

You can alternately start the training for your Agent. You can start typing the sentences that you would like this Intent to be kicked in and for which you would like the Agent to respond with the statement that we have configured. Go ahead and Type “Hi” and “Hello” in the Trainings phrases field. The screenshot is shown below:

Remember to click the SAVE button.

You can also visit the Default Fallback Intent, which is executed when there is no statement that the platform is able to match to the existing intents.

The next thing that we are going to do now is to create our main intent i.e. the Report Feedback Intent.

Go ahead and click on CREATE INTENT.

This will bring up the New Intent screen.

Start by giving a name to the Intent i.e. Feedback Intent.

We are now going to add a few sample utterances in the Training phrases field. Type the following sentences, one after the other. After entering each statement, press Enter (Return).

• I would like to leave some feedback
• I think my rating is 5
• I visited the Mumbai location

You will notice that the application was able to extract out the rating and location entities that we created a while earlier and was able to map it to the entity names that we created (@rating and @resort-location) as shown below:

Go to the Action and parameters section as shown below:

Click on MANAGE PARAMETERS AND ACTION and give a name to our action i.e. feedback.action

Reorder the Parameter list as shown below and do the following:

1. Make both the parameters are required by selecting the REQUIRED option.
2. Add a prompt to each of the parameters as given:

The intent page should look like the following at the moment:

We still need to add our comments parameter. Add another parameter by tapping on the “Enter name” row that you see right below the resort-location parameter. Give it the following details:

Click on SAVE.

Finally, go to the Responses section as shown below:

Click on ADD RESPONSE and give the following text in the Response Text field.

“Thank You for visiting our $resort-location resort and giving us a feedback rating of $rating. We appreciate you taking the time to give us feedback. Have a good day and hope you visit us again.”

The screen is shown below:

Note the use of the variables $resort-location and $rating.

Click on SAVE again.

This completes the definition of the Intent.

Step 4 : Test our Agent

It is time now to start testing our Agent. In the Agent test Console on the right, where you see Try It Now field, you can test out the Agent by typing in sample utterances.

Test Case : Welcome Intent

Go ahead and type “Hello” in the Test Console, that you see on the right.

This should bring up a result as shown below:

You can see that it mapped it correctly to the Default Welcome Intent. The Response field shows that it picked up the Response that we had configured for the Agent.

Test Case : Give Feedback

Go ahead and type “I want to give feedback”. The Test Console for the Agent will correctly map it to the Feedback Intent as shown below. Note that it is able to also bring forward the next question to be asked i.e. what rating would the user like to give?.

Continue with the rest of the conversation. I say 5.

Then it asks me for the city that I visited the resort in? I answer with Chennai.

In the final step, I give some comments as shown below:

You can see that it completed the linear set of questions that we asked and summarized it in the response.

Things to note:

1. The final output in the Test console, shows us the Parameters and the values that you gave.
2. Click on the Show JSON link. In case, you integrate it with an external Webhook that needs to be invoked, then Dialogflow will invoke it with the request as shown. Notice the parameters section. It has the request parameter names like comments, rating and resort-location that are correctly filled out depending on the values that we provided in the test conversation.

Optional :

1. Keep training the Agent with additional statements that you would like it to handle in the “User Says” option for the Intent.
2. Visit the Training option in the main menu. Take a look at sample utterances and how it has mapped to the respective Intent. If you enter statements that it does not understand, it will come up as unmatched over here and you can assign it to specific intents if you want.

Step 5 : Publish your Agent (Web Demo)

We can now publish our Agent, so that we can give it out to people to test. This is not the same as publishing the Agent for everyone to access.

To enable the Web Demo, go to the Integrations page. One of the options visible is Web Demo as shown below.

Slide the button to ON. This will show the URL and a place where you can change the icon and description for your Agent.

Click on that URL to visit the Agent in Action (a sample run is shown below).

Step 6 : Google Assistant Integration

In this final step, you can test out your Agent in the Google Assistant. To do that, we will have to go to the Integrations link again. Dialogflow provides integrations with many popular Messaging Platforms and Interfaces. You will notice that “Google Assistant” is available as one option. The Google Assistant as you know is available on Android phones, Google Home devices and many more devices in the future.

To enable the “Google Assistant” integration, click on INTEGRATION SETTINGS link that you see in the screen below:

This will bring up Actions on Google configuration dialog as shown below:

Click on TEST. This will bring up the screen shown below:

Click on CONTINUE.

You will see a new window open up with the following status:

Finally, if all goes well, you will be lead to the Actions Console for your project as shown below:

This will take you to the Actions for Google console, which is similar to the console that you expect when working with Google Cloud Platform. In fact, if you recollect in the first step when creating the agent, we asked that it create a new Google Cloud project. In fact it has done that as you can see in the top right hand corner of the console, as shown below:

And if you click on the Settings icon on the left hand menu, you will see the following:

Now, here is the cool thing about this console. It is a step by step process for you to set things in place for publishing your Agent for wider distribution and availability to everyone, including testing on your devices or the web based simulator.

Click on the Home icon on the left side menu and you should see the following:

You can see that we have completed Step 1 i.e. the Action has been added from the Dialogflow console that you were using to author your agent.

Optional

The next step (click on EDIT) is to add the Agent App information. This is needed if you plan to submit your Agent for approval by Google and for eventual distributed to the world. We will not go through that in this tutorial but keep in mind that it is similar to publishing an Android App in the Google Play Store. You have to give a lot of details like your invocation name, images, clear description on what your agent does, which countries (Location targeting) that your agent should be available in and surface (i.e. should it only be available on Voice enabled devices i.e. Home or even with screen like capabilities i.e. Google Assistant on your Pixel phone). If you indeed go through with this, you can submit this for Review to Google and they will approve / reject your Agent based on how closely you adhere to a strict set of guidelines.

Google Developer Advocate, Ido Green has written a good article titled “How to pass the Google Assistant App Review”, you should read it.

To test our agent on the web based simulator that Google provides, simply click on the TEST ICON that you see on the left hand menu.

It provides you instructions on what you can do i.e. Just follow it by typing or saying “Talk to my test app” and then it kicks in with the rest of the flow. A sample run is shown below:

You can also see the request / response JSON that are flowing for every conversation request / response and this will be very useful to understand what is going on and also for debugging later on, as you build in custom functionality via code.

Some of the advantages of this Testing console is:

1. You can test out multiple versions of the Google Assistant Action. Currently, we just have draft but you can do test out for multiple versions.
2. You can enable / disable testing as needed.
3. You can choose a surface (Android phone which will have a screen, Google Home device, which will have only voice) for testing to make sure that you have the optimum experience for each of these surfaces. This is very important.
4. You can share the current version for testing / experience by clicking on the share link. This will add project team members to your Google Cloud Platform project and provide them viewership rights.

All of the above are available on the top right side of the console as shown below:

WEBHOOK FULFILLMENT

The example that you have seen so far is giving you a canned response that is returned from the Intent definition itself.

In the real world, once Dialogflow is done with its processing and you have collected all the information that you want, in our case, the feedback information i.e. rating, resort location and comments, you would want that data to be sent to your backend application. In other words, it would be great if the Dialogflow System was able to make a HTTP POST to your backend server, which could then record the data and give back the response to the Dialogflow conversation.

We can do that via the Webhook Fulfillment in Dialogflow. What are are going to do is the following:

1. We proceed with the test agent in Dialogflow and collect all our information (rating, resort location and comments).
2. At the last step, we would like that the call be made to our backend API i.e. HTTP POST.
3. Our backend API Server will extract out the parameters (rating, resort location and comments) and print them (You can definitely do more with it, but you get the drift!) and return back a successful response back to Dialogflow.

Our Backend API Server

This hands-on exercise provides you with multiple backend API Server implementations, which you can use for your testing. Depending on your preference, we have the implementations available in the following languages:

• Python
• Node.js (JavaScript)
• Golang

The source code for each of the API Server implementations is available at the companion Github site. Go to the directory webhook implementations and pick your preferred language implementation.

As mentioned earlier, the JSON Request that will be sent to your webhook and specifically the one with the parameters is shown below (partial response):

So we need to extract out the parameters. A sample Node.js app is shown below but you can take the Golang or the Python implementation too:

The code is straightforward and it simply extracts out the 3 request parameters and prints them out.

Note that we have to return back the response as a JSON in a specific format. This is documented in the Dialogflow documentation (https://dialogflow.com/docs/fulfillment#section-format-of-response-from-the-service) and we are following that.

Running our Backend API Server

We will be running the API Server locally and for that we will need that our local server is available publicly to the outside world. We shall use ngrok, which is a local tunneling software.

It works in the following manner:

Image Referenced and modified from https://ngrok.com/static/img/webhooks.png

To start our application and use ngrok to expose a publicly accessible endpoint for our server, we need to do the following:

1. We start up the respective application (Python, Node.js or Golang). You should be familiar with starting up of the application. For e.g. $ go run or $ python or $ node and so on. The code runs the Server on port 5000.

2. We start ngrok to expose a secure public tunnel on port 5000 via the following command:

$ ngrok http 5000

This will start the ngrok utility and you will see an output similar to the following. It shows the http/https endpoints that it has exposed and where it tunnels the traffic to i.e. localhost running on port 5000. Now, all you need to do is use this Forwarding endpoint in the next section.

$ ngrok http 5000
ngrok by @inconshreveable
Session Status online
Version 2.1.18
Region United States (us)
Web Interface http://127.0.0.1:4040
Forwarding http://532f619b.ngrok.io -> localhost:5000
Forwarding https://532f619b.ngrok.io -> localhost:5000
Connections ttl opn rt1 rt5 p50 p90
 0 0 0.00 0.00 0.00 0.00

Configuring the Webhook Fulfillment in Dialogflow

We need to now get back to the Dialogflow console. Assuming that you are logged in, go to the Fulfillment section from main menu.

This will bring up a screen as shown below:

It is disabled by default. Go ahead and slide it to the right to enable it. This will ask you for the URL. Please copy the HTTPs URL that ngrok utility provided you and put that with the trailing /webhook since that the URL that our API server is listening on. A sample configuration is shown below:

Click on SAVE.

The next step is to modify the Intent to ensure that the Webhook Fulfillment is enabled. Go to the Intents list and go the RESPONSES section first. Remove the Text response that is currently there and replace it with an error message as shown below. This is the default message that will be delivered as the final response in case there is a problem with the Webhook fulfillment.

Finally, go to the Fulfillment section as shown below:

Click on the ENABLE FULLFILLMENT link. Then navigate to the bottom and do the following (as the screen indicates):

1. Select the Enable webhook call for this intent option in the Fulfillment section as shown.

Click on SAVE.

This completes the external Webhook Integration. You can test out the Agent again via the Agent Test Console. Just ensure that your local API server is running and that you have launched ngrok currently and used the correct Webhook URL.

AGENT PERSONALITY

You would have noticed that the conversation that we have created is not very personal or friendly. There are multiple ways in which you could have made it better. Think about the following points:

• If the user gave a feedback rating that less than or equal to 3, you might want to provide a different message and maybe provide a different response.
• You could possibly repeat or confirm the statements so that the feedback helps the reader to confirm if things are going fine.
• In a real system, you might want to even know the context of identifying the user, mentioning about the visit to the resort that they just had and then ask for feedback, first asking for permission.

The above points are just suggestions and the more you add personality to your Agent, the better the experience will be. For a detailed reference and best practices, head over to the Design site on Google.

SUMMARY

This hands-on guide covered how to create a simple and linear conversational flow via the Agent in Dialogflow. This is just a fraction of the features and complexity that you can build into your Agent. For more information, check out the excellent documentation available at the Dialogflow documentation page.

FEEDBACK

Hope you enjoyed building out the Conversational Agent. Please send me feedback on romin.k.irani@gmail.com if you find any errors or have suggestions/improvements. Thanks a ton in advance.

Additional Agent Ideas

Here are a few alternative Agent ideas, that you can try and create:

Pizza Ordering Agent

This agent will help take an order for a Pizza. The entities that you can model are Pizza Crust (thick, thin, etc), Toppings (use multiple toppings like mushrooms, capsicum, olives, etc), Size (Small, Medium, Large) and so on.

Sandwich Ordering Agent

This is similar to the Pizza Ordering one, except that you can model entities like Type of Bread (White, Brown, etc), Sauces (Mayo, Ranch, Barbeque, etc), Veggies (onions, capsicum, olives, jalapeno,etc).

Support Agent

This Agent helps a customer raise support tickets in the system. The customer can provide information like:

• Priority of Ticket (High, Medium, Low)
• Department to address the Ticket to (Admin, HR, Finance)
• Brief Description of the Problem