Developing a Google Assistant app with NodeJs, Dialogflow & Firebase Functions
What is Google Assistant?
Google Assistant is a virtual personal assistant developed by google available on variety of devices like mobile phone, smart home device, on a smart watch or on a TV or even the web. It can engage in two way conversations and can work entirely with voice.
What can it do?
With the assistant, you can control smart home devices, do most of the functions on your phone like placing a call, sending a text etc with just your voice, play content and also interact with apps. It also has routines which is a sequence of things which can be done with a single command. So existing content can be packaged into an assistant app or apps can be built without writing any code using templates or by writing custom apps.
How to develop custom apps for Google Assistant?
For more flexibility we can also develop apps using the APIs provided by Google. There are two APIs we can use are Actions SDK and Dialogflow
Using the actions SDK, your app can perform actions. An action defines an entry point into your app’s functionality and maps an intent that describes the action to the fulfillment that processes the intent. The fulfilment gets the user input as string, processes the input and provides a response. This interaction continues back and forth till the action is complete.
Dialogflow is a conversational platform that lets you design and build actions by wrapping the functionality of the Actions SDK and providing additional features such as an easy-to-use IDE, natural language understanding (NLU), machine learning, and more. So it sits above the actions SDK and works with actions SDK.
Developing the App
Here I will show how to create an app called InspireMe using the Dialogflow API. User can talk to this app to get inspirational quotes and passages.
The flow of the app works like this.
User specifies an intent to Google Assistant or explicitly asks for taking with the app → The welcome intent of the app is triggered → Fulfilment is triggered for the intent → Fulfilment picks a quote and returns in the response → That quote is played in the response -> User is given an option to listen another quote or end the conversation → If user selects Yes -> another quote is picked and played -> when user says no -> App wishes user a good day and the flow ends.
Design a conversation
Here is a sample conversation.
Default flow:
User : Hey google tell me some inspiring quotes/motivational quotes/inspirational quotes
OR if the user knows the app already, he can ask by name
User: Hey google, talk to LitInspire
App: Welcome to Lit Inspire. With great quotes and inspiring passages, I will inspire you.
Here is a quote: Real Artists ship — This was said by Steve Jobs asking us to share our work with the world and not hold back.
Do you want to hear another quote?
User: Yes Please
App: here is another one . <Another Quote>. Do you want to hear another one ?
User: Sure
App: Great. Here is another quote. <Another Quote>. Do you want to hear another one.
User: No Thanks.
App: Hope you are inspired to take on your challenges. Have a good day!
The user could cancel at any time by saying cancel.
Create a project and define actions
To create the project you should first setup a project in the actions on google console.
Then choose a category which your app belongs to. It will show a form asking you to the select the default invocation phrase, i.e the name of the app the user can explicitly ask for and save.
Next click on the ‘actions’ menu and you will be prompted to build your first action. When you click on that you will be take to a menu with many built in actions and custom action. Built in actions help you to build apps based on template or you can customize later. Here we will proceed with a custom action. Select ‘custom action’ and click Build.
You will be redirected to the dialogflow console.
This is the agent which maps user queries to intents and invokes a specific action.
If you see the conversation above, the user first starts the app by asking for a specific quote or asking for the app by name.
So the initial intent could be start_app which is triggered when either asked by name of if google matches the invocation phrase (tell me an inspiring quote) to our app.
So create an intent with name ‘start_app’ and save
It gives options below to enter
contexts: Context is a way to link the conversation. Let us come back to this later.
events: Events are a way to enter the app without matched text. i.e a way to enter the app by calling by name. Select ‘Welcome’ and ‘Google Assistant Welcome’ which says that when users ask for this app by name, start_app intent is triggered.
actions and parameters: Specifies the action that should be triggered when this intent is specified. Dialogflow can also get ask questions to get all the parameters needed before fulfilling the action. So in this screen you can specify the action and also the parameter names to use later. For e.g if you want to listen a quote in a specific category, you can have that as a parameter. Currently the app just tells a quote in any category, so there is not paremeter in this app. So just specify an action name.
training phrases: Training pharses are phrases which the user can say to match this intent without asking for the app by name. for e.g tell me a motivating quote, tell me an inspiring quote, give me some inspiration, tell me a quote etc. Enter different varieties of training phrases all of the different ways a user can ask for quotes etc.
Responses: This section can be used to provide hardcoded responses. You can enter a few quotes here and they will be used if our service fails. If you want to end the conversation here, you can select, end this conversation slider. In this example we don’t want hardcoded response, so don’t select this.
Fulfillment: This is used if we want to create a fulfillment service which can provide user response instead of hard coding. Here in this example we want to use the fulfillment service. So select ‘Enable webhook call for this intent’. Later when we have the fulfillment service ready we can enter the fulfillment url in the fulfillment section.
This action will play the introduction and also return a quote. After that it asks user either to listen to another quote or quit. So we need two more intents one to handle when the user says ‘yes’ and one when user says ‘no’.
Contexts:Though there is only one question for which the user might answer yes here, the user might say ‘yes’ to different questions in conversation, but our app should know in which context the user said yes to. So when the user says ‘yes’ since the app knows the context the user is in, it will only match the action for that context not all the intents which have yes as a triggering phrase.
So create new intents one for one_more_yes and one for one_more_no.
In the one_more_yes, we want to listen to process the yes only if he says it in the one_more context i.e after he has listened to the first quote and wants to listen more. So in the contexts section, set the input context to one_more and save. Set the training phrases to be variations of the yes response like sure, absolutely, yes,please etc. Also enable webhook call for fulfillment. Set the action to be same as the intent name.
Do the same thing for the one_more_no intent. Set the input context to one_more and set the training phrases to variations of no and enable webhook call for fulfillment. Set the action to be same as the intent name.
Fallback intent: Next there may be times where the user might say something totally unrelated to the app and we should have a way of handling that. Google already provides a ‘Default Fallback Intent’. Open that and set an action name like input.unknown and enable fulfillment so that we can handle it in our service.
This should cover all the intents which are necessary for our app. Next lets build the fulfillment service.
Build fulfillment
The fulfillment service should accept a POST request from Dialogflow with the matched intent details. Your service should handle the request and return a response as per the Dialogflow V2 specification.
We could use any language for creating the service, but Dialogflow provides a NodeJS based fulfillment SDK which will make our job very easier. So we will be using the NodeJS SDK.
Our fulfillment service will be a Google Cloud functions for firebase . Cloud Functions for Firebase lets you automatically run backend code in response to events triggered by Firebase features and HTTPS requests. Your code is stored in Google’s cloud and runs in a managed environment. There’s no need to manage and scale your own servers.
Cloud Functions runs Node v6.14.0, so install node v6.14 or higher version.
https://nodejs.org/en/download/
Once nodejs is installed, set up the firebase cli following the instructions here
The steps are
- Install firebase-cli. In many cases, new features and bug fixes are available only with the latest version of the Firebase CLI and the
firebase-functionsSDK. So first install the latest versions of firebase-functions and firebase-admin
npm install firebase-functions@latest firebase-admin@latest --save
npm install -g firebase-tools2. Once the dependencies are installed, do the following to initialize the project
a. Run firebase login to log in via the browser and authenticate the firebase tool.
b. Go to your Firebase project directory.
c. Run firebase init. The tool gives you an option to install dependencies with npm. We will select functions for the function and hosting for any public files. The init tool will guide through various options to choose the language, and the initialization options for functions and hosting
myproject
+- .firebaserc # Hidden file that helps you quickly switch between
| # projects with `firebase use`
|
+- firebase.json # Describes properties for your project
|
+- functions/ # Directory containing all your functions code
|
+- .eslintrc.json # Optional file containing rules for JavaScript linting.
|
+- package.json # npm package file describing your Cloud Functions code
|
+- index.js # main source file for your Cloud Functions code
|
+- node_modules/ # directory where your dependencies (declared in
# package.json) are installedOnce the project is initialized, we can develop our functions.
This is the package.json for the dependencies we have. We are using actions-on-google, firebase-admin, firebase-functions and google cloud datastore. We are using es-lint for linting. Code linting is a type of static analysis that is frequently used to find problematic patterns or code that doesn’t adhere to certain style guidelines.
{
"name": "functions",
"description": "Cloud Functions for Firebase",
"scripts": {
"lint": "eslint --fix \"**/*.js\"",
"start": "firebase serve",
"deploy": "firebase deploy",
"test": "npm run lint"
},
"dependencies": {
"@google-cloud/datastore": "1.1.0",
"actions-on-google": "^2.0.0",
"ajv": "^5.0.0",
"firebase-admin": "^5.11.0",
"firebase-functions": "^1.0.0"
},
"devDependencies": {
"eslint": "^4.19.1",
"eslint-config-google": "^0.9.1"
},
"private": true,
"version": "0.0.1"
}Next let us look at the functions. When user invokes the app by name or google matches the request to any of our intents, then it sends a request to our app with the intent and other parameters. In our function we will implement the callback function for each intent.
'use strict';
//Initialize librariesconst {dialogflow} = require('actions-on-google');
const functions = require('firebase-functions');
const Datastore = require('@google-cloud/datastore');
const {
SimpleResponse,
BasicCard,
Image,
Suggestions,
Button
} = require('actions-on-google');// Instantiate a datastore client
const datastore = Datastore();
const app = dialogflow({debug: true});app.middleware((conv) => {
});//Setup contextsconst Contexts = {
ONE_MORE: 'one_more'
};app.intent('quit_app', (conv) => {
conv.close("Have a good day! come back again. Bye!");
});app.intent('start_app', (conv) => {
conv.contexts.set(Contexts.ONE_MORE,5);
const initMessage = ` Welcome to LitInspire. With great quotes and inspiring passages, I will inspire you.`;return getQuote().then((entity)=>{
return getMessageFromQuote(entity,initMessage,conv);
});
});app.intent('one_more_yes', (conv) => {
conv.contexts.set(Contexts.ONE_MORE,3);
const initMessage = `Great! Here is another one.`;
return getQuote().then((entity)=>{
return getMessageFromQuote(entity,initMessage,conv);
});
});app.intent('one_more_no', (conv) => {
conv.close("Hope you're inspired and ready to take on your challenges. Have a good day and come back for more.");
});app.intent('Default Fallback Intent', (conv) => {
console.log(conv.data.fallbackCount);
if (typeof conv.data.fallbackCount !== 'number') {
conv.data.fallbackCount = 0;
}
conv.data.fallbackCount++;
// Provide two prompts before ending game
if (conv.data.fallbackCount === 1) {
conv.contexts.set(Contexts.ONE_MORE,2);
return conv.ask(new Suggestions('Yes Please', 'No thanks'), new SimpleResponse("Would you like to hear a quote?"));
}else if(conv.data.fallbackCount === 2){
return conv.ask(new Suggestions('Yes Please', 'No thanks'), new SimpleResponse("Welcome to LitInspire. With great quotes and inspiring passages, I will inspire you.Would you like to hear a quote?"));
}
return conv.close("This isn't working.Have a good day. Bye! ");
});function getRandomNumber(){return Math.floor((Math.random()*num_quotes)+1);
}function buildReadableQuoteFromEntity(entity){
let readableQuote = entity.quote +
`<break time="1s"/> This was said by ` + entity.author + ` ` ;
if(entity.comments){
readableQuote += entity.comments + ` `;
}
return readableQuote;
}function getViewableQuote(entity){
let viewableQuote = entity.quote +
`.This was said by ` + entity.author + ` ` ;
if(entity.comments){
viewableQuote += entity.comments + ` `;
}
return viewableQuote;
}function getEndingMessage(){
return ` <audio src="https://actions.google.com/sounds/v1/water/waves_crashing_on_rock_beach.ogg" clipBegin="10s" clipEnd="13s">Consider the quote!</audio>
Do you want to listen to another quote?`;
}function getEndingMessageText(){
return `.Do you want to listen to another quote?`;
}function getMessageFromQuote(entity,initMessage,conv){
return conv.ask(new Suggestions('Yes Please', 'No thanks'), new SimpleResponse(initMessage),
new SimpleResponse( {text: getViewableQuote(entity) + getEndingMessageText(),
speech: `<speak> ` + buildReadableQuoteFromEntity(entity) + getEndingMessage() + ` </speak> ` }));
}function getQuote(){
return new Promise(((resolve,reject) => {
let randomQuoteNum = getRandomNumber();
console.log("the id of the quote is: quote_"+randomQuoteNum);
const key = datastore.key(['quote', 'quote_'+randomQuoteNum]);
console.log("Querying datastore for the quote..."+key);
let readableQuote = '';
datastore.get(key,(err,entity) => {
if(!err){
console.log('entity:'+entity.quote);
resolve(entity);
}else{
reject(console.log('Error occured'));
}
});
}));
}// HTTP Cloud Function for Firebase handler
exports.InspireMe = functions.https.onRequest(app);
Let us look at the intent handler for the start_app. First since we are in the app, we want the user to be able to ask for multiple quotes. Since it is in the one_more context, we will set the context to one_more. Next we need to get a quote and return it. We have the quotes stored in google cloud datastore. So we make a call to the datastore to return a random quote.
app.intent('start_app', (conv) => {
conv.contexts.set(Contexts.ONE_MORE,5);
const initMessage = ` Welcome to LitInspire. With great quotes and inspiring passages, I will inspire you.`;return getQuote().then((entity)=>{
return getMessageFromQuote(entity,initMessage,conv);
});
});
The getQuote() function returns a promise with the quote. And in the intent handler, we use the .then() function of the promise to build the message from quote and return it. In the getMessageFromQuote() method, we can see join together multiple responses and return it. Here we use the ask method, which tells the user our quote and waits for the user response. And in this method we can pass suggestions, and atmost two simple responses. The simple response converts the text to speech. We are using the SSML (speech synthesis markup lanuguage) to specify how to generate the speech. With SSML, we can specify where to pause, and add music to the text. There are other types of responses also like Basic Card, Image, Button and List (Carousel) responses.
function getMessageFromQuote(entity,initMessage,conv){
return conv.ask(new Suggestions('Yes Please', 'No thanks'), new SimpleResponse(initMessage),
new SimpleResponse( {text: getViewableQuote(entity) + getEndingMessageText(),
speech: `<speak> ` + buildReadableQuoteFromEntity(entity) + getEndingMessage() + ` </speak> ` }));
}The two other things to see are how to end the conversation and how to handle unknown input. In the below function you can see that when the user says no, we use the conv.close() function to end the conversation with a message.
app.intent('one_more_no', (conv) => {
conv.close("Hope you're inspired and ready to take on your challenges. Have a good day and come back for more.");
});When the user provides unknown input, google invokes the default fallback function. Lets look at that. We are giving options to the user two times whether to listen to a quote or not. And if the user doesn’t give a valid response even after two times, we are ending the conversation.
app.intent('Default Fallback Intent', (conv) => {
console.log(conv.data.fallbackCount);
if (typeof conv.data.fallbackCount !== 'number') {
conv.data.fallbackCount = 0;
}
conv.data.fallbackCount++;
// Provide two prompts before ending game
if (conv.data.fallbackCount === 1) {
conv.contexts.set(Contexts.ONE_MORE,2);
return conv.ask(new Suggestions('Yes Please', 'No thanks'), new SimpleResponse("Would you like to hear a quote?"));
}else if(conv.data.fallbackCount === 2){
return conv.ask(new Suggestions('Yes Please', 'No thanks'), new SimpleResponse("Welcome to LitInspire. With great quotes and inspiring passages, I will inspire you.Would you like to hear a quote?"));
}
return conv.close("This isn't working.Have a good day. Bye! ");
});Now that’s it. We have written intent handlers for all possible intents and fallback handler also. Now this is ready for testing.
Use the firebase deploy command to deploy the function to firebase functions.
firebase deployOnce deployed you will get a functions url. Go to the dialogflow console and go to the fulfilment menu item and provide the function url as the fulfilment url.
Test the app
Once you have entered the fulfillment url, you can test either in the dialogflow console on the right side of the console or in google assistant.
To test in google assistant, click on ‘Integrations’ in the dialogflow console and select ‘Google Assistant’. On the next screen select ‘test’.
It will open up a test window like this.
We can test each of your intent here. We can also see the request and response and any errors we might encounter.
For e.g. in this app, when I ask to ‘Talk to LitInspire’ , it will retrieve a quote and display on the simulator. It will also read the quote with appropriate pauses as specified in the SSML. It will also show the suggested replies. We can click on the suggested replies to test the next part of the conversation.
We can also view logs by following the link on the simulator. We can see the request and response and exception stack traces also. In the google cloud platform logs console, you can select the right application (actions on google app or the cloud function) to see the logs at the appropriate level.
Once we have verified the happy paths, the fallback paths and conversations and we are satisfied with the result, we can submit the app for approval.
Submit the app for approval
To prepare the app for approval, we have to provide more details so that the app can be visible in the directory. We can provide a short description, more details and invocation phrases to talk with the app.
We also have to provide images in two different sizes. I used the canva app to create these images. We have to provide a banner image and also a logo
Next we have to create a privacy policy.
Google provides a sample doc which we can copy and edit to create our own privacy policy. You can check my privacy policy here.
Next we can select the category the app belongs to and other details helpful for review. We can also select the countries where the app is available and also the devices which can use the app. Then we can submit the app for review.
The app will be reviewed and if there are no issues, the app will be deployed to production. We will get an email informing us of the same. That completes the process of building and deployment.
Summary
In summary we have seen how to design, build and deploy an actions on google app. To summarize the steps, they are
Design a conversation
Create a project and define actions
Build a fulfillment service
Test the actions
Prepare for deployment and submit.
Thanks for staying with me. Hope this was helpful. Feel free to reach out if you have questions. You can comment here or reach out to me via twitter @thisisananth
Join Coinmonks Telegram Channel and Youtube Channel get daily Crypto News
Also, Read
- Copy Trading | Crypto Tax Software
- Grid Trading | Crypto Hardware Wallet
- Crypto Telegram Signals | Crypto Trading Bot
- Best Crypto Exchange | Best Crypto Exchange in India
- Best Crypto APIs for Developers
- Best Crypto Lending Platform
- An ultimate guide to Leveraged Token
- AscendEx Staking | Bot Ocean Review | Best Bitcoin Wallets
- Bitget Review | Gemini vs BlockFi | OKEx Futures Trading
- Best Crypto Trading Bots in the US | Changelly Review
- Earn Passive Income Using Crypto Arbitrage In India
- Huobi Review | OKEx Margin Trading | Futures Trading
- Sparrow Exchange Review | Nash Exchange Review
