Using voice to control a website with Google Home, part 3/3
Creating the website and extending our Action to display all images
Update 01.01.2020
This tutorial was written in 2017, and it is now outdated. Thank you for all the kind feedback and suggestions. If you finished the tutorial, congratulations! :)
In the previous chapters we created a simple Google Action that allows us to speak a number to the Google Home and have it repeat back the number to us. We then set up a database to store and retrieve the command we spoke to Google Home, and then we added code to the Action to store the number in the database. We also looked at how the number can be retrieved using the built-in API of the Google Firebase Realtime Database.
This is part 3 out of 3 chapters in this tutorial:
Part 1: Introduction, prerequisites, and building the Google Action
Part 2: Setting up the database and connecting our Google Action
Part 3: Creating the website and extending our Action to display all images
In this chapter we will write a simple website that displays dog pictures. This website will use the database API to figure out which picture to display. The end result is that the website will be updated with the correct picture every time we ask the Google Home for a picture.
We will also extend the Google Action to handle showing all pictures.
We’re currently at stage 3:
Extending the Google Action to handle all pictures
Before we start writing the website, we will add an intent to show all pictures. To do that, navigate to your Dog Pictures project in Dialogflow, and create a new intent.
Call the new intent ‘Show all pictures”.
Add two user expressions: “show me all pictures” and “show all pictures”.
In the Action input box, type showAllPictures, this is the Action we will use when we add code to the webhook function later. It should look like this:
Navigate to the Fulfillment section of your new intent, and check the box that says ‘Use webhook’.
Your intent should now look like this:
Hit the ‘Save’ button at the top to save your intent.
You don’t need to put anything in the text response field, because this intent is being handled by the webhook function.
Next we need to edit our function to handle this new intent. Navigate to the Fulfillment screen in the left navigating menu:
Using the inline editor, edit the index.js file to handle the new intent.
You will add a new action handler after the ‘showPicture’ handler you added earlier. The new handler is called ‘showAllPictures’. This name needs to be exactly the same as the name you put in the Action field for the intent. So if you called it something else, you need to use that name for the handler.
This is what I put in the Action field:
So I need to use that exact name for my action handler in my index.js file.
The code for ‘showAllPictures’ looks like this:
'showAllPictures': () => {
if (requestSource === googleAssistantRequest) {
saveToDb('0');
sendGoogleResponse('all pictures'); // Send simple response to user
} else {
sendResponse('all pictures'); // Send simple response to user
}
}Notice that we use the number 0 to represent all pictures. When you ask Google Home for all pictures, the number 0 is going to be stored in the database. We will later write the website so that when the API returns 0, we will interpret that to mean all pictures. Also notice that I store 0 as a string. That’s because my previous showPicture function also store numbers as a string. You can of course write your functions so that the numbers are stored as the datatype number if you like.
Your index.js should now look like this with the new code added:
Notice how showAllPictures has been added between the handlers for showPicture and input.unknown. Remember to put a comma after the new handler, like shown in the screenshot.
If you don’t want Google Home to confirm your choice, but just wait for your next command, you need to specify an empty string in the sendGoogleResponse() function call, like this:
Hit the ‘Deploy’ button to deploy you function.
Once deployed, you can test your app in the simulator again. When you ask for all pictures, the number in the database should be updated to 0.
The sequence of commands shown above should change the number to 4, then to 0, and then to 12.
We have now completed our function and our intents, we can handle individual pictures and all pictures. We will now write the website that actually shows the pictures.
Writing the website that will show the pictures
We will create a very basic page that will let you see either all images or one specific image. This can be done in thousands of different ways. We will create a small html and css file, and include some javascript in the html file to make it all work. This file can be located on your computer and then opened in a web browser, you do not need to host these files on a web server.
Please note, the following code has only been tested in Chrome, it may not work as written in IE, Safari, or Firefox.
Here’s the video again to quickly remind us of what we’re trying to achieve:
What we will show
We will show a web page that either displays a grid of dog pictures, like this:
Or a single dog picture, depending on what we ask Google Home for, like this:
To achieve that we will build a simple webpage that has a container–a <div>–that will either display the grid, or display a single picture.
All of the code and files needed for this project is available on GitHub. All you have to do is to download the files, and replace the path to the database in the index.html file with the path to your database.
Please remember to close the web page when you’re done testing, otherwise it will keep asking your API every second for as long as the web page is open on your computer. If you leave it open you may be charged for using the Google Cloud services, so always close the browser when you’re done.
The HTML for displaying the pictures can look like this:
<div id = "image_container">
<div id = "grid_view">
<div class = "image_in_grid" onclick="showFullImage(1)">
<img src = "img/01.png" />
</div>
<div class = "image_in_grid" onclick="showFullImage(2)">
<img src = "img/02.png" />
</div>
<div class = "image_in_grid" onclick="showFullImage(3)">
<img src = "img/03.png" />
</div>
<div class = "image_in_grid" onclick="showFullImage(4)">
<img src = "img/04.png" />
</div>
</div>
</div>This sets up an image_container that by default shows all the images in a grid.
We need two JavaScript functions, one to display the images as a grid, and one to display a specific image (this can of course be done in the same function, but here we’ve used two different functions to hopefully increase clarity).
The first function, showFullImage(image), takes an image as input, and displays that image. The function looks like this:
function showFullImage(image) {
var image_path = ("img/0" + image + ".png");document.getElementById('image_container').innerHTML = "<img src = '" + image_path + "' />";
console.log(image_path);}
This function creates a file path to construct the full path to the image you want to show, it then replaces the content of ‘image_container’ with an <img> tag that contains the path to that image. If ‘1’ is passed to the function, the full path will be “img/01.png”. The console.log(image_path) code is not necessary, but it helps you when you debug the function to see what the value of image_path is.
The second function is showGridView(). Calling this function simply displays all the images in a grid. This function doesn’t take any parameters, and it looks like this:
function showGridView() {
var content_string = "<div id = 'grid_view'>\
<div class = 'image_in_grid' onclick='showFullImage(1)'>\
<img src ='img/01.png' />\
</div>\
<div class = 'image_in_grid' onclick='showFullImage(2)'>\
<img src = 'img/02.png' />\
</div>\
<div class = 'image_in_grid' onclick='showFullImage(3)'>\
<img src = 'img/03.png' />\
</div>\
<div class = 'image_in_grid' onclick='showFullImage(4)'>\
<img src = 'img/04.png' />\
</div>\
</div>";
document.getElementById('image_container').innerHTML = content_string;
}This function creates a variable called content_string which has all the html needed to show all the images as a grid. We then use document.getElementById to put this html in the ‘image_container’.
Finally, we need a function that will access the API we created in chapter 2 and call the appropriate function, either showImage(image) with the correct image number, or showGridView(). This function needs to constantly check the API to see what the value is, since we may change it by talking to Google Home at any given time. To do that we will call the function when the page loads, and then we will have the function call itself every second. Essentially the function will poll the API once every second to see if there are any changes. This is not necessarily the best solution, but it is fairly easy to implement, and it is sufficient for our prototype. For a real production app you may want to look into other technologies such as web sockets or long polling.
With that disclaimer, this is what the refresh() function looks like:
function refresh() {
var xhttp = new XMLHttpRequest();
xhttp.open("GET", "https://dog-pictures-2d717.firebaseio.com/picture.json", true);
xhttp.send();
xhttp.addEventListener("readystatechange", processRequest, false);function processRequest(e) {
if (xhttp.readyState == 4 && xhttp.status == 200) {
var response = JSON.parse(xhttp.responseText);
var numberString = response.number;if(numberString < 5 && numberString > 0) {
showFullImage(numberString);
} else {
showGridView();
}
console.log(numberString);
}
}
setTimeout(refresh, 1000);
}
Remember to change the API url to match the URL of your API.
You have to change this line to point to your API:
xhttp.open("GET", "https://dog-pictures-2d717.firebaseio.com/picture.json", true);The URL of your API was covered in chapter 2. You find the URL by going to the Firebase Realtime Database console for your Dog Pictures project:
In the screenshot above I’ve circled the URL. You need to add picture.json to this URL (remember your URL will be different from this), so that it looks like this:
https://dog-pictures-2d717.firebaseio.com/picture.json
The refresh function creates a XMLHttpRequest() object, and it then opens a connection to the API through this call: xhttp.open(“GET”, “https://dog-pictures-2d717.firebaseio.com/picture.json", true). xhttp.send() then sends the request to the API, and we then set up an event listener for the response from the API.
The response from the API triggers the inner function called processRequest(e). Here we have to check to see if we got a successful response. Note that we’ve not written code to handle errors here, this assumes success, in a real app you will want to handle errors as well.
We take the successful response from the API, for instance: {“number”:”4"}, and we extract the number and put it in the numberString variable like this:
var response = JSON.parse(xhttp.responseText);
var numberString = response.dogPicture;If the number that comes back from the API is between 1 and 4
if(numberString < 5 && numberString > 0)we will call the showFullImage(image) function with the correct image number that is now stored in var number_string like this: showFullImage(numberString);
If the number returned is 0, or larger than 4, we will simply show the grid. If you remember from chapter 2 we defined the function to write 0 to the database when we asked for all pictures. So if the number is 0, showGrid gets called:
else {
showGridView();
}The last thing that happens in the function is that we write the numberString to the console.log for debugging purposes, and we call the function again from within the function (recursively) by using setTimeout to wait 1000 milliseconds before calling refresh():
console.log(numberString);
}
}
setTimeout(refresh, 1000);
}The refresh() function will now be called once every second, constantly asking the API what to show. Again, there are better ways of doing this, but this is sufficient for our current prototype.
Please remember to close the web page when you’re done, otherwise it will keep asking your API every second for as long as the web page is open on your computer. If you leave it open you may be charged for using the Google Cloud services, so always close the browser when you’re done.
You can read more about Google Cloud free limits here.
The full html/js and css file
The full HTML file now looks like this:
<html>
<head>
<link rel="stylesheet" href="default.css" >
<body>
<a href = "#" onclick="showGridView()">back to grid</a>
<p/>
<div id = "image_container">
<div id = "grid_view">
<div class = "image_in_grid" onclick="showFullImage(1)">
<img src = "img/01.png" />
</div>
<div class = "image_in_grid" onclick="showFullImage(2)">
<img src = "img/02.png" />
</div><div class = "image_in_grid" onclick="showFullImage(3)">
<img src = "img/03.png" />
</div><div class = "image_in_grid" onclick="showFullImage(4)">
<img src = "img/04.png" />
</div></div></div><script>
function showFullImage(image) {
var image_path = ("img/0" + image + ".png");document.getElementById('image_container').innerHTML = "<img src = '" + image_path + "' />";
console.log(image_path);}
function showGridView() {
var content_string = "<div id = 'grid_view'>\
<div class = 'image_in_grid' onclick='showFullImage(1)'>\
<img src ='img/01.png' />\
</div>\
<div class = 'image_in_grid' onclick='showFullImage(2)'>\
<img src = 'img/02.png' />\
</div>\
<div class = 'image_in_grid' onclick='showFullImage(3)'>\
<img src = 'img/03.png' />\
</div>\
<div class = 'image_in_grid' onclick='showFullImage(4)'>\
<img src = 'img/04.png' />\
</div>\
</div>";
document.getElementById('image_container').innerHTML = content_string;
}function refresh() {
var xhttp = new XMLHttpRequest();
xhttp.open("GET", "https://dog-pictures-2d717.firebaseio.com/picture.json", true);
xhttp.send();
xhttp.addEventListener("readystatechange", processRequest, false);function processRequest(e) {
if (xhttp.readyState == 4 && xhttp.status == 200) {
var response = JSON.parse(xhttp.responseText);
var numberString = response.number;if(numberString < 5 && numberString > 0) {
showFullImage(numberString);
} else {
showGridView();
}
console.log(numberString);
}}
setTimeout(refresh, 1000);
}
refresh();</script></body>
</head>
<html>
And the CSS file default.css looks like this:
#image_container {
width: 800px;
height: 600px;
float: left;
}
.image_in_grid {
position: relative;
width: 400px;
height: 300px;
float: left;
}
.image_in_grid img {
width: 100%;
}You also need some images of cute dogs, feel free to use these adorable examples, all from Wikimedia commons, they are also available on GitHub:
File structure
You need to put the index.html and default.css in the same directory. In that directory you will create a folder called img, and in that folder you will put the dog pictures. Name the dog pictures 01.png, 02.png, 03.png and 04.png.
You can of course name the files and folders what you like, but the example code assumes that you’ve used those names and this file/folder structure:
The files with this structure are all available on GitHub.
Testing it all
Now you’re ready to test it all, end to end. To do that, open your browser and navigate to the index.html file, you should see a grid of images, unless the value in your database is currently 1,2,3 or 4, in which case you will see an image of one of the dogs. Open the javascript console and you will see that every second the webpage is asking the API about which number to display:
In this example I’ve asked for image number 4, and once every second the web page asks the API for which number, and the answer is always the same until I give Google Home a different instruction.
Open the test app on your Google Home device, and ask for different images, using the expressions you defined for your intents. The web page should now update, showing the images you asked for.
You can also open the Google Assistant on your phone and talk to the app. It should also work.
Again, please remember to close the browser when you’re done, otherwise you will keep hitting the API and you may end up being charged to use the service.
Did it all work? If so, congratulations! You have successfully built a prototype that lets you control a web page with voice using the Google Actions for your Google Home and Assistant, and the Google Firebase Realtime Database.
If it didn’t work, reach out to me and I may be able to help you troubleshoot. You can find my contact information at the end of this article.
This is just the beginning, you can now use this knowledge to build sophisticated apps that can let you control web pages, iOS and Android apps, or basically any service that can use an API.
Hopefully this tutorial has served as an introduction and inspiration to what you can do with Google Home and voice, and hopefully you’ve gotten a glimpse into just a few of the wonderful tools available to you in the Google Cloud/Firebase ecosystem.
If you’ve built something cool, or just want to reach out to talk about design or prototyping, I’d love to hear from you! Please also leave a comment or send a message if you have suggestions on how these posts can be improved and simplified. You can find me on LinkedIn and Quora.
Thank you so much for reading, I sincerely hope this was useful to you. Happy prototyping! ☺
