Video can be a very effective way to use high-production visuals in your Interactive Canvas game for the Google Assistant. In a previous post, we discussed using video loops in an Interactive Canvas web app.

This post discusses the necessary steps to prepare video files and to write the JavaScript logic to play seamless video loops in an Interactive Canvas web app.

Seamless video loop for Interactive Canvas

Media Source Extensions

Creating seamless video loops requires using Media Source Extensions (MSE), which is a browser feature that extends the HTML media element to allow JavaScript to generate media streams for playback.

You have to write JavaScript code to download…


Using Interactive Canvas to create an Action for the Google Assistant combines the best of conversational interfaces with the rich visual capabilities of HTML.

We’ve been using Interactive Canvas for a while, experimenting with various ideas and working with partners to launch their Actions. Along the way, we’ve learned some lessons about what works well, and we’ll pass these on in this post to help you create a successful Action using Interactive Canvas.

Note: At this time, Google is only approving Canvas Actions that are gaming experiences.

Design

Actions using Interactive Canvas are conversational Actions. You should start designing your game…


At I/O this year, we announced Interactive Canvas, a new way to build immersive, full-screen experiences that combine the power of voice, visuals, and touch on Smart Displays and Android phones. Starting today, you can build and deploy your Interactive Canvas Action to users. Interactive Canvas is currently only available for games, but we will consider other verticals in the future.

With Interactive Canvas, you can create Conversational Actions that have rich, full-screen visuals and media using existing web technologies: HTML, JavaScript, CSS, and WebAssembly. …


In a previous post, I discussed the design and implementation of my Magnificent Escape game for the Google Assistant. I’ve received lots of questions regarding the code and the Dialogflow agent. So, I’ve decided to open source the game.

I’ll explain how the project is organized and also discuss the latest learnings from the game in this post.

Github repo

The game’s Dialogflow agent and fulfillment source code is available on GitHub. Some of the main files include:

  • app.js — the main entry point for the Node.js app, which launches a web server for handling the Dialogflow agent fulfillment.
  • fulfillment.js — the…


Did you know that you can let your users record their own audio for a Google Assistant Action? In this article, we’ll show you how to create a simple prototype that allows users to make an audio recording that can then be played back on the Google Assistant.

We need a way for users to record audio. However, we didn’t want to make the user install another app, so we decided on using a web page instead.

Actions on Google supports playback of audio with either SSML, which is limited to 240 seconds, or with a media player for longer…


In a previous post, I discussed the design and publication of my escape-the-room Action called “Magnificent Escape”. Since the game was launched for the Google Assistant, various improvements have been made which are discussed in this post.

Regular releases

Since the Action was launched on October 26, there have been 7 updates published. These have included improved gameplay, many additional training phrases for intents, and some bug fixes.

Dialogflow automatically versions agents that are used for Actions on Google. However, fulfillment isn’t versioned. So, I keep 2 Dialogflow agents for development: one with the development fulfillment URL and one with the production…


I created an escape-the-room Action for the Google Assistant called “Magnificent Escape”. A few interesting issues were uncovered during the development and, in this post, I’ll discuss some solutions that you could use in your own Actions.

Design

The game follows the classic escape-the-room format, but I wanted to make it work as a conversational experience. All the Actions on Google surfaces are supported, but the best experience is playing the game by voice.

The Action is implemented in Dialogflow and the fulfillment is implemented in Node.js using the Actions on Google client library. The fulfillment is about 5000 lines of…


Once your Action is deployed, your analytics in the Actions Console are a valuable resource to understand how your Action is used and how well it is performing.

If you are using Dialogflow to implement your Action, it has an analytics page that gives you insight on how well your agent is performing. The Actions Console also has an analytics section that covers usage, health, and discovery.

In this post, we use the analytics of several deployed Actions to learn how well they are performing and what improvements might be needed.

Let’s start with data

One of our sample apps, Number Genie, was one…


Dialogflow has a valuable feature called History which shows a high-level view of the conversations between your users and your Action. Once your Action is deployed, this page is an important tool to analyze how users are interacting with your Action and how your conversational design might need to be improved.

Lessons learned

We recently launched sample Actions, which let you create working Dialogflow agents with a few clicks. These agents implement our best practices for Actions on Google. …


If you’ve spent any time testing your Action in the Actions simulator, you’ve probably seen various error messages and haven’t always been sure how to resolve them. In this post, we cover some of the most common errors, as well as provide strategies on how to make you a more efficient developer and make your Actions more robust.

Final response must be set

Let’s start with one of the most common errors that developers are confused about:

“MalformedResponse: ‘final_response’ must be set”

The error message sounds a bit cryptic, but it’s explaining that the JSON response the Actions on Google platform received from your Action…

Leon Nicholls

Former Google Assistant Developer Relations Engineer

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store