Create rich, immersive Google Assistant Games with Interactive Canvas

Published in

Google Developers

7 min readAug 1, 2019

At I/O this year, we announced Interactive Canvas, a new way to build immersive, full-screen experiences that combine the power of voice, visuals, and touch on Smart Displays and Android phones. Starting today, you can build and deploy your Interactive Canvas Action to users. Interactive Canvas is currently only available for games, but we will consider other verticals in the future.

With Interactive Canvas, you can create Conversational Actions that have rich, full-screen visuals and media using existing web technologies: HTML, JavaScript, CSS, and WebAssembly. This means you don’t need to learn new languages and tools; instead, you can use your favorite web development tools, libraries, and frameworks.

Familiar lifecycle

The typical lifecycle of an Action starts with the user invoking the Action. The Action starts and prompts the user for a response through voice and visuals. This back and forth conversation between the Action and the user continues until the user exits the Action.

The lifecycle of an Interactive Canvas Action is very similar to that of a Conversational Action. The main difference is that an Action can now return an HTML response that loads a web app on devices with displays.

When a user talks to a device with a screen, the Dialogflow NLP matches an intent and then provides an HTML response that loads the web app. The web app initializes the Interactive Canvas API and can update its GUI to match the intent response data.

You now have a new API for the web app to invoke an intent. The flow for that intent is the same as for voice input. This API is useful for triggering an intent in the web app logic to prompt the user for input. Typically, this is used for supporting custom interactive controls on the web app GUI, but could also be called at any time when the Action needs user input.

It’s very easy to add support for the Interactive Canvas API to HTML. All it takes is including our Canvas JavaScript library and declaring a callback for handling the matched intents:

<html>
  <head> 
    <script src="https://www.gstatic.com/assistant/interactivecanvas/api/interactive_canvas.min.js"></script>
  </head>
  <body>
    <script>
      const callbacks = {
        onUpdate(data) {
          // update game state based on intent response data
        },
      };
      interactiveCanvas.ready(callbacks);
    </script>
  </body>
</html>

This way, the state and other data of your game can be synchronized between your backend logic and the web app graphics and animation.

Rich visuals

Interactive Canvas gives you pixel-level control over the rendering of your UI. You’re not forced to use any templates or UI components that we provide, so you can design a visual experience that complements your brand identity by using your own web fonts, background images, color palette, and icons.

Rich, immersive experiences are supported by Interactive Canvas.

You can add visual payoff to conversations by using custom layouts, transitions, and animations. HTML gives you a variety of ways to do animations, including manipulating the DOM or using JavaScript to dynamically control HTML canvas graphics or vector graphics with SVG.

We’ve optimized WebGL to allow for smooth 2D and 3D experiences. In our tests, we have achieved a steady 60 FPS while animating hundreds of sprites.

You can adapt these rich visuals to different devices and screen resolutions by using standard CSS media queries or using your favorite library that supports responsive web design. You can also use the standard HTML orientationchange event to detect the device orientation.

Custom GUIs

You can now create your own custom buttons, tables, and lists. Use one of the many free libraries and frameworks that provide rich interactive components such as Material Web Components, Polymer elements, JQuery UI, or React.

Custom GUI components and layouts are possible using HTML.

You can also use Standard HTML touch events as an alternative to voice to allow the user to interact with your web app and to select options. For example, you can create your own slideshows and scrolling lists to allow users to explore options provided by your Action.

However, we recommend that you avoid the use of buttons and other GUI elements if there is a better way to use graphics to directly present items or choices. For example, if you want the user to select from one of two game items, show a graphic of each and then highlight the items to make it clear that the user can select either of the items by voice or touch.

Media

Your Interactive Canvas web app can also use powerful media APIs supported by modern web browsers. The HTML media element allows you to play both audio and video. The media element gives you fine-grained control over the playback and allows you to track various media events.

Sound effects can be a great way to provide feedback and set the atmosphere. Delight your users with special sounds when they reach goals in the game or find easter eggs and buried treasure.

Use the Web Audio API to provide background music and feedback. With this API, you can play with effects and filters and mix sounds and music from various sources.

To help you increase the production value of your Action, we provide thousands of sounds in our sound library, which we host for you for free.

You can also use SSML in the intent responses and create layered sounds and music using the <par> tag, which is a unique feature of our platform.

Speech synchronization

Canvas Actions support the standard SSML <mark> tag, which allows you to place custom markers into the text sequence for the TTS:

<speak>
  Go from <mark name="here"/> here,
  to <mark name="there"/> there!
</speak>

As the TTS is played back, events are generated for each mark, which can then be used in your web app logic to synchronize state updates and animations. The platform automatically provides ‘START’ and ‘END’ events for every TTS.

This feature can be used to highlight words as they are spoken or display graphics that match items as they are mentioned in the TTS.

Note: Support for custom marks will be rolling out in the next few months.

Update: We’ve added support for custom marks to synchronize Interactive Canvas animations with SSML events.

Actions features

Canvas Actions can leverage all of the existing APIs and features provided for conversational Actions.

Canvas web apps do have some restrictions like not being able to use cookies or persist data in local storage. However, if you need to persist data within a session, like tracking the number of turns in a game, you can use conv.data within your fulfillment logic. If you want to store data between sessions, like tracking the highest score, then use conv.user.storage to persist data per user. If you need to sync game state across different platforms, you can use a cloud database, for example Firestore.

The Actions simulator fully supports testing your Canvas Actions, and you can use the Chrome DevTools to inspect the DOM, debug your JavaScript code, and optimize the performance.

You can also leverage daily updates, routine suggestions, and notifications to increase engagement with your users.

Earn an income by using our digital purchases API to provide different kinds of digital goods to your users, such as one-time purchases or subscriptions.

Game design

We encourage you to think about new interaction models that weren’t possible on other platforms.

We have a few design ideas to help you be successful with your game:

Interactive visuals: Take advantage of the visual display in your core game experience by using visual information that users can respond to with their voice.
Voice forward: Explore experiences where voice is the right input. Good examples include adventure games, visual puzzles solved by voice, and conversations with game characters to unlock new gameplay experiences.
Shared space device: Smart displays enable local multiplayer games for families to play in a central location. Persistent games allow multiple people to collaborate with a shared game state.

Shared gaming experiences to solve visual games.

It’s early days for these kinds of games and we encourage you to experiment with new ideas and game designs. But mostly, just have fun! Interactive Canvas is an exciting opportunity for you to create voice-enabled games that are complemented with rich visuals and implemented in a language you already know — regular HTML.

Next steps

To get started with Canvas, take a look at our basic sample or our simple game that we have open sourced on Github. Make sure to read our docs and watch our introductory video to learn about the basics of Interactive Canvas.

Let the games begin!

Read our next post on Interactive Canvas.

To share your thoughts or questions, join us on Reddit at /r/GoogleAssistantDev. Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on.