Use video loops with Interactive Canvas

Video can be a very effective way to use high-production visuals in your Interactive Canvas game for the Google Assistant. In a previous post, we discussed using video loops in an Interactive Canvas web app.

This post discusses the necessary steps to prepare video files and to write the JavaScript logic to play seamless video loops in an Interactive Canvas web app.

Seamless video loop for Interactive Canvas

Media Source Extensions

Creating seamless video loops requires using Media Source Extensions (MSE), which is a browser feature that extends the HTML media element to allow JavaScript to generate media streams for playback.

You have to write JavaScript code to download and buffer the video data, which is then handed directly to the HTML video tag buffer using MSE. Since MSE is a low-level API, there are a lot of details to get right, but this post will walk through all the necessary steps.

Prepare the video

First, you have to convert the video to the fragmented MP4 file format for streaming. A normal MP4 file consists of a header and the media data. The header is located at the end of the file. For streaming, the header has to be moved to the beginning of the file.

You’ll need the following tools to prepare the video:

• FFmpeg
• MP4Box

Use FFmpeg to convert the file to have the correct codec:

ffmpeg -i video.mp4 -an -codec:v libx264 -profile:v baseline 
   -level 3 -b:v 2000k videocodec.mp4

Run the following command to put the header at the front of the file and to ensure that the fragments start with Random Access Points:

MP4Box -dash 1000 -rap -frag-rap videocodec.mp4

A new MP4 file is generated with a “_dashinit” postfix in the filename. Upload this file to your web server.

Play the video

Now that the video file is in the correct format, you’ll use MSE to load and play the file with the HTML media element.

First, define an HTML video element in your web app:

<video id='vid' />

Get a reference to the video element in JavaScript:

let video = document.getElementById('vid');

Create a MediaSource object and create a virtual URL using URL.createObjectURL with the MediaSource object as the source. Then, assign the virtual URL to the media element’s “src” property:

let mediaSource = new MediaSource();
video.src = window.URL.createObjectURL(mediaSource);

Wait for the MediaSource “sourceopen” event to tell you that the media source object is ready for a buffer to be added. Create a SourceBuffer using the MediaSource addSourceBuffer() method with the mime type of the video and then start downloading the file:

let sourceBuffer;
mediaSource.addEventListener('sourceopen', function(){ 
   sourceBuffer = mediaSource.addSourceBuffer(
       'video/mp4; codecs="avc1.42c01e"');
   fileDownload('videocodec_dashinit.mp4');              
});

Use XMLHttpRequest to download the file as an ArrayBuffer:

function fileDownload(url) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', url, true);
  xhr.responseType = 'arraybuffer';
  xhr.send();
  xhr.onload = function(e) {
    if (xhr.status != 200) {
      onLoad();
      return;
    }
    onLoad(xhr.response);
  };
  xhr.onerror = function(e) {
    video.src = null;
  };
};

Append the file data to the SourceBuffer with appendBuffer():

let allSegments;
function onLoad(arrayBuffer) {
  if (!arrayBuffer) {
    video.src = null;
    return;
  }
  allSegments = new Uint8Array(arrayBuffer);
  sourceBuffer.appendBuffer(allSegments);
  processNextSegment();
}

Call the play()method on the video element and append video segments to the source buffer when there is less than 10 seconds left in the playback pipeline:

function processNextSegment() {
  // Wait for the source buffer to be updated
  if (!sourceBuffer.updating &&   
       sourceBuffer.buffered.length > 0) {
    // Only push a new fragment if we are not updating and we have
    // less than 10 seconds in the pipeline
    if (sourceBuffer.buffered.end( 
          sourceBuffer.buffered.length - 1) -  
          video.currentTime < 10) {
      // Append the video segments and adjust the timestamp offset 
      // forward
      sourceBuffer.timestampOffset =  
          sourceBuffer.buffered.end(
              sourceBuffer.buffered.length - 1);
      sourceBuffer.appendBuffer(allSegments);
    }
    // Start playing the video
    if (video.paused) {
      video.play();
    }
  }
  setTimeout(processNextSegment, 1000);
};

The video will keep looping forever and there won’t be any delays between each loop.

Next steps

Even if you don’t understand all of the technical details of using MSE, just copy and paste the code above into an HTML file and try it out in a browser first. Once that is working as expected, add the necessary JavaScript for an Interactive Canvas web app and deploy it as an Action.

It’s also possible to use this technique to create seamless transitions between different videos, but that requires managing a buffer per video.

That’s all you need to display high-production visuals in your Interactive Canvas game. Have fun!

To share your thoughts or questions, join us on Reddit at /r/GoogleAssistantDev. Follow @ActionsOnGoogle on Twitter for more of our team’s updates, and tweet using #AoGDevs to share what you’re working on.