Streaming Video Playback on the Web

At BoxCast, we take control of the end-to-end video streaming solution for live and on-demand, which allows us to optimize the user experience, quality, and performance of the video capture and playback at each step.

Screenshot of the BoxCast v3 Video Player

We think of the streaming video lifecycle to include:

  • Encoding: Hardware/software to take frames from a camera and get them ready to send to cloud.
  • Transferring: Protocols for sending data up to the cloud and dealing with network shenanigans.
  • Producing: Adding video overlays, multiple camera angles, etc.
  • Transcoding: Make it so the video can play on any device on any network.
  • Delivering: Scale the video content distribution so it can reach mass audiences across the world.
  • Presenting: Immerse the viewer in the video playback experience.
  • Monetizing: Advertisements, pay-per-view, etc. as necessary to support your business.

Let’s focus on Presenting the video for a moment.

There have been many tools for playing back video in the web browser over the years. Flash ruled the day for a long time because native browser playback was sketchy at best. Now, in 2016, browsers have come a long way and the playing field looks a little different.

Adaptive Bitrate Streaming With HLS

Unfortunately, sending a high quality, reliable video stream to a large audience isn’t as simple as uploading an MP4 file on your web server and instructing a web page to play it. That wouldn’t work because mobile devices on weak networks ought to see a lower quality video than a PC on a good high speed internet — not to mention that it just doesn’t make sense for live video.

Instead, we utilize HLS (HTTP Live Streaming) for delivering our video. HLS is simple in principle — the video gets broken into a bunch of little chunks, and the server sends a few different playlists to the client suggesting which order to play the chunks on based on the bandwidth the client can support. These are all served over HTTP which allows us to leverage our massive CDN (Content Distribution Network) infrastructure to scale to millions of viewers. HLS rose to popularity because of its native support on iPhones (it was Apple’s invention after all) and is really a solid and straightforward way of offering a robust stream that works on all networks.

iOS and most newer Android devices support HLS playback natively. You tell the web browser to play it with a simple <video> tag, and it happily does. The same follows for newer Safari and Edge web browsers. But this is not true of Chrome, Firefox, and IE.

Traditionally, playing HLS streams (and many other video formats for that matter) in browsers without native support meant utilizing Flash. But Flash playback is wrought with issues (security and performance to start).

Media Source Extensions

Media Source Extensions (MSE) is a newer browser technology available in Chrome, Safari (desktop only), Firefox, and IE11.

It allows JavaScript to dynamically construct media streams that get fed into video elements without support for any particular media format or codec and with good semantics around buffering the feed. This was previously something that only a browser plugin like Flash or Silverlight could do, but now we can live in a plugin free browser world!

In 2015, we saw major streaming providers (YouTube, Netflix, etc.) take a leap and prefer MSE for playback. They were able to justify it because viewers have steadily continued to adopt modern browsers:

Browser HLS Video Playback Support | By Quarter, 2014–2016 | Based on browser stats from

With MSE support now approaching 80% of viewers browsers, we can now write code that runs in the JavaScript engine of the browser to do the work of Flash.

Let’s Get Technical For a Bit

The way MSE works in practice for HLS is pretty neat:

  • We use a JavaScript XMLHttpRequest to fetch a HLS .m3u8 file from the server and parse it.
  • We then fetch the subsequent bandwidth-specific .m3u8 file that matches what we calculate to be reasonable.
  • We parse that to find a list of .ts segments.
  • We fetch several of those segments in order, creating a buffer to allow smooth playback.
  • As those are segments are fetched into the buffer, we transmux them into MP4 fragments.
  • We then feed those fragment into a <video> tag as if it were MP4 video all along.

It seems like a lot to have JavaScript doing, but dishing some of this off to a WebWorker thread allows the browser to achieve remarkably smooth playback with much less CPU impact than a similar Flash runtime. And fortunately, as is often the case these days, there are active open source projects (hls.js, mux.js) that provide implementations of that process. These MSE-based engines are being integrated into other open source HTML5 video players such as Clappr and video.js.

Impressions of the MSE-based Player

It’s been surprising to all of us to watch BoxCast live church streams and sporting events side-by-side on our legacy Flash player vs. our new MSE based player. We couldn’t have hoped for a better outcome. The exact same source video is much smoother and has less of an impact on our computer resources. The new BoxCast v3 player is being slowly rolled out to customers as we speak, and we’ll likely retire the legacy player in the coming months.

Obligatory Note About Our Startup

If you’re an iOS, JavaScript, Go, or C developer in Cleveland with a passion for video streaming or have an interest in UX and visual design, we’re hiring! Check us out!