Docler Engineering
Published in

Docler Engineering

Manipulating video in a browser

Have you ever wanted to modify the video streaming from your webcam?

Photo by Tim Mossholder on Unsplash

Now it’s possible!

A new browser API recently landed in the webkit based browsers, it’s called “Insertable Streams for MediaStreamTrack API”. Its superpower is to tap video streams, and allow the application to modify it frame-by-frame, without the need to even display the video on screen.

Before you start experimenting, check out your browser’s support: https://caniuse.com/?search=MediaStreamTrackProcessor

Let’s see a simple, but practical example. We will create a green screen effect ( https://en.wikipedia.org/wiki/Chroma_key ) which means we will swap the green (or blue) background to a certain background image on each frame of the video stream.

The original image
The modified image

Some real world use cases:

Transforming a video stream could be used to extend or modify the stream before sending it further — e.g.: WebRTC.
Possible operations could be:
- cropping or rotating,
- highlighting elements,
- adding various texts or graphical elements to the stream

Additionally, you can also “just” analyze the stream using TrackProcessor to detect something on the frames.
You can also use TrackGenerator to produce a video stream from computer generated content.

Show me the code!

The data flow

Before we dig into the details, check out how simply and elegantly we can use the corresponding browser API.

The MediaStreamTrackProcessor delivers the frames in the form of a “readable-stream”, after which it’s piped through our custom transformer function. The result is then consumed by the MediaStreamTrackGenerator API, which then creates a standard media stream.

The transformer

But let’s dig deeper a little bit! I’ll show you how to implement a simple green screen effect.

⚠ This example is for demo purposes only. It’s not optimized for high performance!

The transformer function looks like this:

First of all, we have to get the pixel data from the video frames. This can be achieved by using the “copyTo” method. By investigating the video frames, we will see that the format streaming from the webcam is not RGBA, but rather a “I420” format. After a quick investigation, we find that it’s a YUV420 format (https://en.wikipedia.org/wiki/YUV)

Some more explanation may be needed for the result of the “copyTo” method. This provides metadata about the data-buffer structure. In our case, it looks like this:

The YUV format consists of 3 consecutive segments for the Y, U, and V channels in the buffer. The “offset” property outlines the start address of each segment, and the “stride” property informs us of the number of bytes in each row.

An interesting fact: in spite of the RGBA format, where each pixel have a Red, Green, Blue, and an Alpha (opacity) channel, stored in 1–1 separate byte in the buffer, the YUV420 format stores the Y luminance channel as 1 byte per pixel, but both of the chromaticity channels U and V have 1–1 byte for a 2*2 pixel area, halving the (color) resolution in both directions.

Just for fun and experience we will transform the frames to RGBA, then use that format for the rest of the process.

But let’s get back on track and check the transform operations! The green screen effect is a simple condition: when the pixel’s green component is dominant ( G > 0.6 * (R + B) ) it will be swapped with the pixel from the background image.

Some thoughts about the performance…

It’s not a surprise that CPU based video processing is very performance heavy, and the results of the experiment supports this claim as well.

On a higher performance CPU, I had the following frame processing times:
640*360 pixel frames: 4–5 ms
1280*720
pixel frames: ~20 ms
This means that the processing time is near to the available 33 ms frame time at 30fps video stream with a far simple operation. We can easily run out of frame-time!

Next steps

In the next article ( Video manipulation with WebAssembly | by Szabolcs Damján | Jan, 2022 | Medium ), we will port the transformer function to WebAssembly to compare its performance!

--

--

--

Get insights from Docler Engineering teams on how we are solving different challenges in live streaming, software architecture, infrastructure and technical management.

Recommended from Medium

Sol Decoder presents: Seamless

Messing around with Perkeep (part 1)

I Solved Two Global Problems in Two Weeks

Alvin’s Dew Drop Daily — Issue #10

Why Serverless is the right tool for your edge application infrastructure

Coding Bootcamp Week 10/13: Building the Frontend and Why I Found it Hard

Android Vulnerability Assessment and Penetration Testing — Part 2

The “Clash of Styles” Series — FP vs. OOP as a Daily Choice

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Szabolcs Damján

Szabolcs Damján

More from Medium

Real-time WebGL video manipulation

Camera orbit, zoom-in, zoom-out movements in WebGL

Ready, set, Babylon.js!

WebGPU January 2022 meetup — takeaways