Image for post
Image for post

Can your activation ditch the green screen?

Test-driving real-time background removal without chroma key

Image for post
Image for post
Results of a quick Google image search for green screen activations — most from this year.
Image for post
Image for post
Dressed to impress. Two publicity photos of (technically) the same thing: movie stars standing in front of backdrops.

Background removal via depth sensor

On Oct. 25th (not quite two weeks ago as I write this), Fast Company reported the end-of-life for Microsoft’s once-groundbreaking depth camera product, the Kinect.

Image for post
Image for post
The image on the left is from The Verge’s iPhone X review, Nov. 2017. The image on the right is from a Kinect user’s YouTube video, May 2012.
Image for post
Image for post

Depth sensor test with a Kinect v2

The depth sensor and the color camera on the Kinect don’t provide information in a corresponding, one-to-one fashion. The depth feed is lower resolution than the color picture and has a different field of view, so you can’t simply lay the depth data over the color data as a mask.

Kinect alternatives

The Kinect v2, introduced in 2013, should be ancient by normal tech standards. That said, to my eye, the “modern-day” market alternatives don’t appear to be stupefyingly more advanced (although they are much smaller, which is nice). Checkout stimulant’s excellent roundup of depth sensors that was updated just this summer.

Background removal via OpenCV (Open Source Computer Vision)

Using code to analyze an image and identify background and foreground objects is potentially a very deep hole to get lost in. There are people training ML systems to do the work, scores of companies building custom software to help driverless cars “see” the road... You can see Nvidia’s recent work on artificially intelligent systems intended as green screen killers here.

Image for post
Image for post
A cheap SDI camera and a Blackmagic card can give you a quality HD video feed.

Test Results

For quick and dirty playing around, the results are promising. The hot pink background I used (a children’s blanket I picked up from a sale rack at Target) has high visibility so you can see exactly where our edges are hitting or missing the mark.

See footage from both the Kinect and Open CV real-time removal tests.

Further Thoughts

If high fidelity is a sticking point for the project, I believe using OpenCV and chroma key together could give very high-quality results:

Image for post
Image for post
An approach that marries and blends chroma-key with an intelligent background removal algorithm could at least allow for more interesting backdrops. How about a purple step-and-repeat?
Image for post
Image for post
Found in Google search, sourced from coolsanddiegosights.com

Planimal Interactive

The semi-official blog for Planimal Interactive.

Christopher Lepkowski

Written by

Technical Director and Partner at Pretty Big Monster, formerly Director at Planimal Interactive. More at PrettyBigMonster.com and planimalinteractive.com.

Planimal Interactive

The semi-official blog for Planimal Interactive. We’re pushing to build unique, creative ways for the public to interact with brands and products. See our work at planimalinteractive.com

Christopher Lepkowski

Written by

Technical Director and Partner at Pretty Big Monster, formerly Director at Planimal Interactive. More at PrettyBigMonster.com and planimalinteractive.com.

Planimal Interactive

The semi-official blog for Planimal Interactive. We’re pushing to build unique, creative ways for the public to interact with brands and products. See our work at planimalinteractive.com