TL;DR: Notch’s keyer is a leap forward for real-time chroma keyers. Its innovative GPU techniques give hardware keyers a run for their money at the fraction of the price. In fact — it’s included with the latest Notch update and free to all our paying users.
Notch’s innovations in automated Clean Plate Generation produce fantastic results with almost no setup or tweaking while providing all the features you’d expect: transparency/hair / liquid handling, global & reflection spill suppression, garbage & hold-up mattes and more.
And all of this within less than a millisecond.
A bit of background
While a new chroma keyer has been on our to-do list for a while, COVID-19 and the nature of remote broadcasts, including in-camera AR effects, has brought the requirement front and centre for a lot of our users.
As we sat down to develop a new keyer, we were surprised by the number of pitfalls in existing real-time keying approaches. Specifically:
- The significant effort and tuning required to get any reasonable real-time key. (Especially for image-based keyers and moving camera setups)
- Compromised quality as a result of having to use a ‘picked colour’ key, especially in less than ideal and varying lighting conditions
- The requirement of pre-shot clean plates to deliver high-quality transparency for hair/liquids etc.
- Limited reflectance spill suppression in a number of real-time keyers
- The high cost of hardware keyers and keying systems
Our goal was to deliver an all-weather, flexible keyer that achieved great results with little-to-no effort and outstanding results with a bit of tweaking. A keyer that performed admirably in both pristine studio setups and simple pull-up green screens behind your desk chair.
This has required a number of innovations in the real-time space, all utilising the processing power of the GPU. In this blog post, we’ll take you through them in delicious detail to show how the new chroma keyer that ships with Notch is a game-changing reinvention of a tool everyone
Automated Clean Plate Generation
The first innovation in delivering a real-time key is in Notch’s clean plate generation technology.
Ask any compositor and they will tell you that green screen backdrops are never one flat tone. Even expensive lighting setups have tonal variations and most of us mere mortals have to deal with less than ideal setups. Yet, the majority of real-time keyers still ask you to pick a single colour and then provide a threshold of error from that tone. By doing so, you are instantly degrading the ability to pull out contact shadows or handle transparencies and edges well.
The alternative approach offered by a lot of real-time keyers has been to provide your keyer a clean-plate / image, but this doesn’t work when the camera is moving. And if your lighting conditions changed during a shoot then all bets are off.
Notch now brings a brand new real-time algorithm that automatically generates a clean plate by predicting what the green screen would look like behind the foreground subject.
This prediction allows better handling of uneven light and gives you a far superior starting point to key from and it works in setups with moving cameras. You don’t even have to pick out a key colour, you just advise whether you’re working on a green or blue screen and Notch’s algorithms do the rest.
The devil is in the transparency
With a high-quality clean plate to work from, Notch can now get down to keying. At this point, we’re interested in dividing our image into areas of:
- Pure Solid
- Transparency
- Pure Key
Pure Solid pixels are pretty easy to identify, you simply look at the chroma distance of your pixel from the clean plate and if it’s sufficiently distant we can say a pixel is Pure Solid.
Likewise, Pure Key is pretty easy to identify. If the pixel chroma is very close to the clean plate pixel chroma, then it’s Pure Key.
The real test of a keyer is in the pixels that are left — the Transparency area. These transparent pixels are the edges, hair or liquid in an image. And this is where the second round of Notch’s real-time innovations kick in.
The first thing to note is that a partially transparent pixel is a mix of:
- The green from the background
- The foreground colour of the actual subject
We want to extract just the foreground colour of the subject with an alpha value consistent with the amount of green screen chroma coming through. To understand how Notch does this, we need to go back a step to the Pure Solid identification.
When Notch identified all the pure solid (safe) pixels, it built a palette of trusted colours from this area. These are colours that do not have any green key colour mixed in and hence they are trusted. Now, when Notch hits an area of transparency there is a very good chance that the underlying subject colour at that pixel can also be found in the palette of trusted colours.
Notch searches through the palette (mixing in the clean plate key colour) and finds the colour that best fits the colour found in this transparent pixel. It then uses the palette colour for its RGB value and derives the alpha value on the basis of the amount of green chroma coming through from the clean plate.
By using this clever and robust technique we end up with strong clean colours at the edges of the subject and in liquids etc. It removes the ‘green halo’ you get in basic keyers. It also removes the tiresome work of fiddling with eroding and blurring of masks — a painful task that afflicts many keyers.
Spill Suppression
Lastly, the green light spill on the subject is a well-known challenge. Notch provides both global spill suppression (just a simple tick box) and reflected spill suppression to substantially reduce that obnoxious green tint.
Global spill suppression does a simple sweep to remove green from the pixel colour if it finds the pixel is heavily tinted with green. This is quick, broad and effective.
Suppressing spill reflectance requires a more sophisticated approach. Just like with transparency, Notch uses the trusted palette from the pure solid sections. However, this time we’re doing a search and estimation taking into account various reflectance luminance levels.
Once we’ve found a matching palette colour we give it the same luminance as the reflection and allow you to tint the light that is hitting the pixel. The result is accurate reflectance suppression and relighting.
..and what does it look like in action, you might be thinking? Well — like this:
And that’s when the fun starts
With your high-quality key delivered, the rest of the Notch suite of features open up to you from colour correction to LUTs, grading to 3D scenes, lighting, real-time ray tracing and anything else from the vast toolset included in Notch — like stepping out from the cramped home office and taking to the skies, like Will decided to do on one of our team Zoom calls:
