Chroma keying as a product design tool

Chroma keying can be used for more than post-decorating royalties.

This article was written by Victor Johansson & Dhruvee Tyagi. Editing was done by Renee Semko Gonzalez.

Face it, time is never going to be your friend during the design process. We somehow always feel like we could use just a little more of it. We can’t magically design something to slow downtime, but we can experiment with tricks and tools that help speed up certain elements of the design process that drag on. While tinkering with our design process isn’t new to Above (see our colleague Arvind Sanjeev’s write-ups on using machine learning and video for prototyping), we recently experimented with Chroma keying as a way to improve the visual presentation of UX concepts to clients.

Chroma keying, also commonly referred to as greenscreen or bluescreen, is the process of filming something in front of a monochromatic background, then removing that background in post-production to be replaced with the content of your choosing. The TV and film industry have perfected this technique over the decades and creative software like Adobe After Effects have brought Chroma keying to the masses, spurring plenty of popular memes. It’s possible to use basically any contrasting color you’d like, but green and blue are historically preferred because their hue was distinctly different from the human subjects. Accordingly, racial bias in both analog and digital photography solutions is something to be mindful of when using Chroma keying as a product design tool.

What we needed

What led us to experiment with Chroma keying? For starters, like everyone else in 2020, the pandemic forced us to get creative in how we accomplished our work remotely. One of the tricky areas we encountered with remote work was exploring and/or presenting multiple versions of an on-product UI in an efficient and informative manner. We needed a tool that would give us high fidelity results in relatively little time that we could share with our clients and potential stakeholders.

Beyond remote solutions, we felt tired of lifeless visualizations of products that usually come in three variants:

1. Static renders with arrows and callouts that explain what various interactions do.

2. Animated UI’s where the UI moves around by itself as if being moved by a ghost hand.

3. Animated UI’s where translucent dots representing disembodied fingertips hover over the product.

Surely there was a better way to capture and visualize this imperfect reality. We wanted the tool to do two things:

1. Validation of interactions and gestures, do they feel natural? Is the hand covering vital information during interaction? Is the UI the right size or is it too small/large?

2. We also had a need to produce dynamic content that fit early on in our design process, where neither product nor experience is yet defined.

Chroma keying could help us with this level of adaptability and scalability. Even after filming, we needed to be able to use and alter the content several times. In this sense, the process became more about prototyping than visualization.

Following the 80/20 rule, our goal with Chroma keying was to establish a quick and good workflow — not perfection in terms of visual quality. Ultimately, we wanted it to be a tool to help us explore, not just produce visuals.

The process really shines when it comes to exploring variations in product UI’s

What we got

After some experimenting, we ended up with a process that allows us to go from an abstract idea to concrete video prototypes that can be shown to potential stakeholders or users within a few hours. No explainer dots, ghost hands, or arrows needed. A step by step guide of our process is detailed below, but here are some key takeaways from our explorations:

• The process allows us to quickly adjust the concept to a large degree, such as completely switching out visual appearance. The shape and size of the product as well as the UI can also be swapped out within certain boundaries.
• Forcing ourselves to record every gesture semi-validates the interaction. Even though this is not as good as building a full prototype for each concept, it’s preferable to working with gestures on a screen only.
• The addition of real hands when presenting product UI’s is definitely valuable to clients and users, especially when presenting remotely.
• The adaptability that Chroma keying affords you is particularly valuable. It allowed us to continue developing our concepts even after the shoot.
• As with most design processes using the 80/20 rule, we were cutting a lot of corners but that made for a very speedy process.

Step by step process

This is not a guide on how to get the most out of using After Effects. Rather, it is a breakdown of how we explored our way to a result that created a lot of value for our team in just a few hours.

We ended up with a process with four key steps:
1. Preparation
2. Shooting
3. Rendering
4. Editing

1. Preparation

For the mockup that we use in the video, we only print a part of our model. You ideally want something that is simple enough to be easily removable in post-production but true enough to the 3D model that you can align it properly for the background render. You can, of course, use any technique you have available to build the mockup but the more true it is to the CAD the better. If you want to be able to use this model for adding shadows on the final result, the surface quality matters also. If it’s too coarse, this might show up in the end.

Here we see our mockup glued to a tripod and covered in blue spray paint. And with some reinforcements to stabilize it.

Painting & Setup
Ideally, the color used for the model matches that of the background perfectly. But, prototyping is never perfect. In the image above, you can see the three distinct shades of blue we ended up using (Dark blue background, light blue spray paint, and a medium blue electrical tape to cover up some last-minute scratches in the paint)
If you are going for Hollywood quality, this might look like a disaster, but if you are going for quick prototypes this is just fine.

Another thing to think about is stability, the mockup needs to be stable so it doesn’t move when interacted with. In the image above we used an old IKEA lamp that we sprayed blue and to this we added some crossbars and sandbags to help stabilize it (if you don’t want to piss off your engineering colleagues I do not recommend using their fancy power supplies as sandbags).

2. Shooting

Our ideal scenario included only one camera angle so that all the footage could be interchangeable. This took some trial and error to find an angle that covered everything we wanted. After you have your angle set, just let the camera roll. Taking a tip from the pros, using a clapperboard is a fantastic time-saver considering that all the files are going to come out of the camera looking identical (Our version of a clapperboard was an A4 paper, however, so not much clapping going on there).

This is what a typical raw video file looks like.

3. Rendering

After the video is shot, the next step is to render out the product that will be replacing the placeholder 3D object from the video.

We used a still frame from the raw video to match the perspective of the render in Keyshot. A good tip is to follow the same settings as your camera, specifically the aspect ratio and focal length. This will help in having an accurate and realistic perspective on your render.

Using screenshots from the video it’s easy to set up the render-scene in Keyshot to match the video.

4. Editing

With the render being the last piece of the puzzle, it’s now time to start editing. Our process followed the below steps:

1. Create a new composition from a raw video clip.
2. Duplicate the clip.
3. On the top of the two layers, apply the “Keylight” effect. Use the eyedropper to select the color you want to remove. Use the gain setting to get the desired result. Ideally, you end up with a completely flat grey background without parts of the hand disappearing.

Raw video to the left and Keylight effect applied (without gain adjustment) to the right.

4. Apply the “Simple Choker” effect to the same clip. This trims the edges similar to the “defringe” tool in Photoshop.

Before and after the simple Choker effect is applied.

5. On the bottom layer mask the area where you want shadows to appear. Desaturate the layer and set the blend mode to overlay.

Here we have masked the flat front-face of our model so that we can use the shadows from the footage in our final composition.

6. Add the background render in place below the two video layers.
And that’s pretty much it. Now you have a video file where you can switch out the product render in a few simple steps while keeping the same video clip in the foreground.

Give Chroma keying a shot and let us know how it goes. And if you also have any tricks up your sleeves, feel free to share with us!

Join our team, we’re hiring! www.above.se/joinus