Directing Volumetric Video Part 1: Beware of Depth Sensor Blow Out!
I’m Sean Kean and I’m writing a series of How-To Guides on how to best use Occipital’s Structure Sensor with Looking Glass’s Holoflix iOS App to take volumetric video
I recently had the opportunity to volumetrically film a sculptor, Kosyo Minchev, while he walked me through his studio and showed me his work and processes using the Holoflix App. This was the first time for me in composing a full volumetric shoot that included a talking person, the physical object they’re speaking about, paired with the impromptu camera movements i.e. “directions”. There’s a lot of tips and tricks that I think will be helpful so I’ll step through some of the key points I want you to keep in mind. I’ll cover a bit in this post and expand more on some of these topics in posts to come.
Directing People in Depth
Be aware of “Depth Sensor Blow Out”
With depth cameras, like the Structure Sensor, you can visibly see when the subject is too close to the camera or too far away by the way the 3D picture holds together, or doesn’t hold together. Too close, and the image breaks up and disappears. Too far, and the subject will get stuck on the last layer essentially becoming flat. It’s important that both you and your subject become familiar with how to avoid a blow out. To see how this might present itself, have a look at the following images for comparison:
Learn how to identify blow out, and how to avoid it — essentially just step back!
We can see that the top image is completely decimated! What is pictured here is a worst case scenario — get used to identifying this while shooting so you don’t end up realizing after your shoot that all the video you shot is useless. Learn how to identify blow out, and how to avoid it — essentially just step back!
The top image, the most broken up, is from holding the sensor about a foot away from the subject. Do not do this! What’s actually happening is here is that the Structure Sensor projects out an invisible IR pattern that you can’t see easily. If you are too close the pattern gets projected too bright and no depth information can be identified — like a flashlight blowing out all the detail on a face in the dark. If you look closer at the top image in this figure you can actually see a photographic/video image of the subject that looks alright underneath/behind the textured 3D mesh video. This means that the photographic image is being recorded — but because you are so close there’s no 3D information being captured to act as a scaffolding so there is no structure to lay the photographic image on top of.
If we work our way clockwise to the smaller images you can see that it takes some distance in order to get a proper image without blow out. At two feet away its still too much loss. At about 3 feet there are some speckles lost around the edges or the face and most notably around/under the nose. I don’t mind shooting with this much loss sometimes because as a trade off you can create an interesting composition and really have the subject pop.
At the distance of the photograph in the lower left corner we are at about 5 feet away and there’s no break up of the image. Just be careful you don’t move too far back from the subject — as you will end up with very little 3D imagery. For example if you point the camera at a city skyline — you aren’t going to get a 3D picture because there’s no way the depth sensor can project that far. Additionally it would just end up on the back plane of the volume — which can actually look fine as a backdrop — its just not going to pop out.
You can always touch the screen in Holoflix and drag your finger back and forth to see how well the scene rotates in 3D as was shown in the previous post (WiFi or Bust!). This is also a good way to tell if you have blow out or not — the photographic image will appear completely flat when wiggled back and forth if the depth sensor has blown out. If it was shot correctly all photographic imagery should be mapped on to a 3D structure — not appearing flat at all.
Depth Communication
One thing to keep in mind when filming in depth is that the subject usually has little idea about what you are seeing on the device or intend to see as the final result — this is all too new! For example, what do you do if you find yourself in the situation where you are static but the subject is moving towards you and getting too close? This happened a couple times walking around the sculptor’s studio. If your subject is getting too close you need to communicate — you can audibly tell them to stop while shooting, stick your hand out with a stop gesture towards them, or stop shooting and explain how the recording breaks when they get too close. You may find that this communicating part isn’t so easy — again all of this is very new. Handing the subject the camera and having them point it at you is the only sure fire way of getting the subject to understand how the camera works and why its bad if they get too close. Depth sensor blow out! It could happen to anyone.
Stay tuned for more trips and tricks — up next — getting depthy compositions!
Is there anything you’re dying to know about Holoflix and volumetric film making? Write me at dragon@lookingglassfactory.com
— Sean
