3D Audio for a 360 ‘Behind the Scenes’ Video
A breakdown of recording, mixing, sound-designing and scoring the BTS video of NBC’s TV show ‘Blindspot’
In this article I thought I’d break down the process of recording, mixing, sound designing and scoring a 360 behind the scenes video. The TV show in question was NBC’s ‘Blindspot’. Later on in the article you’ll find a video showing a breakdown of all the audio sources used in it’s creation, but first, here’s the finished piece:
Capture
From start to finish this job was great fun, and as director Nick Bicanic mentioned in his article; ‘how to shoot/direct 360 video alongside a network TV show’, it required careful preparation and fast execution on the day:
I came to set with the Sennheiser Ambeo microphone (which at that stage was still in Beta), a Sony R26 2 Channel Recorder, a Zoom H2n recorder, a Zoom F8 recorder and bunch of radio mics.
The Ambeo was used in almost all of the scenes and can be seen attached to the monopod using a Dinkum Systems clamp. I am now using a combination of Manfrotto Super Clamps, as they offer a stronger grip and enable the mic to be positioned closer to the monopod, thus reducing the area to be roto’d.
The Zoom F8 was chosen because it’s a cost effective solution to multichannel field recording. It has 8 inputs which can be ganged for ambisonic recording (enabling you to control the input level of 4 channels simultaneously) and also features Bluetooth control, letting you monitor levels and start/stop recording remotely, so you can attempt to exit frame. The preamps are surprisingly good on this thing, too. For more detailed work, I’d turn to a Sound Devices recorder, but for this it worked just fine.
To help the post production process it’s not only essential to announce the scene number and take, but also the orientation of the ambisonic mic, as the mic can be positioned with the capsules up, down, or end firing. Without this information the post sound mixer can only make an educated guess as to how to position the ambisonic sound field to ppicture. Therefore, I might announce something like: “Scene 4, take 2, Ambeo is upright, facing the blue door”. This gives me everything I need to know to orientate the 3D sound field in post.
In addition to the Ambeo, I used a combination of cheaper sound recorders where there was the possibility of equipment getting damaged. In particular, these were the door detonation and motorbike launch scenes. Although recording in stereo, this gave me perfectly useable audio for these short scenes, when used in conjunction with some ambsonic wildtrack.
For the interviews, I used a combination of lav mics and Ambeo to achieve a good blend of direct dialogue and ambient sound, which we all know is essential to the sense of immersion.
Post Production
Moving into post, I jumped into action once Nick delivered his final cut as a video and OMF file. At Aurelia, we use a combination of software solutions depending on the project. For this, we used Reaper and the excellent Blue Ripple tools, to mix in higher order ambisonics.
As previously mentioned, we had received the production sound files, meaning we could use the lav mics used in the final TV production (which were all recorded superbly). Once these were synced and mixed with the ambisonic files, we were ready to dive into sound design.
To demonstrate individual sound elements, we’ve uploaded a video of The Sennheiser Ambeo mic*, SFX and music all separated, so you can hear what’s going on. Check it out below on a Chrome browser and don’t forget to use headphones!
*Martin’s interview was recorded using an Ambeo mic, however, the fight scene after it was recorded by an external sound engineer using a quad mic array, which unfortunately isn’t spatialising as we’d like.
Sound Design
It’s up for debate how extreme you go with sound design and sound effects for a ‘behind the scenes’ (or ‘inside the scenes’ as Nick says) as the aim is to capture the feeling of being there and experiencing what the crew and cast are doing. However, without the impactful body punches, head slams and gunshots the piece will always feel flat, so we went about adding all of those elements to add some excitement.
Music
After the sound FX were in place I then looked at music score. This was designed to fit in with the TV series theme music used at the beginning and end of the video. Without going into too much detail here, the aim was to maintain a sense of energy and rhythm, whilst also creating space for dialogue, when needed. It was also crucial to emphasise the most important moments of action — if you listen to the demo below, you’ll notice most of the main beats of the music land on punches or serve to accent the beginning or end of the scene.
Below is a screenshot of the Blindspot Reaper session (click on the picture to enlarge). Score is in green at the top, then interviews in blue and pink, a heap of SFX in grey that are minimised and then further down in yellow and orange are the Ambeo and quad mic sources. You can also see the video, to which we added a 360 degree grid, which helps us spatialise the sound accurately. You can download it here to use in your projects if you like.
Final delivery was for YouTube, which at the time of writing only supports 1st order ambisonics. Therefore, we exported our mix in the 4 channel AmbiX format and attached to the video using the handy iffmpeg app.
The Near Future
As we all know, VR technology is moving super fast; from capture systems and post production workflows, to publishing platforms and delivery requirements. Because of this, we’ve decided to upload an updated version of the Blindspot video to Facebook, which now supports 2nd order ambisonics and fixed stereo. A little more about that, here:
An essential new development in sound delivery for 360 video, is the ability to split the diegetic 3D sound field, with non-diegetic fixed stereo track. This essentially means that for everything that’s visibly making a sound in the scene, we mix an ambisonic 3D sound field, which responds to the viewer’s head movement. For everything else (in this case, the music score) we render a stereo track which is fixed to your head position. This gives the feeling of the music being ‘in your head’, rather than ‘in the scene’ and really helps the viewing experience.
If you listen again to the demo video above, notice that unlike the Ambeo and SFX sources, the music does not rotate in your ears. This is because we want the music to sound internal, and not in the environment. This is one way that spatial sound technology is helping us tell more convincing stories in 360 video and VR.
Many thanks to the team at NBC, and to Nick Bicanic for making such a fun shoot also look awesome!
+++++++++++++++
Get fresh insights, news and analysis with CinematicVR’s newsletter, delivered to your inbox with l♥ve, every 2 weeks. Sign up now! ++++++++++++++++
