“Cupid” Episode 1 — Making of a 360 action short
I’d been experimenting with scripted narrative in 360 video for a while.
Close-ups, camera movement, dynamic cutting. These are everyday tools for traditional directors and cinematographers — but are woefully underused in 360 for a variety of different reasons.
Some to do with the fact that people don’t understand the medium fully, some because it’s hard to get right (and oh so easy to get wrong). And some simply to do with the fact that small, highly mobile, low parallax, high quality cameras simply didn’t exist until very recently.
Coming from both a traditional filmmaking and software development environment I had thought and talked a lot about this situation for a while and had come close to implementing it in a full commercial project — e.g. the Blindspot 360 piece for NBC had fast cuts — but we ran out of time on set to do a full action scene.
Then one day the subject came up with Kinson Loo, the CEO of Z CAM.
Turns out he too was curious how far things could go so he challenged me to take their entry level professional camera — the Z CAM S1 — and push the limits as much as possible.
That’s where the story of CUPID starts — and it ends a bucketload of work later after a team of people worked hard to try and create something that would move the needle in a positive direction — and inspire people to try new things with their storytelling.
The entire cast and crew is listed at the end of this article and as ever with any film project — I am indebted to all of them
Many didn’t think it was possible to take a comparatively low-priced 360 camera and get this level of quality in a dynamic piece without breaking the budget however look at these two images. Note that there is no color correction here — these images are single frames straight from the S1 camera and taken through automated stitching software.
The moment we saw these images on set — we were confident we could make a great looking short — but lemme tell you how we did it.
But on the topic of quality — even though I own other VR cameras (some much more expensive than the Z CAM S1— I’m always surprised how frequently I reach for the S1. In my opinion on the day of its release it became the best entry level professional camera on the market — and it could easily produce results to rival cameras that cost 15x or more (why yes Ozo, Jaunt — I’m looking at you here). Given the speed of movement of the industry — the fact that it still holds up today is commendable.
Concepts and Prep
We knew we wanted to do something fast moving. And seeing as how I owned a Defender 90 that looked pretty cool in the desert and there are plenty of dry desert lake beds less than 2 hours drive from Los Angeles we had props and location taken care of.
Of course military 4x4’s might well be cool — but first and foremost we needed a story. Actress/Writer Dominika Juillet who wrote an insightful article about Acting in VR had been working on a modern day Cupid series idea — so we reworked the pilot slightly and started to develop the project from there.
First things first — assembling a kickass team.
Dominika Juillet was cast as “Alex” early on — and we were fortunate to find Kayde MacMullen two days before we went to camera. We did have another “Theo” in mind originally — but Kayde was game to step in when we had a last minute schedule conflict and he kicked ass.
John Hendicott (Aurelia Sound) was a given because he is an immersive audio wizard and we were friends and we’d worked together before on NBC’s Blindspot. Interestingly he also moonlights as a photographer. In fact many of the stills in this article were taken by John.
Ginny Galloway (Fear the Walking Dead) came on board to produce the project.
Luis Flores (SNVR) and I knew each other well and were looking for an excuse to work together beyond geeking out over gear…and Land Rovers and deserts and smoke grenades were all the excuses we needed.
The initial shot called for a smooth rover move — and I knew Nick Malukhin (Spherica) and his rover could deliver so that was a given.
Last — but not least — Arnaud Paris (Sysmic Films) came in highly recommended as a drone pilot and he didn’t disappoint. Like most French nationals Arnaud has a dry sense of humor and doesn’t suffer fools gladly — but fortunately he understands sarcasm 😉
That was the core team of ninjas — see end of post for the full credit list.
(Note the above photo doesn’t show Luis Flores — for those who know Luis this is fairly obvious because he’s so large he would obscure the entire Rover ;) Truth be told he had to leave for another shoot before we took the photo. It does however feature Dmitry Korobov’s first time on a film set. He did great — right up until the bit where he backed up over a Pelicase)
Rehearsals and Blocking
After a bit of debating and scheduling (and staring at the piggybank) we decided we’d do a two and a half day shoot.
Half a day for the morning rover shot — and two days out in the desert.
Day 1 of the desert was the Land Rover shots and the static shots, Day 2 aerials.
We found a perfect location in Downtown LA (very close to the Hyperloop offices) — very easy to steal — but as it happens the morning of the shoot we showed to see up two souped up cars with tinted windows blocking our shot. We were told the condemned building we were gonna shoot next to was a well known hangout for crack dealers…so…we moved 😉
There is a lot of stuff in 360 video for which there are no proven solutions, you can’t just rent it or buy it — in many cases you have to figure it out all on your own — often times building it on the day.
We did get a chance to do a whole day of rehearsals and blocking for the dialogue scene. Since I was intending to cut aggressively and move the camera a lot — the actual dynamics of the character interaction was fairly important to nail down because we wouldn’t have the luxury of doing it over and over again on set.
If I had the time I’d have shot these rehearsals to see how they cut together — but I decided it was more important for it to feel authentic for the actors…and I’d figure out how to cover it on the day.
This is a very common strategy for a traditional shoot but is extremely rare in 360 video — because no-one really knows what “covering” a scene in 360 really means. Regardless it worked well for us.
The gear was sorted out — We had 2 Zcam S1’s (1 primary, 1 for backup) — full Teradek Sphere for setup and a handmade (see I told you) director’s monitor with sunshade. As far as the camera department goes. On the audio side John brought a full set of lav mics aswell as a Sennheiser Ambeo and a bunch of other audio things that I’m sure he’ll eventually detail if he writes his own article.
Shooting — day 1
I had a fair amount of experience with 360 cameras before and I had shot on the S1 quite a lot — I’d heard stories of people having problems with the camera on sets — but personally I didn’t run into any.
The S1/S1 Pro (and the V1 Pro for that matter) are as much like computers as they are like cameras.
This is cutting edge stuff. And the price you pay for that cutting edge is…well…sometimes you get cut.
If you’re not completely comfortable with the difference between DHCP and Static IP addresses and don’t understand basic computer networking — you’d better be hiring a DIT and Camera Operator who does. In fact the DIT/Camera Operator roles blend quite significantly on a 360 set as the camera technology keeps evolving and traditional sets can’t quite keep to the same pace.
In any case — the opening shot was all we had to do on day 1.
We had some great on-the-spot set dressing by Rocky Roggio and the whole setup performed exactly as intended.
We started late (hey yo — that’s what happens when a drug dealer is in your parking spot) but as a dry run for the gear it was a great day.
This shot was MOS but I still needed to see the performances so we could figure out if the timing was gonna work — That’s why you see the Teradek Sphere setup in the photo.
The whole system is supported on Spherica’s custom made Rover. I have worked with Spherica on a number of different projects — and their gear has always delivered.
BTW — It’s very important with these cameras to provide a steady source of power on set.
There are multiple different ways of doing this — but the solution I use is a Kastar battery on an Ansso Battery plate. They can be daisy chained so you can swap batteries without turning the camera off — and they haven’t yet failed me.
For what it’s worth — I had zero problems with the entire shoot. We didn’t lose a single frame.
Day 2
We had two nights booked for the cast/crew at a small hotel near Coyote Dry Lake Bed (we were originally scheduled to shoot at El Mirage but had a last minute change of plans due to some racing that conflicted with our drone shoot)
We decided that we would do all our vehicle shots on the first desert day (as well as all the dialogue shots) — leaving the second day for Aerials and any pickups needed that we missed.
The day got off to an auspicious start. You see the RV we were intending to use for Craft Services and as a Prop for Theo (the male character in the story) — broke down en route. The logic for using the RV in the shot in the desert is that gave the crew something to hide behind (or inside) in every static shot.
So we arrive at the location. And we have no RV. But the indie filmmaking Gods were smiling upon us.
Because the dude who was the caretaker of the dry lake bed happened to own a big-ass gray cargo Van — that we commandeered for the shoot.
We shot our dialogue scenes at Sunrise at Sunset (we just rotated the vehicles to make matching in post easier). The actors had most of the stuff dialed in — but as ever we did a lot of improv on the day. Lines wouldn’t feel right so we’d change them. Blocking might evolve as the actors started to “own” the characters. (For example the shot where the Kayde pulls the gun that Dominika is holding into her chest — didn’t exist before — but it just felt right when he explored it — so we went with it)
Some of the reasons we were able to move so fast in the dialogue scenes were as follows
- I made a conscious decision to only shoot with static shots in the interests of speed
- I knew I was likely to be editing the project also — and I already had an idea of how/where I wanted to cut the action — so that guided the camera placement. (the one shot I wanted to do but didn’t have time to — is a low angle reverse of the landrovers arrival. Super low angle. Behind Kayde’s feet. Ah well — next time)
- Teradek Sphere — There is no way to direct scripted narrative 360 video without a Teradek Sphere. You need high resolution preview to be able to judge performances on a take.
Of course you can try to get high resolution preview via some other solutions (e.g. Ozo, Voysys) but the complexity/price of using those systems is far too high. If you want simplicity/portability and the Teradek Sphere + iPad Pro is ideal.
Once we nailed all the beats for the dramatic scene it was time to hit some high action with the vehicle mounts.
Of course people had mounted cameras on vehicles before but I’d never seen it done in a scripted narrative piece with the cameras focused on speaking talent. One of the big reasons for this is that many production companies still insist on shooting with stereo rigs — and those rigs simply cannot get close enough to talent to read the emotions on the faces correctly. This is an older test but entirely still applies — suffice to say that the ideal distance for close-ups of faces is about 1–1.5 feet. But most stereo rigs have a minimum distance of 3–4 feet….not good enough.
Luis and I came up with this concept. We called it the Lollipop rig.
Fortunately the S1 and S1 Pro both have 3/8” 16 threads on top and bottom and they lend themselves to being very tightly mounted if you anchor them with baby pins.
We test rigged this a few days before so we could move fast. But it was only later that I ended up powdercoating all my car rigging gear. (A decision I would grow to regret in post because shiny silver is harder to remove than powdercoated matte black)
The first position was for the “backward” facing angle from the hood looking through the dash.
Note that the concept of angles is often misunderstood and argued about in 360 video. In my opinion there is an arc of attention where I suspect the viewer is likely to be looking (I refer to this concept a lot in my first editing article). Of course a viewer can look anywhere — but if the pace is fast enough and you know how to edit 360 video correctly — you’ll be able to hold their attention.
This was position 2 — the shot was oriented to face directly towards the talent.
John Hendicott mic’ed up the car in multiple places (hidden from view of camera) and we zoomed around at up to 50mph for about 20 minutes doing take after take.
For all of these shots — as you can see from the finished cut there was dialogue/performance going on so I wanted to monitor it so I could direct Dominika.
So while we don’t have a picture of this — I was laying down in the back of the landrover covered by a Tarp — driving along…at 50mph.
In case you don’t realise this — the Defender 90 is a short bed defender. There’s not much room for a 6 foot 2 dude to lie in the back. Yep. Sacrifice for art 😉
And this was position 3 — the low angle shot right over the wheelwell. There are plenty of 360 video experts who will tell you never to do this shot. But I thought it worked very well.
Pro-tip — if you need to — don’t be afraid to flip the camera around 😉
Oh and in case you are wondering if the gear gets dirty on this kind of shoot.
Day 3
And so we came to the last day.
Aerials were a lot of fun to do — under the professional guidance of Arnaud Paris. Like many of us in the VR/360 world — multiple hats are often worn and Arnaud is no exception — but on this he was our drone pilot.
The eagle eyed amongst you might have noticed that Z CAM S1 in this photo is turned sideways — this is because I already knew before shooting that I was going to rotate the active field of view downwards (so you are looking straight down) and as such I wanted to maximise quality when the drone is close to the talent.
Arnaud and I had done tests flying over cars and stitching was actually hard to pull off cleanly at the Nadir with the camera in it’s normal vertical position.
Even though the gusty winds made steady flying more complicated — it helped a lot with our EG-18X red smoke dispersal.
We only had four of these smoke containers — so the photo you are seeing was taken during one of the three aerial takes we shot of the final scene. You can see Arnaud and myself in the distance. Since we had one container left — and the talent was dressed and made up — we did an impromptu cover art shoot.
And with that — production was wrapped — and we started the (always longer than expected on VR projects) Post Production phase.
Post Production
Our post pipeline followed a fairly standard workflow.
QUICKSTITCH, ROUGH CUT, LOCK CUT, FINE STITCH/ROTO, TITLES + EFFECTS, COLOR GRADE, AMBISONIC MIX, CONFORM
The steps are more or less chronological but they are not always dependent on each other (e.g. Audio Mix starts right after the locked cut — but might not get laid in until the final assembly/conform)
We used a wide variety of tools for this process — and I think it’s important to lay them all out for reference.
Quickstitch — this was all done in Mistika VR just because it was easy and fast to drag and drop everything in and batch encode everything in one step. (but more about Mistika VR in a second)
Rough Cut — Once you have everything quickstitched in a project like this you take every take into your editing system — sync up the external audio and start editing.
There is only one possible tool for editing that pretty much everyone in the 360 video world uses — Adobe Premiere combined with Mettle’s Skybox tools. Easily the most common tool is the Skybox Rotate Sphere plugin (though now that Mettle is owned by Adobe that’s quickly become synonymous with the VR Projection plugin). There are multiple other extremely useful Mettle plugins but for a basic edit all you will ever need to do is to Reorient your Sphere and preview your timeline in your VR headset.
Lock Cut — This step is really about just deciding that you are done tweaking your Rough Cut. Don’t forget that the 360 video post pipeline is much like a tradition film/video post pipeline — except EVERY FRAME IS A VFX SHOT. So yeah — lock your cut and keep it locked.
(No snarky comments from the audio post team 😉)
Fine stitch and Roto
Of course this is where it got reeeeeeally interesting. And it all boiled down to — Wonderstitch, Mistika VR, Mocha VR, After Effects
Wonderstitch is Z CAM made software for automatically stitching 360 video footage. Every Z CAM camera that leaves the factory is calibrated at the factory — and these precise calibrations are used to inform the optical flow stitching algorithms about how to specifically treat footage from your camera. Z CAM got a lot of early kudos for being the first company in the world to harness Facebook Open Source Optical Flow stitching algorithm and productise is as a piece of usable software. It’s amazing to see stuff happen in Wonderstitch automatically with a few clicks.
Wonderstitch started out a little bit basic— but the Z CAM team is constantly refining it — and any updates are (in most cases) usable with all older model cameras — so you can go back and restitch older footage as the algorithm improves.
When it works — it works incredibly well — but it can be extremely slow and processor intensive.
However the are certain patterns that are extremely challenging for Wonderstitch and that’s where Mistika VR comes in. I wasn’t aware Mistika existed until I saw a quick demo at NAB — and I was immediately very impressed with the speed, maturity and ease of use.
Let me restate that so it’s entirely clear
Mistika VR is very very fast and extremely flexible. Speed is very much your friend when you are stitching and re-stitching shots. And you can move the stitchlines as you see fit (using something called Edge Points) so it’s extremely useful on many complicated shots (and with the write setup — fast enough to use on-set)
Look out for a separate use case article on Mistika VR itself.
Overall my approach to post production was to use the best tool for the job. In some cases Wonderstitch produced an amazing result — in other cases Mistika VR was the software of choice.
But no matter what you do of course once you have completed your basic stitches you still need to paint out rigs that are in the way.
In a static shot (e.g. the dialogue shots — that’s fairly easy — you just need to remove the monopod and shadow (if any) and you’re done.
But in a moving shot — well — that’s a whole new kettle of ball games.
And that’s where Mocha VR comes in — Remote Controlled rover removals (such as we had in our opening shot) are beyond painful without using the Planar Tracking that Mocha VR has.
Here’s a sequence of Stitched/Roto’d/Graded clips for reference. (No audio)
Mocha VR has a number of VR oriented tools — the two that I used the most were the remove tool (for drone shadows and the rover above) and the Lens tool which was very helpful for Roto painting in After Effects.
The purpose of this article was not to cover post tools in detail but just to illustrate what kind of tools were used. Often times shots require a combination of a number of different tools (and multiple passes).
Some of the landrover shots in the sequence above required a second pass with changed convergence in Mistika VR, third pass for individual cameras, judicious use of clone brush, keyframed mesh warp etc etc
These shots are still not 100% perfect. Discerning eyes will spot minor errors in places (shadows/reflections etc) — whichever way you look at it — doing fast moving shots correctly is a ton of work.
Titles/Finishing
Closing sequence was a simple scroll using Mettle Project 2D in After Effects.
The very cool custom RVLVLR 360 opening title animation (an homage to the old Channel 4 break-apart logo) — was exquisitely (And actually rather quickly) made by a titles and graphics genius Mark Simpson
John Hendicott created some custom spatial sound design for the logo.
And of course we exchanged no less than 14 complete versions of a full mix before we finally called it done.
Nonetheless John and I are indeed still friends 😉
I ended up making the CUPID title that the Defender drives over by hand. It was a subtle touch and many people might not notice…but I was very happy with how it turned out.
And on that note here’s a complete list of cast and crew:
CAST
Alex — Dominika Juillet
Theo — Kayde MacMullen
PRODUCTION
Director -Nick Bicanic
Producer — Virginia Galloway
Executive Producer — Kinson Loo
Executive Produce — Jason Zhang
Executive Producer — Ricky Berrin
Associate Producer — Dmitry Korobov
Writer — DJ
Rover Operator — Nick Malukhin
Camera operator day 2+3 — Luis Flores
Car rigging — Luis Flores
Drone Pilot — Arnaud Paris
Production Sound — John Hendicott
Production Design — Rocky Roggio
Hair + Makeup — DJ
MakeUp day 1+2 — Sarah Dorian
Wardrobe — Moods of Norway, Allie Sieberg
Swing Grip + EPK — Dmitry Korobov
POST PRODUCTION
Sound Design, Score, Mix — John Hendicott
Audio Post-Production Services — Aurelia Soundworks
Roto artist — Adrian Gonzalez
Colorist — Rocío Del Pino Gutiérrez
Post Production Supervisor — Nick Bicanic
SPECIAL THANKS
Z CAM (www.z-cam.com)
Spherica
SGO
Imagineer Systems
Mettle
Dan Taylor of the San Bernardino Film Office
Last but not least — thanks to Eric Cheng, Chetan Gupta and the team at Oculus Video.
