The Road to Moon River: Creating Jacob Collier’s Iconic Music Video
In the spring of 2019, Light Sail VR collaborated with Jacob Collier to create a stunning visual that would accompany his Grammy-winning rendition of “Moon River”. A behemoth of a song that demanded no less of its music video, Moon River turned out to be a technological challenge and a deeply rewarding project. I was one of the three VFX artists lucky enough to help bring Jacob’s song to life.
Initially Jacob approached us asking if we’d be willing to help create the first minute and a half of his new music video, pitching us the idea for a starfield populated by faces of his fans for the intro. We loved the idea and agreed to push ahead with production. Within a few weeks, hundreds of Jacob’s fans had recorded themselves singing the opening note and sent it in. Because this was footage beyond our creative control, our first major technical challenge was unifying clips from hundreds of different devices, frame rates, and lighting conditions to create a single “look” for the faces in the starfield.
After ingesting and transcoding all of the various media from Jacob’s fans, I manually masked out the fans’ faces frame-by-frame to turn them into floating heads. I then passed ProRes files of each with alpha channels over to my talented friend Rocky Borders, who painstakingly aligned every “head” into a vast 3D starfield he’d created in After Effects for the fans’ videos to live in.
We quickly found that our starfield of fans was more than After Effects was built to handle, even on our state-of-the-art custom built PC rigs. Eventually, because of the sheer number of videos playing back in every frame, Rocky reached a point where After Effects would no longer preview at all. To overcome this, he devised a brilliant system of creating simple shape layers and parenting them (along with the footage) to a single null object for each face. This allowed us to toggle off all the face layers but still see their approximate positions with the shapes, which sped up rendering and previews substantially.
By the time the starfield had been completed, Jacob let us know that he was impressed with it and asked if we had any ideas for the rest of the video. He wanted something that was evocative of his usual style but with a fresh presentation. Inspired by Eric Whitacre’s choral arrangement visuals, our executive producer Robert Watts conceived an idea for an entire city of Jacobs (pluralizing his name became a necessity at this point). We’d see him in every window, door, and even the moon itself, each video playing a different part in the symphonic performance. We all loved the concept and began brainstorming. After a few rounds of back-and-forth on the color palette and with the understanding that Jacob wanted the city to look as authentic as possible, we settled on a visual style that didn’t look like a traditional 3D render. Lightly textured buildings and multiple layers of parallax would give us that analog feel, and Jacob was very happy with it. Jacob’s creative director Danna Takako then pitched an idea to film Jacob performing on a spectrum of seamless paper backgrounds to fill the buildings. This allowed Jacob to expand beyond his traditional filming of himself in his room while adhering to his busy schedule that prevented him from shooting footage elsewhere. All in agreement, we began production of the first verse.
While we offered Jacob some direction for actions in his performance videos, it was Jacob himself that came up with most of the fun ‘easter eggs’ that are sprinkled throughout the piece. Some even required precise placement and timing on our part — at 4:43, you’ll see Jacob hand a guitar to himself from a different video in a different building! Another favorite of ours was Jacob reading a book on harmony, which was a subtle nod to his fans’ notorious passion for his extravagant harmonies.
Because of the technical complexity of creating the city while maintaining Jacob’s performance across many individual buildings, we found it necessary to previsualize. This was done by our creative director Matthew Celia as a fully animated 3D sequence in the Unity engine, complete with singing Jacobs. I then took the sequence and recreated the city in After Effects, using the previs as an interactive reference for building placement, clip sync, and camera blocking.
The buildings themselves were prototyped by Matt and built by myself in Photoshop with alpha channels for the windows and building edges. Because the city was to be “2.5D” with flat layers, no 3D modeling was necessary. The depth and parallax was achieved by placing these layers in a staggered 3D space and only moving the camera. Because we wanted the city to feel like a one-shot, and because we’d be pulling back to show the whole city, it was important that all of this took place in a single, really really really wide After Effects sequence. As you can imagine, this was incredibly resource intensive, but it led to the smooth presentation that we really wanted to deliver for Jacob.
During construction of the city scene, I thought occurred to me: that these performance clips would look stunning in a traditional grid mosaic because of the contrasting background colors and the variety of wardrobe. After prototyping this out and seeing some great results, we decided that the grid was to be the basis for the last third of the song. This being an almost nine minute video, though, there were only so many times you can display a grid before the concept becomes stale. This is where Matt jumped in and ran with a few really interesting ideas. The first was more obvious — grid animations. We played with a lot of these and settled on a few that really struck us as dynamic and visually interesting (such as the two Jacobs singing in front of the grid). The really striking effects came to life when Matt took the burned-in grid videos into Unity and began creating some crazy new effects that really brought the grid to life. Not only an ‘infinite mosaic’ of recursive Jacob grids, but also a 3D rendering of the mosaic itself.
Taking the burned-in full res grids, Matt created a 12K(!) composition within After Effects and projected them onto the six sides of a cube. Since this was now 3D space, he placed a VR camera element inside the cube and was able to change the perspective at will. This created an extremely trippy effect that Jacob was delighted with, and so we pushed ahead to render out the sequence in 2D.
The most unexpected revelation of the project came to us just as we had completed the second verse and were excited to see the final product come together — we still had a minute and a half to fill near the beginning of the song! This is where Jacob jumped in and recorded a series of performances on black backgrounds. After some brainstorming we decided to invoke the grid motif, albeit in a much calmer way, to ease the viewer into the song.
With Jacob’s stamp of approval, I worked with Matt and Rocky to fix dozens of small bugs throughout the whole piece for the sake of perfection. Small inconsistencies, sync errors, and animation glitches were expected with a project of this magnitude. Thankfully each section was built in a modular way that allowed us easy access to the individual parts for quick fixes. Because we were rendering out to frame sequences, we often only rendered out the individual frames that needed fixed and overwrote them in the folder. This technique saved enormous amounts of time and was paramount to our successful construction.
Moon River taught us all, myself especially. It showed the limitations of the software I’ve worked with for years and presented us with the opportunity to overcome those limitations through ingenuity. Without such a dedicated team, none of this would have been possible. Jacob’s extraordinary song drove us to create extraordinary visuals by pushing the bounds of our knowledge to build bigger than we’d ever built before.
As Rocky would so frequently put it: “It’s a lot.”
