Troubleshooting Technical Challenges in 360VR Post-Production: Part III
Maria Fernanda Lauret trouble-shoots the most common technical challenges you might come across when adding animations and illustrations to your 360/VR experience.
A number of illustrators and animators in this industry started off working in the traditional format, in which the animation and visual effects are gathered into a frame, and they don’t need to worry about what ends up outside the canvas to make the magic happen. In 360/VR, however, the scene is a bit different. A lot of freelance animators we’ve hired to work with us were creating animations and illustrations for 360° space for the very first time. “Animating for VR was a bit new to me. Movement over distances was a challenge that I faced. Animating on a flat screen is something else, and its implementation on VR is a whole other feel,” says animator and illustrator, Tasneem Amiruddin, who worked with us on I Am Rohingya and Dreaming in Za’atari.
In 360 video and virtual reality, there is no longer a frame you can focus on, so it can be much trickier and challenging to come up with art that suits the environment and enhances the user’s experience. Hugo Hache, who put together the illustrated environment that you see on Yemen’s Skies of Terror, told us that “When it comes to the biggest technical challenges for creating animations in a VR environment, it would have to be letting go of some of the logic you would apply in a typical 2D flat animation. Some effects and tools that would normally be the go-to in order to accomplish certain results, are not necessarily the same ones you would use in a VR animation. For example, drop shadows or glows have to be manipulated considering other axises, while on a 2D, you would have side to side (either on the Y axis or the X axis).”
In general, creating animations and illustrated environments for the 360 format can be challenging in terms of finding a balance on the scale, distance and angles for a realistic look, especially when you are used to fitting everything into a flat rectangular frame in linear videos. It can also be tricky to have a fully illustrated environment and animations all around without experiencing any kinds of lines on the back of the scene, given the edge of the flat image you work with throughout the creating process. We compiled some of the technical challenges that we face in this industry — from creating a fully illustrated or animated 360° environment to fixing tedious problems that come up during the post production phase of such projects — and some tips from our team to make sure you know what to do to solve these issues, if you ever run into them.
1. Creating 360 illustrated/animated environments from scratch:
Problem:
Another important thing to note is that understanding perspective is super important for you to create sketches for 360 videos. You should keep in mind that your board is a panoramic, flat surface and that the edges will eventually meet once you place your work in a 360 video edit and watch it in a VR headset. Hugo Hache, who created animations for some of our pieces, explains that “In VR, you have to keep in mind that the whole 360 environment, in other words the ‘ceiling,’ ‘the floor’, ‘the back’ and ‘the front.’ Despite this, you do create the initial VR composition on a 2D canvas that later on will be seen in VR/360, and that is when you notice all the problems regarding the edges of the 2D comp, not translating as expected. Basically, you end up with lines and borders that just don’t match at all, therefore ruining the illusion and overall just looking terrible.”
Solution:
Hache says that “Fortunately, now there has been great advances in this area and we have tools of these sort created exclusively for VR. These help get around “the edge” difficulty fairly easily by comparison, but yet, when coming from the average animation, there is an inevitable learning curve that, as mentioned at the beginning, demands you to let go of some of the logic you are so used to.”
For guidance and inspiration to create 360 fully animated/illustrated environments, Draw Sketches for Virtual Reality Like a Pro, by the AI/VR/AR product designer Volodymyr Kurbatov, is a Medium article that we shared with freelance illustrators and animators who hadn’t worked within the 360 space and had adapted their creations into our projects in the past. Kurbatov recommends this book for those who would like to learn more about designing human dimension and interior spaces. I would also recommend checking out the 360 drawing workflow and techniques by French artist and illustrator JL Mast, who has worked in several projects with Marvel. There, he explains some of the most basic topics you need to grasp in order to create a 360 illustration in the equirectangular form — creating a 6 points perspective grid, and drawing and transferring your work to Photoshop for further adjustments. Also, these VR Sketches Sheets created by Intel’s UX designer Saara Kamppari-Miller could also be helpful when you first start experimenting with 360 illustrated environments.
2. Issues Adding Flat Illustrated/Animated Elements:
PROBLEM:
Usually, I create motion graphic elements, animations or illustrations separately, and then put everything together in 360 using VR effects on both After Effects and Premiere, so that these elements won’t look curved and deformed on the 360 environment. However, it’s not as simple as it sounds. Very small details throughout the process can make similar elements created on After Effects look different from each other in Premiere in terms of scale, even if you are using the exact same values on the immersive VR effects included in the latest versions of Premiere Pro.
SOLUTION:
When all you have to do is add some animated 2D elements to your video (such as lower thirds, location cards, subtitles etc), bringing the flat file into Premiere and adding VR effects is the easiest way to go. But more often than not, you have several motion graphics, illustrations or animations to be placed, so the best workflow is to create and place all your animations/illustrations the way you want them to look in After Effects, and then integrate your final render into the cut in Premiere. However, if you choose to bring a lot of flat 2D elements (such as linear videos, photos, background filters, animated arrows) into Premiere, you might face some [very annoying] issues with scale and software glitches when you try to adjust the pace that things appear on screen. For example, sometimes you adjust the VR Plane to Sphere effect settings and when you click anywhere else afterwards, the numbers you chose to adjust the different source and projection axes suddenly and inexplicably go back to the numbers you had before. Oh, the stress!
This is a software glitch and I haven’t found a solution yet, but when it happens multiple times, right after I set the values that look good, I quickly copy the effect before clicking anywhere else in my cut. If the values go away, I delete the effect and paste it back.
Also, when applying the VR Plane to Sphere effect on 2D elements in Premiere, the size of the canvas you rendered in After Effects matters — if you want to get a good look and quality of your image when you apply the effect. Let’s say you are creating chat boxes in After Effects to create a messenger-app look on Premiere. To do that, you need to make sure that the canvas size of all the boxes you are rendering are exactly the same; otherwise, their scales will look different.
That is why at the end of the day, it is better to place all the elements the way you want in After Effects and render them as a one-piece that will be brought into Premiere later.
3. Unwanted lines on the back of the 360 image:
PROBLEM:
We have faced issues with unwanted lines on the back of some of 360 videos quite a few times, and we’ve identified some reasons as to why this happens. Often times, when using the VR Plane to Sphere effect in Adobe Premiere to place 2D animated elements on the back end of your project, a line might appear if you are using additional effects such as drop shadow. Resizing a 360° environment is also problematic; scaling a smaller 360 image to frame size in order to fit the rest of the piece will potentially cause headaches with your export. In addition, we have noticed that sometimes rendering a sequence before exporting — and mostly when rendering layers of live footage and animations/graphics — has caused the back line to appear in the final export.
Lastly, there are other cases in which these lines might appear, which are not related to the use of graphics, but they are worth mentioning. In some cases, if you create a separate png mask for your tripod removal and in Premiere, you end up having to readjust the orientation of your shot, that can also cause similar issues.
SOLUTION:
A trick we figured out to get rid of the back line — while applying both VR Plane to Sphere and drop shadow — was to keep the drop shadow first, and then place the VR effect under it in the effect controls tab.
When it comes to dealing with different resolutions in the same timeline, make sure that the specs of the 360 illustration/animation go with the resolution of the full piece, to avoid having to fit to composition size later. So if you’re working with a 4096x2048 sequence, for example, and one of the shots is 3840x1920, it is worth trying to precompose the shot after the tripod removal is done in After Effects and make the original composition 4096x2048 before rendering. Attention: do not change the canvas size before removing tripod because that will mess up with your composition. Another way to go would be to bring the tripod removed 3840x1920 scene into Premiere, rescale it to 4096x2048, export it and bring it back to your video edit. Finally, to solve the issue of adjusting the orientation of both the shot and the tripod mask, it is often better to nest both layers and apply VR Rotate Sphere to the nested sequence instead of applying it on the separate files.
4. Nesting and applying effects to a lot of .png+alpha files
PROBLEM:
One of the worst technical nightmares I’ve faced in 360 post production was dealing with a lot of PNG layers in one Premiere Pro sequence. During the post-production of We Shall Have Peace, because I wanted to have more freedom to control the pace of the voice-over, videos, photos and graphics appearing on screen, I chose to create each separate asset on Photoshop and After Effects, and then bring everything into Premiere. Turns out that stacking too many PNG+Alpha files with VR transitions applied is a terrible idea. At some point, I needed to make a quick fix to get rid of a glitch in one of the graphic sequences that I nested, but because the project was so heavy, it would not reopen.
For those of you who are not familiar with the term, nesting a sequence is a compositing technique that works as a subsequence that gathers a group of files that you’d like to see grouped in your master sequence, for the sake of organization and practicality. So going back to my issue, I tried everything I could possibly think of: cleaned the disk cache, created a new project file and imported the final sequence with the full documentary, updated my Premiere version, tried to open the project in other computers, but nothing happened. I asked many other professional editors and nobody was able to help me. Premiere would crash every single time.
SOLUTION:
That was a difficult one! After trying to open the nested sequence in different computers, I finally managed to open it, only to find that I had to wait another hour to be able to click on any file without crashing Premiere again. I had applied VR transition effects after all the layers for them to fade out nicely. It turns out that this was causing the entire sequence to crash. At that point, after carefully removing effect by effect, I found the problematic one. I deleted the file, brought it back to the project and was able to carry on with editing.
—
It is truly rewarding when you are able to effectively recreate a space that brings viewers into a new world in VR, or potentially enhance their experience by adding graphics to the 360 space. Despite all the technical challenges you might face in post production, watching something you created on the headset and understanding how much this artistic touch contributes to the storytelling component of the piece is priceless. If you’d like to learn more about how and why we chose to use animations and graphics to make some of Contrast’s documentaries more impactful, make sure to check out my blog post “The Use of Illustrations, Animations and Graphics in 360° Videos”.
