Creating Hulu VR
Julian Eggebrecht • Mar 24, 2016
Hulu is passionate about the art of great storytelling and evolving that art of storytelling in TV and movies. Hulu sits right at the intersection of entertainment and technology, so it was only natural to explore how Hulu could bring the exciting virtual reality platform to viewers.
Two years ago a new Hulu division in Marin County, California was founded by a group of video game developers and hardware specialists that have created the Star Wars: Rogue Squadron franchise, the 16-bit classic Turrican games, and collaborated with Nintendo and Sony on the GameCube, Wii, and PS3 platform designs.
For Hulu VR we deployed CDP not onto a real device sitting in a real living room, but onto a virtual TV located in the Unity game. This requires CDP to be running as a multi-threaded application in parallel to Unity, so that both can exchange their video output and control inputs. The CDP-rendered Hulu UI and video playback is displayed on a polygonal surface in the Unity game environment.
Using a finished game as the basis for our virtual world allows us to use typical game features that are important for immersion, not only presenting a static viewing environment as other apps do. A realtime renderer is used to have dynamic volumetric lighting, dynamic shadows, as well as animated objects. To top it off, we employ real-time physics-simulations to give the user Easter Eggs to discover and play around with.
In addition to playing our existing library in a multitude of virtual environments, the Hulu VR app also takes the plunge into full 360-degree film and video content that can only be really appreciated in a virtual reality headset. This presented the challenge of streaming full 360 degree videos, some even running at 60 frames-per-second, and being fully stereoscopic with individual frames for each eye. We knew from the outset that the videos would have to have a resolution close to 4K, but we also wanted to keep the bitrates for streaming somewhat manageable, considering our first device would be the mobile Samsung GearVR.
Two solutions were chosen to solve these problems: One being the first use by Hulu of the new HEVC video codec, which compresses video roughly twice as efficiently while maintaining the same quality as the older H.264 standard. The other being an optimization of the video data. Raw video material for VR is usually presented in the equirectangular format, which was used to unwrap the spherical earth onto the flat, rectangular surface of a map. We developed a new video conversion pipeline for deployment onto our servers, creating six images per eye in a special layout that then gets compressed. In the app itself, the viewer sits in the center of a virtual cube, and as each side of the cube is streamed in, the six images are projected into 3D space as cube maps which gives a perfect illusion of every pixel having an equal distance from the viewer.
Filmmakers say that audio is 50 percent of the viewing experience, and in VR the audio has to be dynamically placed around the listener depending on their head position. We chose multi-channel AAC as our streaming format, and use an advanced spatializer solution to position the audio correctly around the listener depending on their head position.
For the GearVR, we also took platform-specific steps to make our UI and the video content look as good as possible. Using an Oculus-specific feature called “Overlay”, we sidestep some of the resampling being applied on the 3D scene to compensate correctly for the lenses of the headset. Using the Overlay feature, we are able to stream and display Hulu’s regular library in our virtual environments in 720p HD resolution, and make monoscopic VR videos looking much more refined.
Near the end of the development cycle for the first version of the Hulu VR app, two issues needed resolution — performance and heat. To render the UI at speed, as the viewer is panning across the room and gets close to the Hulu UI, a custom depth-of-field shader coupled with a post-processing pass blurs the screen. While that happens the room is being rendered as a simple image and blurred — once this imposter is in place and the user looks at the UI, the 3D world behind it is switched off. Other tricks to gain performance involve dynamic shadows, which are only updated in areas the user looks at, and only when objects animate or are being manipulated by the user. Post-processing for HDR is only applied on certain parts of the 3D and we switch off Anti-Aliasing during VR playback to reduce system bus traffic. We also take full control of the CPU and GPU clock rates to run the CPU as low as possible to keep the heat down. When the user scrolls within the UI, or starts playing with physics-driven objects, we throttle the CPU up to provide enough power to sustain good frame rates.
With the first version of the Hulu VR app finished, we plan to add new VR shorts, films, and fantastic environments with new gadgets on a regular basis. The advantage of a cloud-based service is that shipping the first version of an app is only the foundation, and new content can be added at any time going forward. In the case of Hulu VR, this will mean new 2D shows, originals, movies, and trailers. We will add new 3D environments to watch this content in, and most excitingly more and more 360 degree VR filmed content from a wide variety of creators.
For now, we hope you enjoy this peek into the future — and we will get back to work on the next evolution of Hulu VR — which we are also hiring for at Hulu Marin, so please join the team!
Originally published at tech.hulu.com on March 24, 2016.