Unity, Video and Mobile VR
As the person producing Udacity’s 360 and Immersive Media concentration for the VR NanoDegree, one of the things I spent some time looking into are the different, current options for displaying a 360 video, in Unity, in VR. I will be covering a common developer workflow, where you are trying to do something you may never have done before and aren’t really sure whether the engine you’re using can do it, or if there are third party solutions out there. You have limited time to figure this out, so this is my process, at least, for solving new problems.
As a VR developer, at times you will be tasked with achieving something specific and new to you which may require you to write custom code, or find 3rd party options to solve your problem. This is something which is quite common in development, and while some of the time the solution will be to write your own code, there are often times when a 3rd party solution can help you get the same thing done much faster.
When you’re doing this kind of research, there are a few things to bear in mind.
- Don’t work on the master branch of your code. This is experimental, things will break, things will need to be changed, the last thing you want is for something to break permanently. Version control is important!
- Figure out what your exact priorities and goals are. What problem are you looking to solve? What’s your budget? What are your constraints? Clearly defining the problem will help in the long run.
- Timebox — research should be limited to a specific amount of time. This is not open ended. Give yourself a deadline by which you will decide on the best solution.
So what were the priorities, goals and constraints for this project?
- Add 360 video to Unity, to enable students to make interactive videos
- Must be viewable using Google Cardboard
- Must be free (no paid 3rd party plugins)
- Must be relatively easy to use and work for both Android and iPhone
Option 1: Can Unity do this natively?
The first option to look at in this case was “Can I do this in Unity without any 3rd party software.” In this case, the answer is “well maybe.” In Unity 5.4, video textures are supported with the use of MovieTexture. Given that I already have my 360 video, is it as simple as putting that as a texture on a sphere, and then putting the camera in the center of the sphere? There are a few things to consider here — if I put the camera in the sphere, the normals will be facing the wrong direction.
Normals are the property of a vertex that tells a renderer which way light should reflect from it. When you create a 360 sphere, those normals point outwards — so that you can see the sphere. If you move the camera inside it, you won’t see anything, including your beautiful 360 movie texture. Still, this is a fairly simple problem to solve. This option is free, so there’s no problem there. But the key point here is my constraint that I be able to view this using Google Cardboard. MovieTextures do not work on mobile devices, which means that they also do not work on Cardboard. So the built in solution is not an option in this case. There are also a few additional issues with MovieTexture in that it requires Quicktime to be installed on your machine, but since Apple no longer supports Quicktime for later versions of Windows, this would be a challenge for some users.
Option 2: Will Unity be able to do this in a future version?
The next option I looked at was the upcoming releases of Unity. It turns out that in the roadmap for their future releases, they mention a new Video API, which (hopefully) will include support for mobile. Sadly, in my case this is also not an option at the current time, since that release is not open to the public. Still, if this was something I looked at for a longer term project, checking to see if the functionality you want is in an upcoming build is always a great second option for software that’s in active development. Spending a significant amount of time developing something amazing to solve a problem that you have, only to realize that the next release of the engine you’re using does exactly that can be a frustrating experience, as well as a waste of time you could have spent working on a different feature.
Option 3: MediaDecoder
My next option was to start looking on the Unity Asset store to see if i could find any great VR video packages. I ran across this awesome package created by the HTC Vive R&D team called MediaDecoder. I spent a little while testing it out, and it works exactly as promised, but as with the MovieTexture option, it’s not designed for mobile, only for PC, so once again, isn’t a solution to my problem. Even though it wasn’t a solution in this specific case, knowing that it exists and how it works could be valuable to me in the future, so spending half an hour testing is still a worthwhile use of my time.
Option 4: PopMovie
Another great package I’ve used before in Unity is PopMovie. In this case, this is a Unity package I know works well in VR for all the hardware options I would need, and is extremely simple to use. In this case, though, the problem is that this package is $100. While this would be great for a commercial project where the $100 could save me days or even months of development time, in this case, requiring my students to buy a $100 package in order to complete the course isn’t really an option.
Option 5: AVPro
Finally, one of the other VRND course developers — Matt Sonic — recommended something he had worked with before — AVPro. While this also isn’t free as a full version, costing $450, there’s a trial version of the package that adds watermarks to the videos. Since the purpose of my testing was to find something students can use to create educational projects, watermarks aren’t an issue for me. I tested it out, and the package does what I want and need it to do, and fits all of my requirements. Less than a day later and I’ve worked through my problem, found a solution that fits, and now all I have to do is teach the Udacity Virtual Reality NanoDegree Students how to use it!
Hopefully, this gives you some insight into what it’s like to be a developer — a lot of the time, you don’t automatically know the answers to every problem, instead, you spend a lot of time with google, testing and re-testing until you have a solid path to victory.
