Building a 360 Video live streaming system
UPDATE:
3/2/16 VahanaVR engineer Jeremy Camp and his team were very helpful and the new beta, which will be released soon, has more robust RTMP streaming and we did a test last night and streamed without crashing! Jeremy also offered some advice about the Theta S which I have put into the post below.
3/1/16 Engineers at VideoStich are working with me to resolve the crashing and I will post any solutions.
The reason we were able to watch Neil Armstrong take “one small step for man” is because of an unknown NASA hero, Stan Lebar, who had one very important job — develop a camera and transmission system that could capture video and audio on the moon and broadcast it to half a billion people.
Not that our challenge of building an affordable 360 degree live streaming system comes close to the importance of the footage Lebar successfully captured, but throughout this process, I have felt like I imagine Lebar must have felt during those weeks leading up to the moon landing.
360video workflow is hard enough, but stitching it live and streaming it to the world is a major feat.
My goal was to create an affordable system/process that would enable news organizations to stream events in 360 degree video, everything from big news events to the local town council meeting. After talking with some news executives and editors I decided that if I could do it for $5,000 it just might work for regional publishers. Supporting regional publishers is a goal of a Knight Foundation grant we received at UNC.
It took weeks of work, a burned up video card that spewed blue smoke, various 360 rigs and trying multiple computers to successfully create a live 360 streaming system for less than $5,000. Now, during the process of R&D I spent closer to $14,000 on different configurations of computers, graphics cards and cameras, but it was all equipment we are using for other projects. Did Labar come in under budget? Just wondering because with as many cameras, cables and configurations I have tried, I can’t imagine how many he tried!
Here are the basics of how it works. Try to read this in one breath. Ready, Go!
A 360 rig provides a live preview of the scene through HDMI outputs. Those HDMI signals are captured using a special Maxwell HDMI capture card or dongle (more to come). That is fed into a gaming-level PC (sorry, no Mac solutions) through an NVIDIA graphics card. The Vahana VR Live stitching software handles the stitch and pushes the feed out an RTMP feed. The feed is transcoded through Bitcodin into multiple adaptive versions and all of those files are pushed to Google Cloud Storage. The HTML5, responsive, adaptive player retrieves the files using the HSL and MPD protocol from GCS and delivers it to the user in a 360 view. Whew… got it?
Just like any major problem that needs to be solved, I broke the problem into five problems to take on individually. Those quickly became hundreds of little problems but the major points are:
Cameras, Capture, Stitching, Streaming and Player
Cameras
It came down to three 360 rig options because I only wanted to deal with four inputs or less.
1. Elmo Freedom 360. A four camera rig that uses the Elmo Qbic cameras. This worked well on my desktop and provided a nice image quality that was better than I would be able to stream. The image is not warped and looks better than the other options.
2. Richo Theta S. The price and single camera is nice but I decided not to go this way for two reasons. First, the quality is not great and the image prior to streaming is pretty weak. Second, the HDMI feed is one feed but with both images joined and not stitched, making it difficult to work with. I did find someone claiming it would work but have yet to see it in action. I do plan to test this in the future.
UPDATE 3/2: From Jeremy Camp with VahanaVR:
While the Theta is not natively supported by Vahana due to the fact that it provides both images in one feed, we provide a workaround to use it anyway :
This workaround is not working with Magewell HDMI-USB dongles so not really interesting for laptop setups.
In all cases, we do not really recommend using the Theta, as you write the quality is not that good and it is complex to set up a stream with it.
3. Kodak SP360 4k. This two camera rig is ideal for my laptop solution except for one big problem. The HDMI and USB ports are right next to the tripod mount so you can’t mount the camera and charge/stream at the same time. This is a major design flaw and one I did not see till I actually had the cameras in my hands. I got around it by creating my own mount, first with gaffers tape and then with screws, but still had to cut away some plastic from the HDMI cables to make it fit.
Two cameras are easier to deal with than four, it cost less and I was able to stitch and stream via a laptop and not my PC tower as I had to use for the Elmo setup.
Once you have a camera set up, you need long HDMI cables to take the computer farther away from the camera. Mine are 25 feet, with a Micro HDMI adapter. I suggest a cable version rather than adapter because it is easier to secure. Another option would be to mount the laptop with the cameras and use a second laptop with remote desktop but this seems like one extra thing that could go wrong.
Capture
I first built a massive PC tower with an NVIDA graphics card to handle the processing and added the Maxwell 4HDMI input PCI card. In total, the PC cost about $1,200. This worked for the Elmo rig but dealing with a desktop on the road would be difficult. So for version two, I found a laptop with the proper graphics card and 4GB of dedicated video ram, two USB 3 ports and bought two Maxwell USB Capture HDMI dongles. I only need two because I am using the Kodak SP360 4k.
Stitching
Vahana VR is the only software I can find that can stitch in real time and stream it live. It crashes all the time! I mean, multiple times a day and it can’t stream longer than 32 minutes without crashing. UPDATE: 3/2/16 Engineers at VideoStich were very helpful working with me to resolve the crashing. The solution seems to be the new beta to be released soon.
I like the VahanaVR interface and it is fairly easy to configure and set up. I was able to get both the Elmo rig and the Kodak rig working pretty quickly. The interactive view is nice.
Streaming
VahanaVR has basic RTMP feed configuration for setting up with a live CDN. They recommend Wowza streaming — which I tried and it worked — but I prefer bitcodin.com. I chose them because of their 360 adaptive player integration and I found out later that their tech support was great. Both services have adaptive streaming but Bitcodin seemed to make it easier by providing a player.
Basically, you take the stream from VahanaVR, send it to Bitcodin for encoding, who then sends the files to another storage solution like Amazon or Google Cloud Storage. I set it up with Google Cloud Storage. You need some technical, server knowledge to get this running as there are permission/domain security issues that must be worked through. The support team was very helpful in helping me work through this problem.
Player
The player I used comes from Bitcodin and has a well-documented API for integration and it is easy to turn on VR/360 video. It is skinable and most importantly handles the Adaptive streaming functionality using both MPEG DASH and HLS. This is the industry standard for adaptive streaming (Netflix, Amazon Straming, etc.) and is necessary for quality 360 video on multiple devices. The only problem I have with this player, and it is not Bitcodin’s fault, is that it does not work on iOS Browser phones due to Apple’s policy.
So, if you can do all of those things you could be Stan Labar’s intern and help figure out how to stream in 360 when we take men to Mars!
But since you are not working with NASA’s budget, let’s look at price.
Vahana VR Software: $2,700 (now reduced to $2000)
Magwell USB to HDMI: 2 x $300 = $600
HighEnd Laptop with gaming graphics card and 2 USB3 ports: $1250
Adapter cables: $12
2 pack of HTML Cables 25ft: $18
Kodak SP360 2 Camera Kit: $900
Total: $4,780
Seriously, it was quite difficult and was plagued with problems, some still unsolved, but it works and we can livestream 360 video for less than $5,000.
