Here’s what a 360-degree Snapchat video with Augmented Reality filters looks like

After gathering inspiration at the 2016 SIGGRAPH computer graphics conference, I went home and stitched together a panoramic video overlayed with Snapchat’s facial tracking filters. The goal: to showcase what an augmented reality future would look like in a 360-degree environment.

The first step involved changing the save settings in Snapchat so that all images and videos would be stored in the camera roll. From there, I rigged my Android phone to a tripod and positioned it on my balcony.

Staying within the frame was very important because of the amount of stitch lines. Crossing seams was out of the question for this shoot, that was for sure.

Stabilizing the phone during recording was critical. The difficult part was making sure that the camera stayed still when pressing and holding the record button on Snapchat. Even slight movements of the camera would be surely noticeable.

After the entire scene was individually captured, all footage was dumped onto a computer to sort through. Interestingly, Snapchat’s app recorded the video files at different frame rates. This meant they needed to be re-encoded to match each other before importing them into a stitching software.

This image shows the links and amount of control points between cameras in Autopano

I chose to use Autopano Video Pro because it is arguably the best off-the-shelf solution out there to stitch 360-degree videos. After importing, the files were linked together by matching pixels (aka control points), giving the software plenty of information to work with.

The result was rendered out and uploaded to Youtube.

Will Snapchat users be able to share 360-degree videos in the future?

Just recently, Snapchat quietly introduced a way for advertisers to release panoramic videos playable from their platform. This makes total sense as large studios, like Sony Pictures Entertainment (SPE) who were the 1st to debut a 360-degree video on Snapchat, can afford the time and money to develop this type of immersive content.

Granted, consumer level 360-degree video cameras are hitting the market making it easier for people to create their own videos. But in order for people to upload immersive content to Snapchat, the company has to allow external connections to other cameras so that users can shoot 360-degree videos and post in real time.

Samsung’s Gear 360 is among the most widely adopted consumer level 360-degree cameras

Capturing 360-degree videos within the current version of Snapchat is not a viable solution as it takes too many perspectives to encompass the full environment — depending on the camera’s lenses.

Another option would be to give users the ability to post 360-videos stored on a phone. The 360-degree video player is already embedded into the Snapchat apps. So it would just be a matter of “flipping the switch” for their users.

Snapchat users can post from their camera roll with the new Memories feature

Tips for creating 360 videos with AR Filters

If you want to create an immersive video like this one with augmented reality overlays, the following list will help:

1. Test out fisheye lenses. Recording around 50 perspectives produces too many stitch lines. As long as the software can still track facial features with alternate lenses, the stitching will be easier.
2. Keep in mind that facial tracking on smartphones is limited by distance. The farther away the person is from the camera, the harder it is to detect their face and overlay with animations.
3. Film the action portions of the scene and stitch together the rest of the equirectangular sphere with still images. This reduces the potential motion of the cameras to singular focal points, creating a more stable video.
4. Add stickers to some of the videos. There are so many options!
5. Look into the Line Camera. It is probably the easiest way right now to get an augmented reality overlay in real time. Apps like Snapchat and Snow rely on the person holding the button to record. Line on the other hand gives the option to press the button once to start recording, removing the potential problem of shaking the camera when starting. It also saves clips longer than 10 seconds — which is Snapchat’s current limit.
6. Forget animating in post. The goal of this type of experiment is to record AR overlays in real time. It is more fun that way.

OlloClip is a good option for smartphone fisheye lenses

Contact Me

Creating 360-degree videos? How about AR or VR? If so, email me at terndrupm@gmail.com