Open sourcing our 360 video library for iOS: DDDKit

Published in

Pie | Press & Media Information

4 min readJan 24, 2017

At Pie, we’re open sourcing our library to display 360 videos. We’ve been using it in our app since we launched in Nov 2016 and it’s been working great for us. So we’re very happy to open it up for others to use, get inspiration from, make suggestions for changes and contributions!

It’s now available as a cocoa pod: pod ‘DDDKit’

Our focus is on 360 videos, 360 images and filters. So we designed DDDKit to make those parts easy as pie.

Usage

DDD360VideoViewController

It’s a pre-baked convenience view controller that handles most of the logic so that you can get started quickly. When you get more serious about it, you’ll want to dig deeper in DDDKit:

That’s it!

DDDKit

It’s a more flexible and involved 3D library, similar to SceneKit in some aspects, and DDD360VideoController is just one aspect of it. While our focus is on 360 video, we’ve structured this library as a generic 3D rendering library. So it’s very flexible when you want to get more involved, assuming you’ve some basic understanding of some 3D rendering ideas.

You’ll find this design similar to other ones you might be familiar with, like SceneKit. The high level objects are:

DDDScene: a scene, or a container for your 3D elements
DDDNode: a specific 3D object. It can be added to a scene with scene.add(node: someNode)
DDDGeometry: a shape a node can have. node.geometry = …
DDDMaterial: a descriptions of an external aspect a node can have: node.material = …
DDDShaderProgram: a low level description of the rendering computation happening on a material: material.shaderProgram = …
DDDProperty: a generic parameter that can be passed to a shader program. It can be a number, and image etc

360 video example

This is all quite abstract! Let’s see how we render a 360 video. (You will see that this is almost all what DDD360VideoViewController does.) We’ll create a scene with a spherical object where the video will be displayed and we’ll rotate the sphere when the user swipes through the screen. That’s it!

First we’ll render things within a DDDViewController that you should absolutely use:

We then load a video:

and set up the scene:

Videos are a bit annoying to handle, because they get decoded as YUV which is two layers that represent the brightness or luminance plane (Y plane) and the color plane (UV plane). So it’s like getting two inputs that each have part of the video information. We’ll have to put that together in a shader program (there’s not much more to understand in the code below other than that we’re converting the YUV to RGB!):

We attach this program to the node’s material:

and finally, we pass the video input as a property to this material:

Almost there! We just need to let the user turn the video:

Note that all DDDKit does here is in videoNode.rotation = …

Adding filters

Filters are great to make media look much nicer. You can use them with Pie of course :)

We wanted to have super easy support for them without losing the whole range of filters we can create. So we left them at the shader program level, but made an easy to use API inspired by SceneKit’s shader modifiers

Remember the shader code? There’s two commented lines // header modifier here and // body modifier here that are placeholders for code blocks we can insert. To do a simple black and white filter, we replace the program definition by:

Full code is here.

Why not SceneKit?

SceneKit is Apple’s 3D rendering framework. It has a broad focus and for many things is quite robust and well designed. However we decided to build our own replacement because:

SceneKit introduced memories leak in iOS 10.0
The support for videos is not natural and is done through SpriteKit, Apple’s 2d rendering framework.
We found SceneKit’s online documentation to be extremely poor for the features related to videos and shader modifiers.
When you’ve an issue, there’s no code to look at, a very small SO community to turn to, and no one to respond to your bug reports in a timely manner. Apple != open source!
Our focus is smaller than SceneKit’s and we think we can offer better solution within that tighter scope.

Looking forward

Here are a few things we’ve on our mind:

Improve our own documentation.
Look into Metal support
Add unit tests

How can you help

Use DDDKit, open issues with bugs and suggestions. We’ll be responsive.
Let us know how you’re using or planning to use DDDKit and if some features are missing.
Give us a star!