Snapchat Lens Studio for Unity Developers

TL;DR — Snapchat Lens Studio is a powerful and expressive AR tool which happens to have a lot in common with Unity. For developers who spend much of their time in Unity, this article offers a quick overview of how the two programs are similar and different, including examples of congruencies between the two, so you can dive right in with a comfortable baseline.

Our team at Fathom has been asked about developing Snapchat lenses several times since the tool for creating them, Snapchat Lens Studio, went public in early 2018. To be honest, we had initially discounted the software as a sort of plaything, but once we started digging in, we found the tool to be super robust and that it offers an exciting opportunity to quickly build immersive, AR micro interactions. I would wager, too, that the component-based architecture and overall UI of Lens Studio was heavily inspired by Unity, hence this article!

We’ve come to see Lens Studio as a platform for “interactive haikus,” tiny AR worlds that can be an experiment in AR interactivity, an engaging branded experience, or a purely creative exploration using face-based inputs. These experiences can be published within minutes and shared to world using a generated Snapcode.

You can start building Snapchat lenses immediately by downloading the application here.

And just by chance, it was recently reported that Snapchat will be announcing its own gaming platform in April, 2019. What better way for Unity game developers to explore this new ecosystem than through the existing Lens Studio tools?

Before we dive into the two programs’ similarities, here’s how Lens Studio projects are different than Unity ones:

  • Small project size: packaged lenses must be under 4 MB
  • Untyped JavaScript: vanilla JS, no C# with typing
  • Limited inputs: touch and gesture input, face-based input, AR surface and image marker tracking, so no gamepads, keyboards, mice, etc
  • No compilation: all scripting here, which keeps the editor workflow quick and seamless
  • Publishing: lenses can only live on Snapchat’s mobile platform, no external build files can be shared
  • Ad-hoc distribution: published lenses get a Snapcode which is the gateway to your custom lens experience
  • No built-in physics: plenty of linear algebra classes, but you’ll have to code your physics and collision system, or fake it
  • Limited community codebase: lots of examples of working lenses, but really not much code sharing in the community. Snapchat’s templates are the best existing resource that we’ve found so far. There’s no equivalent to Unity’s Asset Store for bringing external resources into your project

Built-in Features

  • Face detection and mapping—2D/3D anchoring with simple head occlusion
  • Face gesture events—mouth open/close, brows raised/lowered, kiss, smile
  • Face manipulation — distort/liquefy, slice, duplicate/composite, image effects (teeth whitening, tone mapping, eye color change)
  • 2D/3D assets — mesh, text, gifs/images as billboards, rigged animation
  • Environment tracking — image markers, Snapcode markers, 3D surface tracking

Interface

So much of the UI between Unity and Lens Studio is equivalent, we’ll just show side-by-side images of the main panels and note how they differ.

Hierarchy = Objects
The main scene tree is displayed in the Objects panel. In Lens Studio, there is no concept of multipleScenes like there is in Unity. The entire project exists in a single global object of type ScriptScene.

Unity Hierarchy panel and Lens Studio Objects panel

Project (Assets) = Resources 
All imported resources for the project are managed in the Resources panel. These are mapped to the {Project Path}/Public folder in the same way a Unity project’s Assets folder is mirror of the file structure. You can drag external files onto this panel to add them to the project.

Unity Project panel and Lens Studio Resources panel

Scene = Scene
The main editor space for both programs is the Scene view. The same transform tool shortcuts in Unity along the QWERTY keys also work in Lens Studio. The main difference is 2D Scene View in Lens Studio, which will show a contextual 2D view, depending on which SceneObject is selected. For instance, a selected FaceMask object will display the 2D face binding anchors when View 2D Scene is selected.

Unity and Lens Studio Scene views

Game = Preview
You can preview your live lens at all times in the Previewpanel. Unlike Unity, there is no Play or Pause here. The lens is running constantly and will restart when it detects scripting or scene changes. You can force restart the lens by clicking the circular arrow to Reset Preview. You can view the lens through your webcam by clicking the small icon. Lastly, you can stream the lens right to your phone once you’ve paired Lens Studio to your device. This is super useful for AR-tracked lens interactions.

Unity Game panel and Lens Studio Preview panel

Inspector = Inspector
When a SceneObject is selected in the Objects panel, the Inspector will show the settings and components attached to that object. Look how similar!

Unity Inspector and Lens Studio Inspector

Console = Logger
Get feedback from your scripts in the Logger panel. Instead of Debug.Log("You say Console"); a la Unity, you can use the global function print("I say Logger");.

Unity Console and Lens Studio Logger

GameObject, meet SceneObject

Unity’s generic element in a scene is the GameObject and the equivalent in Lens Studio is the SceneObject. Every SceneObject has a Transform that defines its orientation in 3D space. They are arranged in a parent-child hierarchy, so empty SceneObjects can be used as folders that contain multiple children.

There is also a Prefab system similar to Unity’s. Any SceneObject can be dragged into the Resource panel to create a prefab.

Creating a Prefab in Lens Studio

The big difference here is that a prefab is configured on the instance, not on the prefab asset in the Resources panel. That is to say, you first instantiate a prefab by dragging it into the Objects hierarchy, make some changes, and then hit the Apply button. This will push changes to all other instantiated prefabs of this type, and all future instances.

Hit the Apply button to commit prefab changes

MonoBehaviour, meet Component

One of Unity’s best features is its component-based architecture. As you may have noticed in the Inspector panel comparison above, Lens Studio uses an almost identical system. You can attach any number of Components to a given SceneObject to define its behavior.

All Unity components derive from the class MonoBehaviour and all Lens Studio components derive from the class Component. Every component you can attach to a SceneObject and every script you create will automatically derive from this class.

There are a number of built-in components that can be used to create custom behavior. Hit the + Add Component button in the Inspector panel to view them.

Adding a built-in component to a SceneObject

Enough About the UI, Let’s See the Code!

Ok, ok. We’ve seen how similar Unity and Lens Studio are from an interface perspective. Let’s get into the code!

Attaching a script to a SceneObject
Like Unity, all custom logic is created in instances of ScriptComponent. First create a Script resource in the Resources panel.

After creating and naming your Script resource, you can now attach the script to any SceneObject from the Inspector panel. A ScriptComponent can hold any number of script resources, whose logic will be triggered by a corresponding ScriptEvent. For scripts that don’t have any state, you can simply choose the relevant event and the code will run accordingly.

If you’re like me, and want a more Unity-like scripting experience, you can run a single script from the Initialized event and bind callbacks to any other event that occurs in the scene. Here’s an example of that syntax.

Events are bound using their Event class names as a string. You can find all available events here.

If you want to include state in your script (which exists per instance of the script in the scene) use variables at the top of the script scope and run the script during the Initialized script event at the component level.

Now you’re probably thinking about all the weird micro games you can make using eyebrow detection as the input! I know I am…

Creating script properties in the Inspector
In Unity, public properties are exposed to the Inspector at the component level. Of course, Lens Studio has an equivalent feature, though the syntax looks different. You can provide many types of Input field data and references to other SceneObjects and Components using the following code:

// The @input symbol after comment slashes indicates an Input field
//@input int myScore
//@input vec3 myColor {"widget":"color"}
//@input SceneObject someObjectReference
//@input Component.ScriptComponent someOtherScript
//@input Component.MeshVisual someOtherMesh
// Access all of these properties from the `script` object 
print(script.myScore);

Notice how specific Component types use the Component. prefix to expose them to the Inspector. Without this prefix, the script will throw an error.

In some cases, you’ll want a better UI for the type of data you’re exposing to the Inspector. Unity does a great job choosing the right input UI for the data type. With Len Studio, you’ll have to be a bit more explicit. See the myColor widget line above and find more examples of Custom Input UI here.

Now, all of our properties are available in the Inspector

Making Things Happen

Grabbing SceneObject and Component references
Everything starts with the script variable, which is a reference to the current script instance, like this in Unity context. Once you have a reference to the SceneObject, you can start grabbing components and manipulating their values. Here’s how to grab those references in the code:

// Get a reference to this SceneObject
// Equivalent to `gameObject` in Unity
var so = script.getSceneObject();
// Get a reference to this SceneObject's Transform
// Equivalent to `transform` in Unity
var t = so.getTransform();
// Equivalent to `gameObject.GetComponent<T>()` in Unity
var meshViz = so.getComponentByIndex("MeshVisual", 0);

Find more methods available to the SceneObject, like adding or removing components at runtime here.

Communicating between scripts
When writing a script, you can expose “public” properties and methods via the script.api object.

// Script 1
// Prints a message
script.api.showWelcomeMessage = function(msg) {
print(msg);
}

// Script 2
// Set the reference to the other object in the Inspector
//@input Component.ScriptObject welcomeScript
// Check for the Script1 reference and that the 
// method we want to use exists
if(
script.welcomeScript &&
script.welcomeScript.api.showWelcomeMessage
){
// Send a welcome message using our reference to Script 1
script.welcomeScript.api.showWelcomeMessage("Hi there!");
}

Lastly, the global namespace is available for static access to properties and methods. Values set here are available to all of your, as long as they are defined before you try to access them:

global.myMessage = "Hello, World!";
global.sayHello = function(message)
{
global.myMessage = message;
print(global.myMessage);
}

Next steps

By now, it’s probably clear how similar Unity is to Lens Studio, which is great for Unity creators who want to start experimenting with sharable AR lenses. There’s plenty more to learn, like triggering 3D animation, handling touch input, and sharing your creations.

If you’ve experimented with Lens Studio, share your lenses in the comments. Or if you are Unity user who jumped into lens development, let us know what you think of the tool!


Eric Howard is Chief Technologist and Co-founder of Fathom, a digital creative studio that brings brand experiences to life through storytelling and interaction. www.fathom.company