Sensor Input with Unity3D

Here at Truth Labs we do plenty of work with interactive 3D environments. The projects that call for this type of work can range from web apps to large scale installations. There’s a full house of technologies and frameworks that will deliver results — WebGL, Cinder, Processing, Flash, OpenFrameworks… the list goes on. But when it comes down to turning around high-end interactive 3D visuals efficiently and effectively, Unity3D is an obvious choice. This post will introduce an example of interacting with Unity using other applications and hardware.The source files are available on Github.

Why Unity3D?

Unity3D is a game engine popular with indie game makers for its ease of development and cross-platform delivery to mobile devices, web browsers, desktops, and consoles. The platform boasts impressive graphics and physics engines out of the box. For Truth Labs, it means a designer-friendly route for building powerful interactive content because of features like native support for Javascript and integration with popular 3D modeling software like Cinema4D or 3ds Max.

Talking to Unity

While Unity has native plugins for popular interactive controllers such as the Kinect or Occulus Rift, these plugins will typically only work with the paid “Pro” version of Unity. For our purposes we’ll be building with the free “Indie” version, which is a great reason to check out ways to add a communication layer between applications.

Let’s explore talking to Unity and Processing via OSC. We put together a simple “hello world” example that can be used as a starting point.

• OSC (Open Sound Control) is a simple network protocol originally created as a network alternative to MIDI. It is a great method for sending small packets of information between applications at very fast rates, so its an essential tool in the belt for interactive installations and physical computing projects.
• Processing is a ubiquitous creative coding environment based in Java. Its huge ecosystem of add-ons makes it trivial to get OSC and hardware-based input up and running.

Trying it out

The example files are available on Github. Go ahead and grab Unity and Processing if you don’t have them installed already. You’ll also have to download and install the oscP5 library for Processing.

“Hello World”

Once everything is installed let’s go ahead and run the Unity scene and the Processing sketch.

Open up OSC_Processing.pde and run the sketch. Roll your mouse over the black applet that launches. You should see coordinates printing to the Processing console.

With the Processing sketch running, open up the Unity project and play the scene.

You should see something like this:

Under the hood

In Processing (OSC_Processing.pde) we use the oscP5 library to publish mouse coordinates over a local network to a port that Unity will subscribe to. We’ll put each coordinate on a separate channel to keep things clear.

In Unity an empty game object called “OSC” holds the Unity OSC libraries (Osc.cs, UDPPacketIO.cs) and our receiver (OscReciever.js). The receiver uses the same ports and channels that were defined in the Processing sketch, and stores the coordinate data for the other game objects to access. After that, the “Cube” game object uses the scaleObject.js script to access the receiver data and transform the size of the cube. An auto-rotate script (autoRotate.js) was added to the cube just for kicks.

Leap Motion + Unity

We went ahead and built an experiment using a Leap Motion to control a Unity environment. Processing interfaces with the Leap and broadcasts hand position data to Unity. The goal was to test out Unity’s impressive real-time 3D rendering and built-in physics. In addition to our Processing and OSC component, we introduced some simple game logic, models made in Cinema4D and a particle system.

Here’s the experiment:

Wrapping it up

In this post we touched on a simple technique that opens up a wide world of interaction design possibilities. Not only is Unity a powerful platform for creating rock-solid, cross-platform interactive experiences but OSC is a great communication layer with support in a variety of languages and software. This example is just a small piece of the puzzle. We’d love to hear feedback or see what you’ve made.

Hit us up @truthlabschi or find me @kevinports.

Originally published at blogs.truthlabs.com on May 23, 2014.