Interrogating the Strokes

Image courtesy of

While guerrilla investigative journalism into the New York garage rock revival scene is close to my heart, this story will be about using Google’s evolving Tilt Brush Toolkit to unlock new interactive possibilities in Unity.

While the Toolkit is great for whole-sale mesh import of art via FBX/OBJ import straight into Unity, there is a layer of data available via the Python API. Here is how to get to it.

First, figure out if you want to write Python in Unity or write a separate Python executable to get at that sweet stroke data. If you want to stay nice and warm inside Unity, there are plugins available this but I didn’t personally evaluate this pathway.

Then, figure out the size of the data you are working with. Tilt Brush data is quite dense: a sketch with two 1 metre straight lines yields about 300 control points or 9600 bytes without compression. This will inform if you want to process the data during runtime or design time. I chose design time.

Now, write a Python script that dumps out the data you need to an intermediate file that you can include in your Unity project filesystem. Here is a quick sample showing how to get just control point positions. Maybe you want to turn a Tilt Brush sketch into particles that fly apart! Maybe you don’t.

tiltBrushData = Tilt(filename)
with open('strokeData.bytes', 'wb') as strokeData:
# First, write out the number of strokes to process
strokeData.write(struct.pack('<i', len(tiltBrushData.sketch.strokes)))
# Then write out the data you're interested in per stroke, starting with number of control points
for (i, stroke) in enumerate(tiltBrushData.sketch.strokes):
strokeData.write(struct.pack('<i', len(stroke.controlpoints)))
strokeData.write(struct.pack('<f', stroke.brush_size))
     # Now write out the data for each control point of each stroke
for cp in stroke.controlpoints:
strokeData.write(struct.pack('<f', cp.position[0]))
strokeData.write(struct.pack('<f', cp.position[1]))
strokeData.write(struct.pack('<f', cp.position[2]))

Finally, read and animate your data inside with a Unity with a script that matches the format of the data you just wrote. Be sure to use the file extension .bytes for your TextAsset, without this, Unity will employ some arbitrary encoding to your BinaryStream. In the documentation this is listed as “unexpected behaviour”. Here is an example of walking through the sketch data procedurally in C#

Stream strokeDataStream = new MemoryStream(YourTextAsset.bytes);
BinaryReader strokeDataBytes = new BinaryReader(strokeDataStream);
int pos = 0;
while (pos < strokeDataBytes.BaseStream.Length)
int numStrokes = strokeDataBytes.ReadInt32();
pos += sizeof(Int32);
  float brushSize = strokeDataBytes.ReadSingle();
pos += sizeof(float);
  int numControlPoints = strokeDataBytes.ReadInt32();
pos += sizeof(Int32);
  // You could modulate how far you walk through the stroke each frame to animate the drawing of the sketch
for (int cpCount = 0; cpCount < numControlPoints; ++cpCount)
Vector3 cpPos = new Vector3(strokeDataBytes.ReadSingle(), strokeDataBytes.ReadSingle(), strokeDataBytes.ReadSingle());
pos += sizeof(float) * 3;
    // Append to YourMesh or spawn your particles or do whatever magic you like here!
YourMesh.vertices = new Vector3[YourMesh.vertices + cpCount];
YourMesh.triangles = new int[YourMesh.triangles + cpCount];
  Graphics.DrawMesh(YourMesh, cpPos, transform.rotation, YourMaterial, 0);

Magic! It turns out getting realtime data using the Tilt Brush Toolkit isn’t so hard to explain.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.