Geeking Out Backstage on a Camera Piano
We went to CES 2018 and our friends on the engineering team at Intel took us backstage for a closer look at a camera-based instrument called “The Firefly,” used in Intel’s Keynote pre-show.
It has only been a few short years (2015) since Intel released the first iteration of their RealSense 3D cameras. Since then, a myriad of projects have been created using the platform, including autonomous robots, smart mirrors, and even haptic feedback wearables for blind navigation. Intel engineer Matt Pinner tasked his RealSense cameras for a different kind of project — a camera-based digital musical instrument known as The Firefly.
The Firefly looks and sounds like something straight from a sci-fi show! It got its name because the CNC-cut outer casing of the instrument looks a bit like a futuristic spacecraft. Matt decided on the name “in honor of the Joss Whedon television series, Firefly.” The instrument itself acts much like a theremin — as the musician’s hands get closer to the metal rods, it produces a more pronounced sound or modulation through the oscillation of frequency. In the case of The Firefly, instead of a magnetic field to modulate tone, a pair of upward-facing 3D cameras track hand positions to produce musical notes, sound effects, and volumes.
Matt’s Firefly was part of Intel’s Keynote pre-show at CES, where famous composer Kevin Doucette (Dead Rising 4, Pele: Birth of a Legend) performed The Killers’ song Human using his hands to manipulate the instrument’s LEDs and virtual keyboard. This was done using hand gestures and various distance positions while standing over The Firefly itself.
This unusual musical instrument has a super cool design concept that begins with the casing — the enclosure is CNC-milled and engineer Adelle Lin modeled it on Intel’s Signature Chime waveform.
Its original composer, Walter Werzowa, used just four notes (D flat, G flat, D flat, and A flat) to represent the syllables of the company’s slogan “In-tel In-side,” which was translated into The Firefly’s unique look.
Inside each instrument houses the magic — a pair of Intel RealSense 3D cameras outfitted with IR (infrared) and RGB sensors along with an IR laser projector and onboard image processor. These are mounted at each end of the enclosure to cover the range of motion of both hands while standing in front/over the musical platform.
Each camera provides depth and RGB video at a rate of 90FPS with the data garnered being processed for hand shapes, their center mass, identifying right verse left, as well as their exact position in camera X, Y, and Z. For performances, Matt expands all of the collected data to Open Sound Control (OSC) packets and the data is sent over a multicast group over a local network. Each hand is given their own respective audio and video parameters simultaneously while playing.
The lighting effects are provided by a pair of 88 individually addressable LED strips. Each LED represents a key on a piano as well as showing the orientation of the player’s hands — too far forward and the back strip will fade slightly; too far back, and the front dims. Coupled with a substantial amount of resizing and soldering, the lighting effects turned out quite nicely with bright, vibrant illumination that can be seen independently for each slice in the casing.
As for the software controlling The Firefly, Matt stated, “LibRealsense works fairly well on Linux, Mac, and Windows for building your own applications. I’ve used the cameras mostly on Windows in Unity, TouchDesigner, and for this, I based the initial code on C++ examples found in the RealSense SDK.”
While The Firefly camera-based musical instrument is a sight to behold, Matt worked with several talented individuals and companies who gave their all so that our eyes and ears could enjoy. Congratulations to all on an incredibly inspiring feat of engineering:
- Adelle Lin: Design, Modeling, Fabrication
- Claire Goolsby: Assembly, Configuration, Operator
- Nerdmatics: Cycling74 Max and Ableton
- Form Factory: CNC
- Kevin Doucette, Performer
