Making music with gestures
Attention! We are giving a workshop on gesture controllers for music at Slush Music this Wednesday. In it we cover the history of gesture controllers and introduce new designs. Get your tickets and register for the workshop here.
My colleagues and I at Genki Instruments are designing a modern gesture controller for music. We call it Wave and hope to release it early 2017.
Two integral steps in the design of a new product such as Wave is (a) acquiring domain knowledge and (b) understanding what customers want. Towards step (b) we have interviewed numerous musicians in the Icelandic music scene. The interviews have led us to believe that musicians need better tools to interact with their laptops. Laptops today are the main instrument and studio for most musicians. To gain domain knowledge we have sought to understand the design of gesture controllers for music done in the past along with reading up on current research trends in DMI (Digital Musical Instrument) design.
In this post we hope to share with you what we have learned from pouring over gesture controllers designed in the past.
Keep this in mind
By looking at gesture controllers we have been able identify a few key points that matter in the design of novel gesture controllers for music.
- Is it a gimmick?
- Is it expressive?
- Can the musician do other things?
- Does it force the musician into being stationary?
- Does it provide other feedback than auditory for the musician?
- Can an audience tell what the musician is really doing?
These points matter to us because we want to create something of value: A device that expands creative freedom and enables control of digital assets in a more expressive or, well, human way.
What follows is a compilation of gesture controllers from the past. This is by no means an exhaustive list of gesture controllers, only the ones that we have learned the most from.
The Theremin
This is where it all got started. León Theremin, an avid inventor, was working on spy devices for the Soviet government. In 1919 he figured out a way to use the proximity of his hand relative to an antenna to control pitch and amplitude of a wave which he then played using a speaker. Playing an instrument without any physical touch must have appeared divine at the time. Suffice to say the new instrument was received with great acclaim and León Theremin filled concert halls around Europe.
The fact that there is no tactile feedback, like the touch of a string on the guitar, makes the Theremin difficult to play. That explains in part the use of vibrato (modulation of frequency) in the characteristic sound of the Theremin. It is next to impossible to find and hold the right note for an extended period of time without any feedback mechanism. Playing a fretless guitar is hard enough for a guitar player. Now imagine playing it without a neck altogether and what you get is something similar to playing a Thermin.
“I know that the Theremin doesn’t provide tactile feedback. The Theremin is a special case. It’s very expressive, but in a somewhat quirky way.” — Bob Moog
For information on León Theremin’s remarkable life, see the documentary Theremin: An Electronic Odyssey
The Laser Harp
Popularized by Jean Michael-Jarre, who plays it in concert still to this day, the Laser Harp is without doubt the most visual gesture controllers out there.
The controller is essentially an array of laser beams. Each beam acts as an on/off switch. One common mode of operation is to have beams in the off-state by default. A musician then blocks a beam which is subsequently sensed by a controller and an on-state triggered. These on/off signals are then translated into sound… In Jean Michel Jarre’s case, a 80s power-synth.
As with the Theremin, the visual aspect of the controller is appealing but there is no feedback other than sound. Furthermore the Laser Harp has only two states per note which makes it inexpressive and gimmicky.
The Hands
In 1984, one year after the MIDI protocol was released, Michael Waisvisz premiered a rather peculiar device called The Hands.
Two wooden frames containing several sensors (which types we don’t really know) and small keyboards attach to both hands of the performer. This combination allows the performer to play music with hand and arm movements along with fingered playing. By translating data from the sensors into MIDI signals, by use of microprocessors, The Hands could control a variety of digital equipment that had the new MIDI standard.
The Hands were remote controlling three Yamaha DX7’s programmed with special, very responsive, sounds. Even though a crucial cable broke during the concert causing a hilarious situation — because some of the sounds kept stupidly repeating until I managed to reconnect the cable, while still sort of playing, in full spotlight ! — this premiere of The Hands was an immediate success. — Michel Waisvisz
The Hands are a controller where there is some kinesthetic feedback (feeling from sensors in your muscles, joints, tendons). That means that it should enable the artist to gain better control of the sounds produced. We found it difficult to understand how gestures are used to create the sounds in the video above. Whether the artist knows what he is doing to a full extent is another story. Furthermore the device is all consuming and performing other tasks while wearing it is impossible. It is as easy to move around while wearing the Hands as it is to move around while playing a guitar or bass plugged into a an amplifier. This is not case for neither the Theremin or the Light Harp.
RadioDrum
Originally developed at Bell Labs in the mid-1980s as a 3-D mouse, the RadioDrum was supposed to replace the trackball mouse. Needless to say, that mission failed. Max Mathews, the father of computer music, was at Bell Labs at the time of its development and recognized the potential the RadioDrum had for music making. He carried on development of the device throughout his lifetime.
The RadioDrum works by a similar principle as the Theremin. The RadioDrum consists of two parts: a rectangular drum embedded with an antenna and two batons that emit two slightly different radio frequencies. The antenna on the rectangular drum then receives these frequencies and uses them to derive 3-D position of the head of the batons hovering above it.
The RadioBaton exhibits a similar drawback in terms of feedback the Theremin, at least when used to hover above the drum. Hitting the drum however will provide the performer with similar feedback as hitting a drum. As seen in the video above the device can be very expressive but requires the musician to be stationary while performing.
For more information about the Radio Drum and Baton can be found in this article by Engadged.
Buchla Lightning
The first Bucla Lightning was introduced to market in 1991. The product was composed of two parts. Two wands emit beams of infrared light to a box unit that receives the beam and instantly translates the signal into location, velocity, acceleration and direction in 2-D space. It obtains these signals by means of optical triangulation.
What is particularly interesting about the Lightning is its capability to map gestures, not plain values, to events. These events range anywhere from the triggering of notes to continuous change of timbre/sound. This mapping of gestures is obvious in the video below. Pay special attention to when the performer with the dark glasses (Don Buchla himself!) waves his left wand in a circular motion which in turn emits an event that creates a definite sound effect.
Unencumbered by wires, these wands provide complete freedom of movement within a performance space that can be as large as 12 feet high by 20 feet wide. — Buchla service webpage
The Lightning was a innovative and expressive gesture controller at the time, but it still had some drawbacks. While holding the wands does give kinesthetic feedback, trying to sense when an event is triggered is not easy without additional feedback. Receiving the infrared signal can also be cumbersome since it enforces a certain degree of immobility on the performer.
Gypsy MIDI controller
Animazoo is a company that made a variety of motion capture (‘mocap’) products for the film, media and games industries in the early 2000s. Sonalog was a sister company of theirs who released the Gypsy MIDI Controller to market in 2007. The Gypsy is essentially a wearable motion capture exoskeleton.
Each side of the skeleton has six sensors that detect shoulder, wrist, hand and elbow movements. A dedicated box gathers the sensor data, processes it and sends out as MIDI signal.
In terms of both feedback and mobility the same applies to the Gypsy as The Hands. The suit was however been criticized for being too expensive ($1000+), lacking dynamic range and having poor build quality. We could not find a place to buy the Gypsy today.
Arduino Air Drums
The Arduino platform brought about very inspiring maker projects. One of these projects that went viral is the Arduino Air Drums. By connecting an Arduino Uno to off-the-shelve accelerometers and light dependent resistors Maayan Migdal, the maker, was able to create a cheap yet rockin’ virtual drum-kit.
If you listen carefully to the drum sounds in the video above you can hear that he is indeed playing in real-time. The dynamics of the controller are however very limited as each part (sensors embedded in jump rope and sandals) is either in an on or off state. A simple YouTube search reveals that many makers have made made their own Arduino Air Drums. Some have even added more dynamic precision.
Hot Hands
The Hot Hands by Source Audio consist of a wireless sensor ring, charger, receiver base station, and sensor cable. The ring is powered by a rechargeable/replaceable lithium ion battery, with an average operation time of 6 hours between charges.
The Hot Hand has a similar accelerometer as the Arduino Air Drum placed in a wearable with built-in charging and wireless circuitry. The mapping of gestures to sounds is easily visible for the viewer and the dynamics of the device seem quite good. Again the major drawback is lack of feedback but also on-device controls so the artist needn’t turn to the computer to enable/disable the device, change what it’s controlling etc.
Mi.Mu Gloves
Now comes the project that we believe is the best designed gesture controller for music ever made. What we mean by best is the fact that the Mi.Mu Gloves have have a wide array of sensors, ranging from flex sensors to sense the flex of fingers to IMU (Inertial Measurement Unit) that measures orientation of hand relative to the earth. It has haptic and visual feedback from a LED. All this is packaged in a glove with promise that the performer has free use of his hands while wearing it.
A Kickstarter campaign under the title Mi.Mu Glove for Music was initiated on March 19, 2014, but the campaign was not a success financially. You cannot buy a Mi.Mu Glove today since they are sold out. The selling price per glove was £2,500 in pre-order.
Final thoughts
We hope that you are convinced that there is a lot to learn from the past when it comes to designing gesture controllers for music. Genki Wave has surely benefited from this research. We can’t wait to share it with you soon.
If you have something to add please feel free to comment on this post, we appreciate the input!
