The Harmonic State : audio case study
Web based sequencer for the interactive web experience
Watson is IBM’s AI system for business. Together with GPJ and ActiveTheory we at Plan8 got the chance to create an interactive game experience to show the capabilities of Watson.
Modifying the music in real time
The core concept of the site is to show how Watson can help a business to bring order in something disorganized. The interactive visuals and sound should reflect that change as the game is progressing. Each level starts in dissonance and finishes in harmony as the user collects orbs representing insights from Watson.
We set up a list of musical parameters which we thought would be interesting to play around with to transition between those two states.
For some of the transitions we could use real time effects like filter and reverb. But for others, like “detune” and “unstructured”, we had to affect the playback of individual samples. This meant that we couldn’t use a pre-rendered track.
We usually use web sequencers when building beat maker sites where users create music themselves. For experience sites and games we mostly work with pre-rendered tracks or sequencers with larger steps like a bar or a fourth note. This time we needed to have each note exported separately to be able to change timing and pitch in real time.
Controlling tone js with reaper
Over the years we have been trying to figure out the smartest pipeline for exporting and importing audio files when working with interactive audio projects. A big project could have hundreds of files which need to be exported from the DAW, converted, updated and imported to the project build. Doing that manually takes an insane amount of time and often leads to mistakes and confusion.
We recently switched to Reaper as our main DAW when composing music. One big difference with Reaper compared to other DAWs is that it can be extended with custom scripts.
When we started this project we looked into how we could export information about the musical arrangement directly from Reaper. The information we wanted to export was which sound files were used, where on the transport timeline they should play and how the sound files were grouped in folder tracks.
We ended up creating a custom reaper action which outputs the information to a json file. The python script uses Reapy which is a wrapper around the ReaScript Python API. That in combination with Reaper’s “Region render matrix”, which allows you to easily render all regions as separate files, makes Reaper a great choice for interactive audio production.
Next step was to use the exported json to set up a sequencer to play in the browser. Using the web audio api framework tone.js that was an easy task. Tone’s Part component does exactly what we want and can use the timing exported from Reaper to playback each sample at the right time.
Syncing the game to the music
The brief stated that the music should play an important role in the game. So we wanted the orbs you catch in the game to appear in front of the catcher in beat sync. A task that was complicated by the fact that the games were using a variable timescale which controls the speed of the game. The user also has the option to slow down the game at any time and play in slow motion.
Luckily we had the music broken down in sixteenth note parts in our sequencer. This allowed us to directly link the game’s timescale to the sequencer bpm. The result is a music track that follows the game tightly, keeping the orbs in sync with the beat.
With the music playing in this granular sequencer we also made it possible to affect the pitch independently from the tempo. We used this in the game by starting the music slightly pitched down and slowly ramping the pitch to normal as the game evolved.
About Plan8
Plan8, a design agency creating music and sound for brands, products and experiences. Plan8 operates in the intersection of audio, emerging technologies, brands and art. Our team of consists of music and sound designers, technologists and audio strategists. Founded 2008. We have offices in Stockholm and LA and work globally.
Follow our work on IG: @plan8music WEB: www.plan8.se
