Mixing Music With Machine Learning
My quest to create smooth background music for coding and concentration, automatically
In my day job, I work as a software developer in a large open plan office. There is a lot noise around me most of the time from co-workers chit-chatting. To be able to concentrate on coding, I wear noise cancelling headphones and listen to music while I work.
So far so good, the question is: which music to listen to? There are many playlists available on Spotify, YouTube, etc. There are even musicians creating music specifically for coding.
What makes good background music?
I’ve used many of these options drown out distractions. Some work well for me, some not so much.
There are three basic requirements for me personally to be able to concentrate well:
- No vocals — if someone is singing, I’m bound to listen to the lyrics instead of concentrating on my work
- Slow tempo — research indicates that a tempo of 60 to 80 beats per minute is ideal to stimulate alpha-waves in your brain, which is useful for concentration
- Repetition — when I listen to music, I like action, dynamics, surprises, interesting musical ideas; when I use music, however, I find repetitive, almost boring music to be most helpful
The age of Reason
Aside from my day job, I’m a DJ and musician, creating electronic music.
I use a software called Propellerheads Reason to experiment with synthesizers, sound effects and drum machines, come up with new sounds and sequences.
Over the years, I have created hundreds of Reason loops on my hard drive. These loops are short pieces of electronic music, typically drums, bass line and some synth, that repeat infinitely after a couple of bars.
One day at work, when I got tired of listening to the same Spotify playlists again and again, I got the idea of just hitting “play” on Reason and listen to one of my loops to drown out background noise while working.
This works fairly well:
- there were no vocals
- I can adjust the tempo to my liking
- it is very repetitive, thus non-distracting
Have you tried turning it off and on again?
Then again, it is a bit too boring to listen to the same eight bars of music for hours on end. I thought about arranging one of my loops into one very long piece, with minute variations over time.
I had done something similar with this 30-minutes-track, “jogging music”, which I had created for my morning runs:
But I wanted not 30 minutes, but ideally 8 hours and more of music! That would be a lot of work, indeed.
Then it hit me: why bother spending so much time and effort arranging and tweaking a track until it provides background music for a whole day of coding, when I can write a program that does this for me?
My basic idea:
- Create a Reason loop with a lot of different channels (drums, percussion, bass, synth, etc.)
- Write a program that turns these channels off and on by random every time the loop repeats — this is commonly known as a technique called “muting”
In the video above, you can see the mixer view in Reason. The red buttons on each channel turn it off, muting its audio signal. By using these buttons while the music is playing, you can create an interesting, compelling mix of the music, similar to what a DJ does when they are performing a live set.
My program would take the role of that DJ and create an endless mix of my reason loop.
Get yourself connected
How can my program tell Reason what to do? Reason supports the Musical Instrument Digital Interface standard (MIDI) — just like every electronic instrument and music software since the early eighties.
Pictured above are the MIDI jacks of the Yamaha DX-7 synthesizer, pretty much the defining machine used in the eighties. Through these three jacks, musicians were able to plug in MIDI cables to control other synthesizers through the DX-7’s keyboard, record what they played on the DX-7 with a sequencer, or have that sequencer make the DX-7 by itself.
Today, you rarely use MIDI cables anymore. Devices are connected mainly through USB, or everything is happening inside one’s computer without any external devices. The MIDI protocol, however has stood the test of time and is still key in connecting (software) instruments.
So this is what my program will do: play Propellerheads Reason through MIDI.
The Secret Sauce
My first attempt at automating music was a MacOS app I had called SceneScreen.
It allows the user to set up up to 16 “scenes” that MIDI commands to change the muted channels and sound parameters of a track running in Reason. It has an “autopilot” feature that makes it change the scenes randomly.
It was a fun side-project, but I abandoned it after a while, because I discovered one thing in the process of making it:
Letting the computer decide by random how to mix music gets boring quickly and produces unnatural results.
It takes human creativity to make a compelling, interesting mix. A software that automatically mixes music should at least mimic this spark of human creativity.
How to do that? The answer is machine learning.
Machine learning is a programming technique that is revolutionizing software development. It lets computers recognize your friends in photos, dream up realistic portraits of people who don’t exist, lets you talk to Alexa or Siri and yes, composes music.
This technique is so powerful that people confuse it with artificial intelligence, which to this day doesn’t really exist yet. Computers cannot think or be creative, but with machine learning, they can do a pretty good job acting as if they could.
With machine learning, in a nutshell, you collect data to create a model, then write a program that tries to create data that matches that model. In a long series of attempts, various parameters are automatically tweaked until the desired result is achieved.
This would be the missing link, the secret sauce for my “background music generator DJ whatevs program”:
- Program runs a random combination of channels and sound parameters of my Reason loop
- Listeners provide feedback if they like what they hear
- This trains a machine learning model
- The program uses the collected data to “learn” which channel combinations and parameter settings work well
- The program creates an increasingly more pleasant music mix
Zapper… what?
At some point I realized that “background music generator DJ whatevs program” is not such a great name, so I decided to call my love child Zapperment. The word doesn’t really mean anything, but I think it sounds cool both in English and German.
The company I work for encourages its employees to regularly try out new technologies and explore new ideas in what we call Innovation Days.
During these Innovation Days, together with two colleagues, I created a first working prototype of Zapperment.
Zapperment is a web server, written in JavaScript, based on Node.js, with a React single-page-application as frontend, using socket.io for real-time user interactions and brain.js for the machine learning part.
What this technobabble means is: you can open up a URL in a browser on your computer or mobile phone, listen to the music and provide feedback how you like it by clicking “boo” or “clap” buttons.
Here’s a video of the first Zapperment prototype:
The music changes every eight bars. We can see the user clicking on the “clap” and “boo” button, and so does another user who is also connected from another computer. At one point (around 1"05), the two are in disagreement — one likes the currently running loop variation, the other hates it, causing both clapping hands and piles of poo to rain from the top of the browser window.
The prototype still uses machine learning in a very primitive, “I have no idea what I’m doing” way. Nevertheless, my colleagues and I were surprised how well it works. After training the model by listening and clicking those clapp and boo buttons for an hour or so, the mix created by Zapperment started to slound less random.
The road ahead
Unfortunately, you, gentle reader, are not able to try out Zapperment by yourself just yet 😢. I need to find a way to run it on a server in the cloud, with a proper web domain. Coming soon…
Also, there are still some technical challenges to surmount:
- When listening to the music through the browser, there is a long delay between Reason creating the signal and the user actually hearing it, so when the user clicks the applause button, many times the software “thinks” they are applauding to the current loop variation, while in reality, they are still hearing the previous one
- Timing stability — as you can tell from the demo video, the tempo of Reason is unstable, varying more or less
Besides these performance issues, a more interesting problem to solve is improving the classification of the music that is playing and turn it into a more fine-grained model to feed to the machine learning algorithms. Currently, I only store the information which channels were turned on, some synthesizer settings on one of the bass channels and the rhythm pattern the percussion is channel is playing. I want to come up with a way of properly describing the music playing in the various channels to my “artificial intelligence”, so that it can better transfer its trained model from one Reason track to another.
Stay tuned…
By the way, if you’re a technical person, you can check out the source code of my prototype on GitHub.
Acknowledgements
Thanks to my colleagues Kris Urbas, who wrote most of the machine learning code, and Roman Tayupov, who created the flying poo animation.
Thanks to you for your interest, for reading this far! I’d love to hear your feedback in the comments section 👇🏻 or on Twitter.
