The Watson Beat: Using Machine Learning to Inspire Musical Creativity
TL;DR
IBM has released The Watson Beat as open source on GitHub. You can use it to create rich compositions from simple input melodies. Listen here for a sample of different types of music written by The Watson Beat [ 7 min 20 sec ], and listen here for The Watson Beat’s first album, “Space” [ 1hr 50 min ]. These compositions were arranged entirely by The Watson Beat, and are provided as samples of what is possible using this newly open sourced technology.
A Digression on Being Creative
IBM has given me the opportunity to be a small part of The Watson Beat project, which can create rich compositions in several styles, based on a simple input melody. It’s wonderful, yet, is it creative? The second definition M-W offers for creative is:
having the quality of something created rather than imitated
By this measure, I contend that The Watson Beat uses mathematically creative ways of imitating the rules of western music theory and applying them to compose rhythms and melodies, and its output compositions are not strictly creative. Let’s call this computational creativity. However, the music assembled by The Watson Beat software can be used in collaboration with human artists, and that output is most definitely creative. Machine Learning works best as a tool to help elevate the output of a human creative mind, not to replace it.
Teaching Music to The Watson Beat
Background
The Watson Beat is a project that began as the brain child of Janani Mukundan, formally of IBM Research. It has evolved over the years, and under Janani’s guidance several people have contributed to the project. IBM Watson has collaborated with several people, including musician Richard Daskas to compose the music for an IBM/Red Bull F1 commercial.
The Watson Beat code employs two methods of machine learning to assemble its compositions; reinforcement learning that using the tenets of modern western music theory to create reward functions, and a Deep Belief Network (DBN) trained on a simple input melody to create a rich complex melody layer.
Rhythm
The rhythm section of The Watson Beat is responsible for the drum beat, bass line, and chord progressions of the composition. There are many parameters that you can pass into The Watson Beat, including time signature, tempo, and mood [2]. Using these parameters, and an understanding of music theory, The Watson Beat selects the next note to be used in the progression (the action). It utilizes Reinforcement Learning (RL), a type of machine learning diagramed below. If the note chosen for the next line in the progression is a standard concept (like a 1–4–5 in G), then the system is rewarded, and if the next note is odd (in a music theory sense), then system is penalized. This method steers the RL Agent to imitate, with a sense of randomness, a rhythm and progression for the composition.
Melody
The melody is derived using the human provided simple input melody, which is passed to The Watson Beat as a 10–20 sec midi file. This input melody is used to train a neural network, which is a DBN of stacked Restricted Boltzmann Machines (RBM). The resultant machine learning model then outputs music that uses the world of the input melody as a sort of framework and randomly generates melodies inside of the rules of this framework. Deterministic rules were added to make the selected notes for the melody to fit the same western music theory constraints as the rhythm section.
Collaborate and Create Compositions
Running the Source Code
If this all sounds like something you want to explore on your own, you can clone the project on GitHub today (https://github.com/cognitive-catalyst/watson-beat). It all runs locally on your machine, and takes in parameters and the inspiration melody as inputs and outputs a directory full of midi files. If you don’t have any midi files laying around, we have added some open domain tunes to the source code repository to help you get started. Every time you run The Watson Beat, you get different output, so I encourage you to try it several times with the same input midi and “ini” parameter file and see what happens.
MIDI to Music
The Watson Beat creates a whole directory full of midi files, but unless you are used to electronic music composition, like me, you might see this directory full of small midi files and wonder exactly what you are supposed to do with it. This is where your collaboration with The Watson Beat begins. Electronic music is composed in a digital audio workstation (DAW for short). To listen to the music assembled by The Watson Beat, you must load the midi files into a DAW, apply a virtual instrument to each track, then mix the track to create an mp3 file with all of the layers.
At this point, you can use as much, or as little of The Watson Beat output as you’d like. For instance, if you loaded a song, and you liked the chord progression and the melody, but not the drum, you could delete the drum track and add in your own creations.
Listen
Using the output of The Watson Beat, and systematically applying virtual instruments to each track (I used REAPER), I assembled some videos to give you a quick glance at what is possible with this technology.
In this first video, I play the short inspiration melody provided to The Watson Beat as input, and follow it with the output files from The Watson Beat rendered into a musical composition. There are a variety of styles that The Watson Beat can use, and I provide an example of “Anthematic,” “PopFunk”, “BomMarch,” “Space,” and “EDM” applied to four public domain melodies.
Then, I used The Watson Beat to create an entire album, “Space,” almost two hours of extremely interesting, if not quite human sounding music. It is uploaded to youtube in 9 parts.
Contribute
The great thing about open sourcing software code, is that if you have to opportunity not just to examine and use the source code, but actively make it better. I encourage you to create your own forks, and submit pull request if you create a new mood, optimize something, or in any way make incremental improvements on the code as it exists today.
Thanks!
I had such a great time getting to know this project. I’m an amateur musician and a professional data scientist, and never before had these two paths crossed before. A giant thank you to the IBM team that originally wrote this project, especially the founder Janani Mukundan. Also thanks to IBMers Elizabeth Transier, Krishna Srinivasan, and designer Rob Harrigan for the cool poster. And a humongous special thanks to Richard Daskas for teaching me the ways of REAPER.
