AI Development Through Art
How I built a weather-driven music generator with AI software using Wekinator — and what this tells us about the merits of experimentation in software development
--
By Robert Greene, Senior Software Engineer at BCG Digital Ventures
In my work as a Senior Software Engineer at BCG Digital Ventures, I build digital products. Outside of this, I’m also an artist. Although these fields might not seem related, I’m always trying to bring them together to see what’s possible.
At DV, we have a very thorough innovation process, where we come up with new ideas, create mockups and test them before deciding which direction to go in.
However, when we’re working with AI concepts, it’s difficult to inject this technology into the prototypes we build. Through experimentation, I think I’ve found a way to limit the development time so that we can start to prototype our AI-driven ideas with minimal overhead, and therefore provide even more comprehensive and impressive prototypes to our clients.
I’ve been exploring how to use AI as part of my art practice. By experimenting with software called Wekinator, I’ve been able to build AI software without the hours of development it has previous required. Here’s how I did it.
Machine Learning for Musicians and Artists
I recently had the chance to take a class titled Machine Learning for Musicians and Artists, taught by Goldsmiths, University of London lecturer Dr. Rebecca Fiebrink. Dr. Fiebrink is the creator of an open source program called Wekinator, which allows users to build new interactive systems by demonstrating human actions and computer responses instead of writing programming code.
During the class, we learned about different algorithms for machine learning, and different approaches to creating models that work with intended functionality using Wekinator. For a final project, I decided to experiment with creating a piece of software that would create music based on the earth’s weather.
Getting Started with Wekinator
For those who aren’t familiar with Wekinator, here’s how it works.
Wekinator takes a set of input features using Open Sound Control (OSC) messages. OSC is a protocol for communicating among computers, sound synthesizers and other multimedia devices, optimized for modern network technology. As the inputs change, you can start to train the model for a set of outputs to feed back into the OSC pipeline. When your model is trained, you can run it and start to see the results of machine learning.
Each Wekinator project requires an input and an output program that can communicate with Wekinator. These programs can be bundled into the same package, or be two completely different things. If you want to explore Wekinator yourself, there’s a fantastic example kit that contains patched programs ready for use with Wekinator.
My Project: Wekinator Earth Sounds
My project, Wekinator Earth Sounds, enables artists to choose a geolocation, and have the weather data of this geolocation influence a musical output.
I built the project on React, NodeJS, Socket.io and Electron. It uses websockets to communicate with Wekinator and the application’s frontend. The program contains a Google Map that the artist clicks around to receive an area’s coordinates. Once the longitude and latitude are set, a request for the weather is called.
The program then gathers weather information by sending the coordinates to the API of Open Weather Maps. This service provides the program with eight input features: current temperature, humidity, minimum and maximum temperature, pressure, cloudiness and wind speed. Wekinator will consume this data once the artist applies the feature within the program.
The model is fairly large: it contains 68 different outputs. Each output has a set of classifications that range from one to 235 for the first four outputs and one to two for the remaining 64 outputs. These outputs control different parts of the MIDI loops. The first four outputs control the type of instrument to play for a given element of the song. The remaining outputs toggle the step controls. One means off, two means on. There are 16 steps per track, which gives the artist room to make slower or faster, simpler or more complex loops.
Explaining and controlling the MIDI controls from Wekinator is fairly difficult due to the number of outputs and how they are presented within the program. In order to simplify this process for the artist, Wekinator exposes an endpoint that can be called using OSC that will automatically set the outputs as the artist moves through the Wekinator Earth Sounds program. As you change the MIDI controls in the app, Wekinator will automatically set the corresponding output values.
Training the Model
Next, we want the model to learn and determine the best values to play according to the weather and the training set provided.
As no location has exactly the same weather conditions, we need to provide the model with enough training examples to fill the gaps — around 20 examples per country works well enough for experimentation purposes.
To train the model, we turn on training mode in the Wekinator Earth Sounds program and record within Wekinator. The artist sets the MIDI loop, selects a location on the map and hits the ‘apply’ button within the Wekinator Earth Sounds program to add a training example into Wekinator.
Once the artist has recorded enough training examples, all that’s needed is to train the model by clicking the ‘train’ button within Wekinator. Once training mode has been turned off, we can see the results come to life — the weather of any selected location alters the MIDI data, and thereby the musical output.
The Results
Once the model has been sufficiently trained, we can start to click around the map and see how the MIDI loop effects sounds according to the location’s weather. Clicking ‘play’ within the Wekinator Earth Sounds program starts the MIDI loop, and clicking on a location on the map changes the loop.
For example, in my training set, warm weather is set to play a single snare hit per measure. As I select locations with colder weather, the loop becomes more intense. If something in the model sounds incorrect, we can add more training examples or retrain the model by deleting the previous examples within Wekinator. We can also change the algorithm per output and even focus on certain outputs, such as humidity.
You can view a demo of my training set here, as I click around the map to change the musical dynamics:
If you would like to learn more about Wekinator, or take the machine learning class, check out Kadenze, a hub for learning about creative computing where you can find classes from top universities and professors from around the world.
Applying What I Learned
Building prototypes is an essential part of creating any new business or application. This process can be really expensive and time-consuming, so anything we can do to make them easier and more efficient is welcome. With AI becoming more market-ready, we need our prototypes to include this technology, but AI is currently one of the most difficult and time-consuming technologies to implement.
As engineers at BCGDV, instead of going through the arduous process of building out AI models ourselves, we can use Wekinator to validate these concepts with far less overhead, saving us development time, money and resources. Using Wekinator as a prototyping tool allows us to create smarter prototypes for our clients and explore new applications. Moving forward, I can see us using Wekinator to validate our innovative AI concepts before passing them through to our engineering teams to build. This also helps us to provide clients with real examples of what the machine learning output could look like without us much time spent on the model.
I’m excited to see how this will be used at BCGDV as machine learning and AI start to become the standard within our innovations. As an artist and engineer, this program has been a real game changer. Being able to easily experiment with complex concepts and tie them into the latest, most bleeding-edge hardware is an exciting possibility for my new projects. I can’t wait to see what comes out of this amazing tool.
Without my experimental work as an artist, I would not have been aware of the possibilities Wekinator provides — exploring new technologies for fun outside of their practical application is a vital part of providing innovation value. As engineers, we should always be open to the possibilities we can discover through experimentation.
Interested in working with us at BCGDV? Want to find out more? See our current vacancies.
Find us on Twitter @DV_Engineering.
