I Build 2 Games To Make Machine Learning Fun
AI and Machine Learning are definitely powerful technology. However, the terms around these things are often abstract and obfuscating for people who don’t have much relevant background knowledge, which is almost everyone.
I started doing research about musical machine learning in Music and AI Lab in Academia Sinica last year. While exciting progress is being made, the research focus has been mostly on the model development part, rather than the presentation of the generation result.
I found that a lot of my friends who are artists and musicians are willing to know more about these technology. However, it’s still really hard for them to take the first step.
Gamification for Machine Learning
I want to demystify the abstract idea of using machine learning to create music. Playing game is a more interesting and engaging way to introduce new technology to people. I think gamification with a proper design can make musical machine learning easier to understand.
While I was investigating whether there is any prior works, I found a project called Semantris. It is developed by Google AI to showcase their Natural Language Processing (NLP) algorithms.
It’s a word association game that is powered by this technology. When you enter a word or phrase, the game ranks all of the words on-screen, scoring them based on how well they respond to what you typed. — Google AI
I was encouraged by their awesome work and started building mine for musical machine learning algorithms.
Sornting = 🎸 song + 🕵️♂️ sort
Some machine learning algorithm can learn that how to reproduce the data in the dataset using a lower dimension. Using the same mapping, the algorithm can somehow guess what the samples between two existing data are like, which is “interpolation”. My first project of using gamification for musical machine learning is Sornting. It’s a puzzle game based on a musical machine learning algorithm to interpolate different songs. The player has to figure out the original order of the interpolation between two different melodies, and the difficulties will increase as the game progresses.
I’m not trying to justify the result of the interpolation is perfect here. The user will find that some weird effect of interpolation while playing the game and listening to the melodies carefully. It will help the user to not only understand the model but also find out the weakness of model.
On the other hand, the game can also used as an analysis tool for researchers. For example, if more players are able to figure out the answer, it probably means that the interpolation is more musical meaningful for human, and vice versa. Also, the game was taken as a challenge for some musicians who thinks they have good listening skills.
. source code: https://github.com/vibertthio/runn
. demo: https://vibertthio.com/runn/
RUNN = 🏃♂️ Run (side-scrolling game) + 🤖 RNN
However, there is still a barrier for most of the people to enjoy playing Sornting. They have to know a bit about interpolation and some confidence with music listening. It made me consider the question: how to make everyone want to listen to the machine-generated music? When I discussed with one of my friend, Chris Donahue, he told me that the scores of music are similar to the screen of a side-scrolling game. I thought this might be a way to introduce new music to people who can play side-scrolling, which is almost everyone. I shared the idea to my colleagues at Taiwan AI Labs, and they also gave me a lot of useful feedbacks.
The second project is RUNN. It is “run” (side-scrolling game) plus RNN (Recurrent Neural Network). RNN are used to generate sequential data, like language and music. The user can play with the generated output without any prior knowledge about musical machine learning. Everyone can just play the game and enjoy the generated music. Also, they can keep the good vibe (the background beat) by controlling their avatar, or not. All the levels are generated realtime with a MusicRNN model.
. source code: https://github.com/vibertthio/runn
. demo: https://vibertthio.com/runn/
Conclusion
Both of the projects are not limited to a certain machine learning model. Also, there are a lot of forms of gaming which are potentially useful for the similar purpose. I want to make it possible for everyone to expressing themselves using music. Gamification of the idea is just an experimental method to approach the vision. Please tell me if you have any thoughts about the future of music technology and possibilities.
Thanks for the help from Music and AI Lab for the great modes, Magenta (Google Brain) for the awesome open source project (magenta-js), friends at Taiwan AI Labs for the brilliant advices.
About Me
Vibert Thio
Thio defines himself as a poetic technologist, playing around the disciplines of music, art, and technology. He aims to “democratize musical expression” and has developed a wide variety of projects to approach it, including musical game, interactive albums on the web, musical instrument, and art rehabilitation.* vibertthio.com
* mailing list
* github
* medium
