A story about an inclusive music instrument and how we can program expression.
Last week I participated in a music hackathon at Spotify. The goal of the hackathon was to design music games. Participants were presenting their own ideas and projects and formed groups and some of us got very interested in the challenge launched by Danielmusic.org center where they do therapy through music for disabled people which was founded by a musician who lost partially his mobility.
As it happens I was carrying with me a bare conductive touch board (always have a board on you). I immediately thought of this video I saw where people were using the same board to enable disable people to play music.
I shared my idea of building similar instruments with the Danielmusic team and the other interested people and we start to explore multiple design scenarios.
The team representing the music center gave us the following constraints based on their experience of running musical activities at the center:
- build an extremely collaborative musical game (their students get stressed if they compete)
- enable users to play with any body parts (they have students that could only play with their tongue)
- make it extremely visual and intuitive(colors, big controllers)
- allow people to practice for it at home
- allow people to match beats and patterns
- make it resistant and unbreakable.
- make it customizable in terms of sounds/controllers .
- make it safe.
- last but not least always play Michael Jackson when in trouble as students love him :)
Those are a lot of requirements for an afternoon prototype but we were very motivated and we decided to give it our best.
We formed a team of six people: 1 ux designer, 2 musicians, 3 developers: 1 front-end, and two back-end (myself included here). It was the ideal team for this project.
We quickly realized we won’t be able to fulfill all the requirements at once and we decided to pick our battles and prioritize our game core goals. Based on the initial list of constraints we decided:
- Build a musical game where students have to match sounds and colors by recreating patterns that were played for them. Something like Simon Says but with music.
- To make it collaborative we decided to use the browser and websockets so more people can play music at the same time on the same page. This would also make it responsive and portable to phones, tablets which they are currently using in the center.
- We will use the touch board and capacitance actuators to enable the students to play music with any conductive object/liquid.
Final demo:
How we made it
- Set-up the bare conductive touch board with all our computers: http://www.bareconductive.com/make/setting-up-arduino-with-your-touch-board/
- Controller app built with express.js and socket.io to send data to a web game (the git repo)
- Send the data from the board to our app with the node serial port library: (very similar to the ruby serial port gem I showed on my last post) https://github.com/voodootikigod/node-serialport
- Web game built in javascript (http://phaser.io/tutorials/making-your-first-phaser-game)
- Sounds made with logic software.
- Colors from here: http://www.colourlovers.com/palette/3860804/Winter_Wedding_6
- Final repo code: https://github.com/stefania11/MIchaelSays
Node SerialPort library
Example of serial port call from node.js in our musical game.
The node-serialport library provides a very simple interface to the low level serial port code necessary to program Arduino chipsets (used also by our Bare Conductive touch board), X10 wireless communications, or even the rising Z-Wave and Zigbee standards. Full break down of why and how it works available here NodeBots — The Rise of JS Robotics.
Socket.io for seamless collaboration
Socket.IO enables real-time bidirectional event-based communication.
It works on every platform, browser or device.
Example integration with express.js.
Lessons learn and next plans
We had a blast working on this project mainly because we were so different and we saw how all our skills could come together and create a immersive experience.
Choosing and creating the right sounds, running to buy string cheese so we can use it as an instrument and figuring out how websockets and game engines work on the fly made us laugh so much and create a safe space for experimentation where we could push our boundaries, while being motivated by higher goal.
We plan to visit the music center and test our game with their students and see how we could continue to program musical expression and make it accessible to everyone.
@Stefania_druga, September 10, 2015
