[CC Lab 21春] Mixed Reality Interface for AI-based Musical Expression vol.1
Hello.
I’m Ryogo Ishino, a student at Keio SFC, and CC Lab. This semester, I’ve been working on some projects that use MR(Mixed Reality) as its interface. Let me introduce the motivation and what we’ve done so far.
Contents
- What’s MR?
- Previous Works
- Our Motivation
- MR4MR: Mixed Reality for Music Reincarnation
- Future Works
What’s MR?
First of all, I would like to explain what MR(Mixed Reality) is. MR is like a hybrid of VR(Virtual Reality) and AR(Augmented Reality). Various data can be displayed in the space in front of you, and you can also interact with it. MR fuses the virtual world to the real world, and enables users to feel the virtuality even more real.
Previous Works
Bloom: Open Space
Brian Eno 2018
article(jp)
Features
- When a user does a “pinch” gesture (bringing his/her thumb and index finger together), a “bloom” appears.
- The pitch of its sound depends on the height it appeared at.
- The system decides which sound to be included in the output.
Drawing Sound in MR Space
Nagoya City University 2018
Features
- Users can “draw sounds” in front of them with the “pinch” gesture.
- The drawing action makes a sine-wave sound. The pitch of it also depends on the height.
- There’s a small object tracing the drawn line. Collision of those objects triggers an audiovisual effect. The type of the effect depends on which layer the collision happened in.
Our Motivation
Is there any kind of musical expression that can only be performed with MR? One unique feature of MR is that users can interact with not only virtual objects, but also real-world objects around them, such as walls, tables, chairs, bottles, and so on. By interacting with those objects, can’t we generate music which is unique or specific to that place? That’s how we started our new project called MR4MR: Mixed Reality for Music Reincarnation.
MR4MR: Mixed Reality for Music Reincarnation
The whole system can be divided into 3 sections: Hololens, Python, and Max for Live. In brief, Hololens is the MR interface, the Python part generates music continuously with MusicVAE, and the Max for Live part becomes the interface for the generator system and also outputs the generated MIDI data to play. Since I’ve been taking charge of the MR interface, let me focus on explaining that part.
In this project, we use Microsoft Hololens for the MR device. Hololens applications can be developed with game engines such as Unity and Unreal Engine. Here I chose Unity because there were more documents and examples than UE.
About the interface, first I wanted to smash a virtual object with my own hands and make it bounce off the wall or other real-world objects. To do this, I referred to Mr. van Schaik’s project which already implemented this system. Hololens recognizes real-world objects and cover them with a mesh after you start the app and wait for a few seconds. This mesh is the virtual wall that virtual objects bounces off. When a virtual object collides with the mesh or another virtual object, an OSC(Open Sound Control)message will be sent to the python server. This message includes data such as the object’s position(x, y, z), the object’s velocity(x, y, z), and the object’s ID(ex. sphere-1). In the python part, a sequence of these data (several consecutive collision data) is converted into MIDI and become the input for the music generator, MusicVAE.
In addition, there are two more functions. In order to prevent the virtual objects from flying so far away, there’s a huge invisible sphere covering the space around the user, and when a virtual object goes outside the sphere, that object will immediately spawn in front of you. Another function is, there’s a “special effect ball” (the orange one in the demo video) that randomly changes the virtual world’s gravity. There are five types of gravity: earth gravity, moon gravity, mars gravity, inverse mars gravity (which means the objects drop upward), and zero gravity. This function is for rearranging the pattern of collision by changing the physical condition and the objects’ behavior.
Future Works
Next update will mainly be on the generator part rather than the interface. Here are some of the issues.
- How to handle the sequence data neatly
- How to determine the beat
- Which sound (instrument) to use
- What kind of space can offer satisfying experience
Besides MR4MR, I’m also working on my graduation project, which also includes MR and AI-based music generation. This project has different point of view from MR4MR, so I’m looking forward to reporting that in the next volume.
