[CC Lab 20秋] “x-magic” Project — Process of Creating a Platform for Magical Audiovisual Performances —
Hello.
I’m Ryogo Ishino, a student at Keio SFC, and CC Lab. Since April 2020, I’ve been working on the “x-magic” project. In this page, let me introduce what I’ve done this semester.
Contents
- My Previous Work
- Prior Research
- My Work for This Semester
- Demo
- Review & Future Work
My Previous Work
The concept of this project and what I did last semester are written in my previous report. Please check it out before reading further!
Prior Research
In my last report, I introduced three examples that can be “magical” in musical context. Here I will show three more examples this time in visual context.
- Drawing Sound in MR Space (DSMR) project proposes a multi-user interactive audiovisual art system with mixed reality.
- herering is a 3D ambient audiovisual installation art that expresses simulated chromesthesia experience.
- パラレルインク(Parallel Ink) is an AR app for iOS that you can repaint the world in front of you with your fingers.
The visual arts of these works are triggered by the player’s hand position and gesture. However, these visual effects are not much related to the sound. In the first example, the pitch of the sound gets higher linearly with the height of your hand position in real world. In the second one, the pitch corresponds to the color as described in previous study on chromesthesia. Other elements of sound, such as intensity or timbre, are not mainly considered here.
With this in mind, I set up my mission to establish a new way of visualization and sonification for audiovisual experience, considering not only the pitch but also other elements.
My Work for This Semester
First I started to make a new device as a controller, to replace Joy-Con in my previous work. Here are the main parts I used.
- Raspberry Pi Zero WH
- PiSugar (battery)
- Universal board
- MPU-6050 (accelerometer & gyroscope)
- FSR400 (pressure sensor)
- MCP3002 (A/D)
I connected these sensors to RPi referring to these websites (in Japanese).
Although I made this device, and it actually worked, I decided not to use this for now due to technical reasons. Instead, I’m now using Microsoft Azure Kinect for all the input to control the system.
With this device and TouchDesigner, you can easily acquire 3D position data of your hands, neck, shoulders, knees, ankles, and so on. The acquired data are used for producing visual effects handled by TouchDesigner, and sound produced by Max for Live devices on Ableton Live. Position data are sent to Max over OSC.
The usage of data is like this:
- Left hand xy coordinates
Control the visual effect and the timbre produced by NSynth
- Right hand x coordinate
Control the pitch
- Right hand y coordinate
Control the velocity
- Right hand z coordinate
Control the duration
- Flip your right hand (show the back of your right hand to the camera)
Turn on the Gesture Effect Trigger (GET) Mode
- Left hand xyz acceleration (while GET mode is ON)
Gesture recognition
Sound System
The sound system is almost the same as my previous work. The difference is the mapping of input data and sound control, since the controller (Joy-Con) is replaced by Azure Kinect.
Visualization
As a prototype of visualization, I implemented fluid simulation on TouchDesigner referring to ↓
At the moment, this visualization system is not reacting to any elements of sound, which is one of the key points of this project. I will solve this remaining task by next publication.
Demo
Review & Future Work
As you’ve seen, my progress of the project wasn’t enough this semester. It just ended up creating only a platform for future development. Before working on visualization, there are several things to do research on, and some issues to reconsider based on some feedback I received.
- The history and facts of magic
- How people recognize phenomena as magic
- The goal of this project (as a performance, or as a product?)
- Is it an instrument? Or an installation?
- Is it just made for me? Or can it be played by anyone? Or maybe a toolkit that anyone can create their own magic wand?
Let me give some time, and I’ll try to bring a conclusion by the end of spring break. Don’t miss it!
