Interactive Musical Video System: Synesthesia and CPU Update
This week I have been focusing on staying healthy by getting enough sleep, getting vitamin D3, walking, etc. I now only have to edit and further mix and master the song I am working on because I am pretty much done with the sound design and melodies I want to incorporate. My electronic drum set’s audio sometimes drops out for a second and then comes back when I am playing, but, as of now, this still does not affect my project much. I have also become better at beatboxing basic drum beats to better use the trigger feature in the Dubler Studio Kit.
Because my project has similarities to TheSushiDragon’s Twitch stream, I have gained more insight into how limited I may be technologically in terms of latency and CPU power. I also have decided that I want to focus more on video performances due to being in lockdown. TheSushiDragon uses 3 or 4 computers to run his setup successfully and his CPU power percentage still reaches around 80–90% as shown in the top right corner of a recent stream of his I was watching. I think this is because OBS, which is the main software he uses to stream and trigger visuals and live edits, takes an enormous amount of CPU power when triggering video files as I have tested in the past.
During a recent stream, however, TheSushiDragon switched to the Synesthesia VJ program, which is similar to Zwobot with different unique features, which understandably requires less CPU power. I downloaded the demo of Synesthesia because, based on TheSushiDragon’s stream, it seemed to have interesting live video editing capabilities using one’s webcam as the video input. I was very excited about discovering this as focusing more on video and live editing, DJing, and drumming can benefit immensely from more comprehensive live video editing than what is possible with Zwobot. From testing and researching Synesthesia, this software allows one to control how the animations move with motion through one’s webcam and live edit through with a webcam as the video input. For example, if I move my hand up, the VJ loop may turn a different color where my hand was or move in the direction of my hand surprisingly with little latency.
Synesthesia is a separate program from Ableton Live as opposed to Zwobot which is directly incorporated into Ableton Live tracks so it may require less CPU power. More testing will be needed, and I am still confident that expanding my project in the ways I have been results in a live interactive system that I can use based on my CPU power and latency limitations.
