Etude #3
Milestone
For the checkpoint, I experimented entirely with camera facial expression input. My end goal is to create an expressive musical instrument with the capability of controlling melody and harmony. To that end, I experimented with choosing 1–2 facial features, like mouth width and eyebrow height, and mapping them to musical features like pitch and volume (skipping the Wekinator AI training process). But I found that the individual feature values were inconsistent and hard to control. I hope to continue to work towards selecting individual, easy to control features for my end instrument.
For now, I’ve made a simple demo that recognizes facial expressions and plays either a happy or a sad clip (or nothing) depending on the facial expresison. I think the musical aspects of this demo work well with the visual aspects, since the emotion on my face is directly tied to the emotion of the music. They reinforce each other, which creates a multi-sensory experience that aims to be slightly humorous. Check out the demo at the bottom of this page.
Final
This time, I was able to implement the regression based musical instrument. While developing this instrument, I found that the video input was very finicky and noisy, and it was hard to get consistent and controllable musical sounds. So my general philosophy for this particular Wekinator was to design the sounds such that it would be really hard for the instrument to sound bad, regardless of the input. I had one Wekinator output, which was meant to control pitch. Instead of mapping directly to the frequency of my instrument, I instead quantized it to the pentatonic scale. I also added some stability against noisy inputs by only changing the instrument’s frequency if the raw input signal was non trivially different from the previous frequency.
I also spent a lot of time on the sound design for this demo. I was inspired by the end of Runaway by Kanye West. At the end of Runway, Kanye passionately sings with heavily distorted vocals. It feels very raw and emotional. I wanted to replicate that effect, so I added distortion to my instrument. For the live performance, I tried to emulate Kanye’s raw emotional singing style.
Check out my demo below (it’s called Instrument demo). Also check out the other Wekinator demos I made!
// ---------------------------
// Setup Sound
// ---------------------------
// setup reverb and filters
NRev reverb => dac;
LPF filter => reverb;
NRev reverb2 => dac;
.5 => reverb.mix;
.1 => reverb2.mix;
// connect the bar to the effects chain
ModalBar bar => ABSaturator sat => Gain g => LPF lpf => reverb2;
// setup bar settings
1 => bar.preset;
0.2 => bar.stickHardness;
1 => bar.volume;
4 => bar.vibratoFreq;
0.5 => bar.vibratoGain;
0 => bar.damp;
// setup distortion effect for the bar
150 => sat.drive;
1 => sat.dcOffset;
0.15 => g.gain;
2500 => lpf.freq;
// setup LFO for the pad
SinOsc lfo => blackhole;
2::second => lfo.period;
// setup pad
BlowBotl pad[4];
for (0 => int i; i < 4; i++){
pad[i] => filter;
pad[i].noiseGain(0);
}
// setup drums
NRev drumverb => dac;
.15 => drumverb.mix;
SndBuf snare => drumverb;
SndBuf kick => drumverb;
SndBuf hat => drumverb;
SndBuf open => drumverb;
"sounds/snare.wav" => snare.read;
"sounds/kick.wav" => kick.read;
"sounds/hat.wav" => hat.read;
"sounds/open.wav" => open.read;
kick.samples() => kick.pos;
snare.samples() => snare.pos;
hat.samples() => hat.pos;
open.samples() => open.pos;
1 => snare.gain;
1 => hat.gain;
1 => kick.gain;
1 => open.gain;
// ---------------------------
// Define Musical Constants
// ---------------------------
// Define the scale - MIDI notes for C major from C3 to C5
[0, 2, 4, 7, 9, 12, 14, 16, 19, 21] @=> int pentatonic[];
// define some chords to use later
[36, 64, 67, 72] @=> int C[];
[36, 55, 62, 64] @=> int Cadd2[];
[38, 65, 67, 72] @=> int Dm11[];
[40, 64, 67, 72] @=> int CoverE[];
[40, 64, 67, 71] @=> int Cmaj7overE[];
[41, 67, 69, 72] @=> int Fadd2[];
[41, 65, 69, 72] @=> int F[];
// ---------------------------
// Setup OSC Receiver
// ---------------------------
// OSC setup remains the same
OscIn oscin;
OscMsg msg;
12000 => oscin.port;
oscin.addAddress( "/wek/outputs" ); // Listen for binary messages now
<<< "listening for binary OSC message from Wekinator on port 12000...", "" >>>;
// ---------------------------
// Define helper functions
// ---------------------------
// Function to map a float between 0 and 1 to a scale index
fun float mapToScale(float val) {
// Map the float to the scale array length
Math.floor(val * 21) $ int + 72 => int midiNote;
// Find closest note in C major scale
pentatonic[0] => int closestNote; // Initialize with the first note
for (0 => int i; i < pentatonic.size(); i++) {
pentatonic[i] + 72 => int note;
if(Math.abs(note - midiNote) < Math.abs(closestNote - midiNote)) {
note => closestNote;
}
}
return Std.mtof(closestNote);
}
// Modified waitForEvent function
fun void waitForEvent(){
// Initialize the previous frequency
0 => float prevFreq;
0 => float prevRaw;
while( true ){
// Wait for OSC message
oscin => now;
while(oscin.recv(msg)){
msg.getFloat(0) => float rawFreq;
// control for noise in the raw freq signal; only update if the raw freq is significantly different
if (Math.fabs(rawFreq - prevRaw) < 0.04) {
rawFreq => prevRaw;
continue;
}
rawFreq => prevRaw;
mapToScale(rawFreq) => float freq;
// only rearticulate if the frequency has changed
if (freq != prevFreq) {
<<< "Received freq: ", freq >>>;
freq => prevFreq;
freq => bar.freq;
0.5 => bar.noteOn;
}
}
}
}
// play a chord
fun void playChord(int chord[]){
for (0 => int i; i < chord.size(); i++){
Std.mtof(chord[i]) => pad[i].freq;
0.5 => pad[i].noteOn;
}
}
// play the background music
fun void playBackground()
{
while( true )
{
playChord(C);
4::second => now;
playChord(Dm11);
4::second => now;
playChord(CoverE);
2::second => now;
playChord(Cmaj7overE);
2::second => now;
playChord(Fadd2);
2::second => now;
playChord(F);
2::second => now;
}
}
// Start the OSC receiver loop and play the backtround music
spork ~ waitForEvent();
spork ~ playBackground();
// Keep the program running
while( true )
{
1::second => now;
}// setup sound
NRev verb => dac;
.15 => verb.mix;
SndBuf bababooey => verb;
// load sound files
"bababooey.wav" => bababooey.read;
bababooey.samples() => bababooey.pos;
1 => bababooey.gain;
// OSC setup remains the same
OscIn oscin;
OscMsg msg;
12000 => oscin.port;
oscin.addAddress( "/wek/outputs" ); // Listen for binary messages now
<<< "listening for binary OSC message from Wekinator on port 12000...", "" >>>;
// Modified waitForEvent function
fun void waitForEvent()
{
while( true )
{
oscin => now; // Wait for OSC message
while( oscin.recv(msg) )
{
msg.getFloat(0) => float classMessage;
<<< "Received class message: ", classMessage >>>;
if( classMessage == 2 ){
bababooey.pos(0);
}
}
}
}
// Start the OSC receiver loop
spork ~ waitForEvent();
// Keep the program running
while( true ){
1::second => now;
}// Sound file objects
SndBuf happy => dac;
SndBuf sad => dac;
// Load sound files
"happy.wav" => happy.read;
"sad.wav" => sad.read;
// OSC setup remains the same
OscIn oscin;
OscMsg msg;
12000 => oscin.port;
oscin.addAddress( "/wek/outputs" ); // Listen for binary messages now
<<< "listening for binary OSC message from Wekinator on port 12000...", "" >>>;
// set the initial gains to 0
0 => happy.gain;
0 => sad.gain;
// Variable to keep track of the last played sound
0 => float lastPlayed;
// Modified waitForEvent function
fun void waitForEvent()
{
while( true )
{
oscin => now; // Wait for OSC message
while( oscin.recv(msg) )
{
float binaryMsg;
msg.getFloat(0) => binaryMsg; // Assuming the message is an integer (0 or 1)
<<<binaryMsg>>>;
// Check if the message is different from the last played sound
if( binaryMsg != lastPlayed )
{
// Update last played sound
binaryMsg => lastPlayed;
// Play the corresponding sound file based on the binary message
if( binaryMsg == 1 )
{
// Stop any currently playing sound
sad.pos(0);
sad.gain(0);
// Play sound1
0 => happy.pos; // Rewind to the start
1 => happy.gain; // Ensure the sound is audible
}
else if( binaryMsg == 2 )
{
// Stop any currently playing sound
happy.pos(0);
happy.gain(0);
// Play sound2
0 => sad.pos; // Rewind to the start
1 => sad.gain; // Ensure the sound is audible
}
else if ( binaryMsg == 3 )
{
// Stop any currently playing sound
happy.pos(0);
happy.gain(0);
sad.pos(0);
sad.gain(0);
}
}
}
}
}
// Start the OSC receiver loop
spork ~ waitForEvent();
// Keep the program running
while( true )
{
1::second => now;
}