CS 470 Wekinate Your World
Mewing Streak Break Detector
A humorous little gadget with little to no actual utility. “Mewing” as a concept refers to the placement of one’s tongue against the roof of one’s mouth to accentuate physical features such as a jawline. “Mewing” has been adopted by netizens as a part of a recent “looksmaxxing” craze—namely the desire to engage in activities such as self care and fashion to better one’s physical appearance. This lethal combo of “mewing” and “looksmaxxing” has been meme’d on the internet and is done to poke fun at the target demographic of these activities (young men hoping to employ these techniques to appear more sexually desirable) as seen below:
One could say that the so-called Platonic ideal for looksmaxxers is the giga chad personality who took the internet by storm in 2019 by appearing to be an archetype representing an ultra-masculine, physically attractive male. The chad in question is pictured below:
Methodology: Now that the context has been fleshed out, let’s talk about what the tool constructed actually does. The Mewing Streak Break Detector does what it says on the tin—it uses faceOSC to learn the facial cues assocciated with mewing and looksmaxxing and generates electro-pop club sounding music in response, and ceases to do so when the mewing facial cues vanish. The recording below shows me interfacing with the system. The flashing lights were played from my iPad as an artistic choice, and I am using the gigachad 2.0 filter available on the instagram app to accentuate chad-like features while mewing. The effect is intended to be hyperbolic and simultaneously humorus, with a healthy amount of “cringe” involved. The end result of the tool is that now you can tell when your friends break their mewing streak (and of course, call them out accordingly).
Learning Edge Assistant
The assistant uses faceOSC along with the Java Processing Language to create an interactive tool which monitors where a student is in their learning process.
The tool relies heavily on the concepts of Comfort Zone, Learning Edge and Unsafe/Danger Zone model forwarded by Arao, B., & Clemens, K. in their paper “From safe spaces to brave spaces: A new way to frame dialogue around diversity and social justice.” To briefly summarize the concepts:
- Comfort Zone: While remaining inside our comfort zone we are not being challenged, and thus we are not learning.
- Learning Edge: We call the edge of our comfort zone the learning edge. When we are on the learning edge, we are most open to expanding our knowledge and understanding — as well as expanding our comfort zone itself.
- Unsafe/Danger Zone: An extreme challenge, which may be distressing to the individual in question and may not be conducive towards arriving at a solution.
These concepts were posited in an interpersonal communication context, and I have co-opted them here in an academic learning sense. The Academic comfort zone is representative of facts and knowledge the student is comfortable with, and is able to provide and work with on demand. The Learning Edge is the outer envelope of the student’s knowledge—facts which must be struggled with to recall or problems which may challenge the student’s prior assumptions. The Danger Zone here refers to an extreme challenge, where the student has hit a roadblock and may not become unstuck without help (or a sudden burst of inspiration).
The process of learning or applying knowledge involves these three zones on some level, and the use of facial mapping software in the form of FaceOSC allowed facial expressions to be mapped to these different stages of learning.
Methodology: FaceOSC is hooked up to the Continuous Color Wheel example provided on the Wekinator.org walkthrough page, and the model was trained on various facial expressions of me struggling with some Leetcode problems online. I also recorded myself struggling with some problems and was able to “Human in the loop” identify what expressions if mine I thought most closely mapped to the various stages of learning. For example, I noticed that I only squinted when a problem was frustrating me (Danger Zone), and stick my tongue out pensively when wrestling with a problem (Learning Edge). This self-knowledge allowed me to fine tune my training of the model, and the final performance of the system involved me struggling to name all the countries in Asia with the color wheel model operating in Synch in a window to the right.
I was pleased to note that the color shifts were consistent with my mental process throughout the video: I start out confident and the mostly green and yellow color pallet reflects that. Later on as I struggle to recall more countries, some moments give way to frustration, with brief spurts of inspiration and relief allowing me to name yet another country (flashes of green).
Future Work: Such systems may be useful in online learning platforms such as code.org to see what part of the problem students struggle with the most. SAT, GMAT, and GRE test centers that require webcam access can also use this data to determine which problems on the exam students were challenged by. While the number of correct answers is a conventionally tried and tested method to discern question difficulty, test makers could benefit from determining which questions students found challenging but did well in, and which were challenging with low correctness scores. At the end of the day, it can serve as another dimension of analysis test makers can use.
Notes: The recording on Zoom regrettably is of low quality, and furthermore the presence of many open applications on my laptop means that the frame rate is quite low—therefore the color changes in the Processing window and the typing in the safari browser may not completely be in sync.
//This demo allows wekinator to control background color (hue)
//This is a continuous value between 0 and 1
//Necessary for OSC communication with Wekinator:
import oscP5.*;
import netP5.*;
OscP5 oscP5;
NetAddress dest;
//Parameters of sketch
float myHue;
PFont myFont;
void setup() {
//Initialize OSC communication
oscP5 = new OscP5(this,12000); //listen for OSC messages on port 12000 (Wekinator default)
dest = new NetAddress("127.0.0.1",6448); //send messages back to Wekinator on port 6448, localhost (this machine) (default)
colorMode(HSB);
// NOTE: The size of the window is the only major change
size(2000,3000, P3D);
smooth();
background(255);
//Initialize appearance
myHue = 255;
sendOscNames();
myFont = createFont("Arial", 14);
}
void draw() {
background(myHue, 255, 255);
drawtext();
}
//This is called automatically when OSC message is received
void oscEvent(OscMessage theOscMessage) {
if (theOscMessage.checkAddrPattern("/wek/outputs")==true) {
if(theOscMessage.checkTypetag("f")) { // looking for 1 control value
float receivedHue = theOscMessage.get(0).floatValue();
myHue = map(receivedHue, 0, 1, 0, 255);
} else {
println("Error: unexpected OSC message received by Processing: ");
theOscMessage.print();
}
}
}
//Sends current parameter (hue) to Wekinator
void sendOscNames() {
OscMessage msg = new OscMessage("/wekinator/control/setOutputNames");
msg.add("hue"); //Now send all 5 names
oscP5.send(msg, dest);
}
//Write instructions to screen.
void drawtext() {
stroke(0);
textFont(myFont);
textAlign(LEFT, TOP);
fill(0, 0, 255);
text("Receiving 1 continuous parameter: hue, in range 0-1", 10, 10);
text("Listening for /wek/outputs on port 12000", 10, 40);
}Poetry Interpretor
A tool which uses a combination of handPoseOSC and Wekinator to map hand gestures to lines from poetry.
Inspiration: The concept was partly inspired by musical conduction — much like an orchestra conductor can make gestures which prompt certain instruments to play, what if we had a tool which could prompt verses of poetry to play.
A major source of inspiration for this piece was the tradition of poetry mehfils (equivalent to the 19th century salon where artists could share their work in a group setting) in the Urdu Language. The poet would share their work in a crowd, and the audience would express their appreciation at verses by exclaiming and gesturing in approval. This sets up a sort of positive feedback loop between the performer and the audience, such that the performer is getting active cues from the audience which guide them on what verses to emphasize and repeat. An example of this can be found in the live rendition of Urdu poet Parveen Shakir’s Poem “Baat toh Sach hai Magar Baat Hai Russwai Ki” as performed here.
I wanted to emphasize the performative aspect of poetry, and so lifted lines from the aforementioned poem (which happens to be my favorite Urdu poem) and wanted to maintain the poetic structure while performing it in a new way. Alternative approaches of making the lyrics more “musical-sounding” such as frequency and pitch modulation were briefly considered, but I wanted to stay true to the source material. Therefore the project is me reciting the poem—just through hand gestures and audio mappings instead of words.
Methodology: Inputs from handPoseOSC were relayed to Wekinator through a hand-rolled relay inspired by the faceOSC example provided to us. Wekinator itself was run in Dynamic Time Warping Mode, with 6 gestures (corresponding to 6 verses of the poem) with a total of 43 examples recorded between all of the gesture-verse mappings. The gestures mapped were mostly supposed to mirror those that an actual audience member at one of these mehfils would do (barring the second last gesture which was contrived), and the actual playback of audio was achieved by linking the gestures with the gain of the sound buffer associated with the particular verse audio clip. So gesture -> gain is the low level abstraction, and the muting of all non-relevant sound buffers achieves the desired effect of one verse at a time.
Thoughts: I wish I had more time to train the model—having trial and error-ed extensively with continuous inputs, Dynamic Time Warping was a welcome change. There were brief hiccups with some gestures being too similar to each other, but all in all the stability of the system was brought within acceptable bounds with a lot of elbow grease and training time.
The transitions between audio can also come off as a bit harsh—some effects that could mitigate this could be incorporation of a fade effect instead of the dramatic “on/off” of the audio. More accurate audio splicing of the original sound clip could also yield less choppy results, and a cooldown timer between verses may be useful for cohesion purposes.
/* Code for creating the Wekinator to audio mapping */
// create our OSC receiver
OscIn oscin;
// a thing to retrieve message contents
OscMsg msg;
// use port 12000 (default Wekinator output port)
12000 => oscin.port;
// All inputs
SndBuf input => Gain g => dac;
0 => g.gain; // Initial gain setting
1 => input.loop;
SndBuf baat => Gain g_baat => dac;
0 => g_baat.gain; // Initial gain setting
1 => baat.loop;
SndBuf tera_pehloo1 => Gain g_pehloo1 => dac;
0 => g_pehloo1.gain; // Initial gain setting
1 => tera_pehloo1.loop;
SndBuf tera_pehloo2 => Gain g_pehloo2 => dac;
0 => g_pehloo2.gain; // Initial gain setting
1 => tera_pehloo2.loop;
SndBuf kaheen_bhi => Gain g_kaheen_bhi => dac;
0 => g_kaheen_bhi.gain; // Initial gain setting
1 => kaheen_bhi.loop;
SndBuf harjai => Gain g_harjai => dac;
0 => g_harjai.gain; // Initial gain setting
1 => harjai.loop;
// Set the audio file paths
"../audio/kaise_keh_doon.wav" => input.read;
"../audio/baat.wav" => baat.read;
"../audio/tera_pehloo1.wav" => tera_pehloo1.read;
"../audio/tera_pehloo2.wav" => tera_pehloo2.read;
"../audio/kaheen_bhi.wav" => kaheen_bhi.read;
"../audio/harjai.wav" => harjai.read;
// Listen for any incoming OSC message
oscin.listenAll();
// print
<<< "listening for OSC message from Wekinator on port 12000...", "" >>>;
// expecting 5 output dimensions
5 => int NUM_PARAMS;
float myParams[NUM_PARAMS];
// function to map incoming parameters to musical parameters
// fun void map2sound() {
// // time loop
// while (true) {
// // adjust input gain based on Wekinator output
// myParams[0] => g.gain; // Kaise keh doon gain
// myParams[1] => g_baat.gain;
// // advance time
// 100::ms => now;
// }
// }
// spork mapping function
//spork ~map2sound();
// continuous loop for audio playback
while (true) {
// wait for OSC message to arrive
oscin => now;
// grab the next message from the queue.
while (oscin.recv(msg)) {
// print stuff
cherr <= "RECEIVED: \"" <= msg.address <= "\": \n";
//printArgs(msg);
// Handle different OSC messages
if (msg.address == "/kaise") {
// Silence everything else
0 => g_baat.gain;
0 => g_pehloo1.gain;
0 => g_pehloo2.gain;
0 => g_harjai.gain;
0 => g_kaheen_bhi.gain;
// Play our verse from the start
0 => input.pos;
2 => g.gain;
} else if (msg.address == "/baat") {
// Silence everything else
0 => g.gain;
0 => g_pehloo1.gain;
0 => g_pehloo2.gain;
0 => g_harjai.gain;
0 => g_kaheen_bhi.gain;
// Play our verse from the start
0 => baat.pos;
2 => g_baat.gain;
} else if (msg.address == "/tera_pehloo1") {
// Silence everything else
0 => g.gain;
0 => g_baat.gain;
0 => g_pehloo2.gain;
0 => g_harjai.gain;
0 => g_kaheen_bhi.gain;
// Play our verse from the start
0 => tera_pehloo1.pos;
2 => g_pehloo1.gain;
} else if (msg.address == "/tera_pehloo2") {
// Silence everything else
0 => g.gain;
0 => g_baat.gain;
0 => g_pehloo1.gain;
0 => g_harjai.gain;
0 => g_kaheen_bhi.gain;
// Play our verse from the start
0 => tera_pehloo2.pos;
2 => g_pehloo2.gain;
} else if (msg.address == "/kaheen_bhi") {
// Silence everything else
0 => g.gain;
0 => g_baat.gain;
0 => g_pehloo1.gain;
0 => g_pehloo2.gain;
0 => g_harjai.gain;
// Play our verse from the start
0 => kaheen_bhi.pos;
2 => g_kaheen_bhi.gain;
} else if (msg.address == "/harjai") {
// Silence everything else
0 => g.gain;
0 => g_baat.gain;
0 => g_pehloo1.gain;
0 => g_pehloo2.gain;
0 => g_kaheen_bhi.gain;
// Play our verse from the start
0 => harjai.pos;
2 => g_harjai.gain;
}
}
}//----------------------------------------------------------------------------
// name: hand2wek-relay.ck
// desc: relay handOSC messages to Wekinator
//
//----------------------------------------------------------------------------
// our OSC receiver (from FaceOSC)
OscIn oin;
// incoming port (from FaceOSC)
8338 => oin.port;
// our OSC message shuttle
OscMsg msg;
// listen for all message
oin.listenAll();
// destination host name
"localhost" => string hostname;
// destination port number: 6448 is Wekinator default
6448 => int port;
// our OSC sender (to Wekinator)
OscOut xmit;
// aim the transmitter at destination
xmit.dest( hostname, port );
// just two of the many parameters
float THUMB;
float INDEX;
float MID;
float RING;
float PINKY;
float PALM;
// print
cherr <= "listening for messages on port " <= oin.port()
<= "..." <= IO.newline();
// spork the listener
spork ~ incoming();
// main shred loop
while( true )
{
// can do things here at a different rate
// for now, do nothing
// advance time
1::second => now;
}
// listener
fun void incoming()
{
// infinite event loop
while( true )
{
// wait for event to arrive
oin => now;
// grab the next message from the queue.
while( oin.recv(msg) )
{
// print message type
cherr <= "RECEIVED: \"" <= msg.address <= "\": ";
// print arguments
printArgs( msg );
// handle message
if( msg.address == "/annotations/thumb" )
{
// save
msg.getFloat(0) => THUMB;
// (optional) could normalize here
// Math.remap( MOUTH_WIDTH, MOUTH_WIDTH_MIN, MOUTH_WIDTH_MAX, 0, 1 ) => MOUTH_WIDTH;
}
else if( msg.address == "/annotations/indexFinger" )
{
// save
msg.getFloat(0) => INDEX;
// (optional) could normalize here
// Math.remap( MOUTH_HEIGHT, MOUTH_HEIGHT_MIN, MOUTH_HEIGHT_MAX, 0, 1 ) => MOUTH_HEIGHT;
} else if ( msg.address == "/annotations/middleFinger " )
{
// save
msg.getFloat(0) => MID;
// (optional) could normalize here
// Math.remap( LEFT_BROW, LEFT_BROW_MIN, LEFT_BROW_MAX, 0, 1 ) => LEFT_BROW;
} else if ( msg.address == "/annotations/ringFinger" )
{
// save
msg.getFloat(0) => RING;
// (optional) could normalize here
//Math.remap( RIGHT_BROW, RIGHT_BROW_MIN, RIGHT_BROW_MAX, 0, 1 ) => RIGHT_BROW;
} else if ( msg.address == "/annotations/pinky" )
{
// save
msg.getFloat(0) => PINKY;
// (optional) could normalize here
//Math.remap( LEFT_EYE, LEFT_EYE_MIN, LEFT_EYE_MAX, 0, 1 ) => LEFT_EYE;
} else if ( msg.address == "/annotations/palmBase" )
{
// save
msg.getFloat(0) => PALM;
}
}
// reformat and relay message to Wekinator
send2wek();
}
}
// reformat and send what we want to Wekinator
fun void send2wek()
{
// start the message...
xmit.start( "/wek/inputs" );
THUMB => xmit.add;
INDEX=> xmit.add;
MID => xmit.add;
RING => xmit.add;
PINKY => xmit.add;
// send it
xmit.send();
}
// print argument
fun void printArgs( OscMsg msg )
{
// iterate over
for( int i; i < msg.numArgs(); i++ )
{
if( msg.typetag.charAt(i) == 'f' ) // float
{
cherr <= msg.getFloat(i) <= " ";
}
else if( msg.typetag.charAt(i) == 'i' ) // int
{
cherr <= msg.getInt(i) <= " ";
}
else if( msg.typetag.charAt(i) == 's' ) // string
{
cherr <= msg.getString(i) <= " ";
}
}
// new line
cherr <= IO.newline();
}