CS 470: Music and AI – Etude #1
9 min readJan 20, 2024
Introduction to the Assignment
In our class CS 470: Music and AI, we were given a creative challenge: to create poetry using AI. Not just any poetry, but “poems in time” that intertwine text, sound, and the dimension of time, leveraging rudimentary AI as our brush and canvas.
My poems were built with the ChucK programming language, specifically the ChAI extension with Word2Vec capabilities. The mission was to craft experimental poetry using this AI-driven language tool. The task involved generating text through Word2Vec, mapping these words to sound using various parameters like pitch and volume, and orchestrating the entire ensemble in a temporal flow.
We were tasked to create two distinct poetic works. One was to be fully autonomous, a self-running code that breathes life into a poem. The other required interactive elements, engaging the user in the creative process, be it through text inputs or other interactive means.
Introducing My Poems
- In “I am,” I aimed to distill the essence of life’s ephemeral nature. Beginning with a beautiful harmonious chords, symbolizing a reflective journey at life’s end, the piece contrasts serene, almost heavenly music with the real-time emergence of words, each letter resonating with a note from the pentatonic scale. As the letters are revealed one by one, a startling message is revealed. The stark contrast between the music and the words underscores the poem’s core message. Next, The AI’s role in crafting the phrase “I am __” reveals a narrative about the perceived gravity of moments. In the immediacy of life, situations may feel overwhelmingly significant. However, viewed from the perspective of life’s twilight, these intense experiences acquire a fondness and nostalgia. This etude reflects on the transformation of life’s experiences over time, from moments of intense significance to cherished, nostalgic memories.
- “Anxiety”: In “Anxiety,” the exponentially increasing rate at which words are presented and altered plays a crucial role in simulating the escalating nature of an anxiety attack. This acceleration mirrors how anxious thoughts often multiply and intensify rapidly, creating a sense of overwhelming urgency and panic. The exponential increase in speed is not just a technical feature but a deliberate artistic choice, designed to evoke the physical and emotional sensations of anxiety. As the words flash by more quickly, the listener might experience a sense of breathlessness and disorientation, mirroring the physical symptoms of an anxiety attack. This rapid succession of words and the accompanying escalation in the pace of the music create a visceral, immersive experience for the listener, providing a glimpse into the tumultuous nature of anxiety.
Demos of my Poems
Reflection
Creating these pieces was a venture into the unknown, blending music and words in an artistic symphony. Initially, creating the particular musical sounds I wanted via ChucK was difficult. But once I got more comfortable programming with ChucK, I was able to implement my artistic vision.
The success of these poems, in my view, lies in their ability to evoke emotions. They transcend intellectual comprehension, focusing instead on creating a visceral experience for the audience. The music played a pivotal role, not just as a backdrop but as a language that communicated feelings and set the tone.
Timing the words added a sense of immediacy and intimacy, as if the computer was conversing with the audience. This aspect, combined with the unpredictable nature of AI, introduced a layer of organic fluidity. Each run of the poem offered a new, unique experience, akin to a live performance.
Code for my Poems
i_am.ck
//---------------------------------
// name: i_am.ck
// desc: a poem about the ephemeral nature of life
//
// author: Guinness Chen
// date: Jan 2024
//---------------------------------
//---------------
// setup audio and model
//---------------
// declare our audio objects
NRev reverb => dac;
NRev drumverb => dac;
ModalBar bar => dac;
LPF filter => reverb;
// setup reverb
.5 => reverb.mix;
.2 => drumverb.mix;
// setup bar
1 => bar.preset;
0.2 => bar.stickHardness;
0.3 => bar.volume;
// setup LFO
SinOsc lfo => blackhole;
2::second => lfo.period;
// setup pad
BlowBotl pad[4];
for (0 => int i; i < 4; i++){
pad[i] => filter;
pad[i].noiseGain(0);
}
// setup samples samples
SndBuf kick => drumverb => dac;
kick.samples() => kick.pos;
kick.read("sounds/kick.wav");
kick.gain(1);
SndBuf snare => drumverb => dac;
snare.samples() => snare.pos;
snare.read("sounds/snare2.wav");
snare.gain(1);
SndBuf ahh => reverb => dac;
ahh.samples() => ahh.pos;
ahh.read("sounds/ahh.wav");
ahh.gain(0.5);
// load the word2vec model
Word2Vec model;
model.load("glove-wiki-gigaword-50.txt");
//---------------
// define constants
//---------------
// define some chords to use later
[36, 64, 67, 72] @=> int C[];
[36, 55, 62, 64] @=> int Cadd2[];
[38, 65, 67, 72] @=> int Dm11[];
[40, 64, 67, 72] @=> int CoverE[];
[40, 64, 67, 71] @=> int Cmaj7overE[];
[41, 67, 69, 72] @=> int Fadd2[];
[41, 65, 69, 72] @=> int F[];
[41, 60, 62, 68] @=> int Fm6[];
//---------------
// helper functions
//---------------
// play a note from the pentatonic scale
fun float getPentatonic(int scaleNum){
scaleNum % 10 => int note;
[0, 2, 4, 7, 9, 12, 14, 16, 19, 21] @=> int pentatonic[];
72 + pentatonic[note] => float pitch;
Std.mtof(pitch) => float freq;
return freq;
}
// print spaces so that the next sentence is centered
fun void printCenter(string sentence){
for (0 => int i; i < (80 - sentence.length()) / 2; i++){
chout <= " ";
chout.flush();
}
}
// print a sentence one character at a time and play a note for each character
fun void typeWriter(string sentence, int waitTime){
// center the sentence
printCenter(sentence);
// loop through the sentence one character at a time
for (0 => int i; i < sentence.length(); i++){
// wait a little longer if it's a ~
if (sentence.substring(i, 1) == "~"){
waitTime::ms => now;
continue;
}
// print the current character
chout <= sentence.substring(i, 1); chout.flush();
// play a note if it's not a space
if (sentence.substring(i, 1) != " " && sentence.substring(i, 1) != "\n"){
getPentatonic(Math.random2(0,24)) => bar.freq;
0.5 => bar.noteOn;
waitTime::ms => now;
0.1 => bar.noteOff;
}
// pause slightly longer if it's a space
else {
500::ms => now;
}
}
}
// play a chord
fun void playChord(int chord[]){
for (0 => int i; i < chord.size(); i++){
Std.mtof(chord[i]) => pad[i].freq;
0.5 => pad[i].noteOn;
}
}
// play the background chords
fun void playBackground(){
// loop the chord progression forever
while (true){
playChord(C);
4::second => now;
playChord(Dm11);
4::second => now;
playChord(CoverE);
2::second => now;
playChord(Cmaj7overE);
2::second => now;
playChord(Fadd2);
2::second => now;
playChord(F);
2::second => now;
}
}
// play the drum beat
fun void playDrums(){
while (true){
// play the kick
0 => kick.pos;
0.25::second => now;
0 => kick.pos;
0.75::second => now;
// play the snare
0 => snare.pos;
1.25::second => now;
// play the kick
// play the kick
0 => kick.pos;
0.25::second => now;
0 => kick.pos;
0.5::second => now;
// play the snare
0 => snare.pos;
1::second => now;
}
}
// print the words in the sentence at the same time as the drum beat
fun void drumSentence(string adjetives[]){
// do this loop 8 times
for (0 => int i; i < 8; i++){
// print first sentence
adjetives[Math.random2(0, 21)] + " " => string adj;
printCenter("I am " + adj + ".");
chout <= "I ";
chout.flush();
0.25::second => now;
chout <= "am ";
chout.flush();
0.75::second => now;
chout <= adj + ".\n"; chout.flush();
1.25::second => now;
// print second sentence
adjetives[Math.random2(0, 21)] + " " => adj;
printCenter("I am " + adj + ".");
chout <= "I " ;
chout.flush();
0.25::second => now;
chout <= "am ";
chout.flush();
0.5::second => now;
chout <= adj + ".\n"; chout.flush();
1::second => now;
// make a new stanza every 4 lines
if (i % 2 == 1) {
chout <= "\n"; chout.flush();
}
}
}
// get a random similar word to x that's not x
fun string getSimilar(string word){
string result[10];
model.getSimilar(word, 10, result);
return result[Math.random2(0, 9)];
}
//---------------
// main program
//---------------
// play the background chords in parallel
spork ~ playBackground() @=> Shred background;
Shred drums;
kick.samples() => kick.pos;
snare.samples() => snare.pos;
ahh.samples() => ahh.pos;
for (0 => int t; t < 3000; 1 +=> t){
// type the first sentence right away
if (t == 0){
spork ~ typeWriter("I just pooped my pants.", 500);
chout <= "\n";
chout.flush();
}
// wait 16 seconds to print the second sentence
else if (t == 800){
printCenter("AAAAAAAAAAHHHHHHHHHHHHHH!~~ AAAAHHHHHHHHHHH!");
spork ~ typeWriter("AAAAAAAAAAHHHHHHHHHHHHHH!~~ AAAAHHHHHHHHHHH!\n", 250);
ahh.pos(0);
chout <= "\n";
chout.flush();
}
// wait 32 seconds to print the next section
else if (t == 1600){
spork ~ playDrums() @=> drums;
string adjetives[32];
model.getSimilar("smelly", 22, adjetives);
spork ~ drumSentence(adjetives);
chout <= "\n";
chout.flush();
}
// every frame, modulate the filter frequency via the LFO
350 + (lfo.last() + 1) * 250 => filter.freq;
// advance time by one frame
20::ms => now;
}
//stop the background chords
background.exit();
// play the penultimate chord
playChord(Fm6);
4::second => now;
chout <= "\n";
spork ~ typeWriter("Things are never as bad as they seem...", 50);
chout.flush();
// play the final chord
playChord(Cadd2);
8::second => now;
// close all instruments
drums.exit();
for (0 => int i; i < pad.size(); i++){
0.5 => pad[i].noteOff;
}
4::second => now;
chout <= "\n"; chout.flush();
anxiety.ck
//---------------------------------
// name: anxiety.ck
// desc: an interactive poem about anxiety
//
// author: Guinness Chen
// date: Jan 2024
//---------------------------------
//---------------
// setup audio and model and input
//---------------
// setup the ConsoleInput
ConsoleInput in;
StringTokenizer tok;
string line[0];
// setup bar
ModalBar bar => NRev reverb => dac;
.1 => reverb.mix;
7 => bar.preset;
// setup pad
NRev padVerb => dac;
BlowBotl pad[4];
for (0 => int i; i < 4; i++){
pad[i] => padVerb;
}
padVerb.mix(0.5);
// load the model
Word2Vec model;
model.load("glove-wiki-gigaword-50.txt");
//---------------
// define constants
//---------------
// define a set of anxiety words
[ "torment",
"dread",
"panic",
"anguish",
"paranoia",
"terror",
"meltdown",
"claustrophobia",
"agony",
"restlessness",
"desperation",
"hysteria",
"phobia",
"trauma",
"obsession",
"nightmare",
"manic",
"distress",
"horrified",
"apprehension",
"unease",
"trepidation",
"heartache",
"suffocation",
"dismay",
"despair",
"frantic",
"overwhelm",
"hypervigilance",
"neurosis",
"foreboding",
"panic_attack",
"frenzy",
"jitters",
"shell_shock",
"psychosis",
"disquiet",
"twitchiness",
"haunting",
"alienation"] @=> string anxiety_words[];
// define some chords to use later
[36, 64, 67, 72] @=> int C[];
[36, 63, 67, 72] @=> int Cm[];
[41, 65, 69, 72] @=> int F[];
[39, 66, 70, 75] @=> int Ebm[];
// define the interpolation ratio
0.15 => float alpha;
//---------------
// helper functions
//---------------
// play a note from the pentatonic scale
fun float getPentatonic(int scaleNum){
scaleNum % 10 => int note;
[0, 2, 4, 7, 9, 12, 14, 16, 19, 21] @=> int pentatonic[];
72 + pentatonic[note] => float pitch;
Std.mtof(pitch) => float freq;
return freq;
}
// print a sentence one character at a time and play a note for each character
fun void typeWriter(string sentence, int waitTime){
// center the sentence
// loop through the sentence one character at a time
for (0 => int i; i < sentence.length(); i++){
// wait a little longer if it's a ~
if (sentence.substring(i, 1) == "~"){
waitTime::ms => now;
continue;
}
// print the current character
chout <= sentence.substring(i, 1); chout.flush();
// play a note if it's not a space
if (sentence.substring(i, 1) != " " && sentence.substring(i, 1) != "\n"){
getPentatonic(Math.random2(0,24)) => bar.freq;
0.5 => bar.noteOn;
waitTime::ms => now;
0.1 => bar.noteOff;
}
// pause slightly longer if it's a space
else {
500::ms => now;
}
}
}
// play a chord
fun void playChord(int chord[]){
for (0 => int i; i < chord.size(); i++){
Std.mtof(chord[i]) => pad[i].freq;
0.5 => pad[i].noteOn;
}
}
// play the background chords
fun void playHappy(){
// loop the chord progression forever
while (true){
playChord(C);
4::second => now;
playChord(F);
4::second => now;
}
}
fun void playScary(){
// loop the chord progression forever
while (true){
playChord(Cm);
4::second => now;
playChord(Ebm);
4::second => now;
}
}
// interpolate between two words
fun string interpolate(string input1, string input2, float t){
// get vectors
float vec1[50];
float vec2[50];
model.getVector(input1, vec1);
model.getVector(input2, vec2);
// interpolate
float vec3[50];
for (0 => int i; i < 50; i++){
vec1[i] + (vec2[i] - vec1[i]) * t => vec3[i];
}
// find the closest word
string closestWords[2];
model.getSimilar(vec3, 2, closestWords);
return closestWords[1];
}
//---------------
// main program
//---------------
// start the happy background music
spork ~ playHappy() @=> Shred backgroundHappy;
// prrint the words and prompt the user
typeWriter("What is something that makes you happy?\n", 50);
// prompt
typeWriter("Enter a word:", 50);
in.prompt( " " ) => now;
// read
while( in.more() )
{
// clear tokens array
line.clear();
// get the input line as one string; send to tokenizer
tok.set( in.getLine() );
// tokenize line into words separated by white space
while( tok.more() )
{
// put into array (lowercased)
line << tok.next().lower();
}
// if non-empty
if( line.size() )
{
// do something with the input
execute( line );
}
}
// do something with the text input
fun void execute( string input[] )
{
// play the background music in parellel
backgroundHappy.exit();
Shred backgroundScary;
// make a new line and set the first word to be the input
chout <= "\n";
input[0] => string nextWord;
250::ms => dur wordDur;
// print 200 new words
for (0 => int i; i < 200; i++){
// start playing scary music at i=50
if (i == 20){
spork ~ playScary() @=> backgroundScary;
}
// sample a random anxiety word
anxiety_words[Math.random2(0,anxiety_words.cap()-1)] => string anxiety_word;
// interpolate between the next word and the anxiety word
interpolate(nextWord, anxiety_word, alpha) => nextWord;
// print the next word
chout <= nextWord + " "; chout.flush();
wordDur => now;
// play a note
Math.random2(200, 2000) => bar.freq;
bar.noteOn(0.5);
// increase the speed and increase the interpolation rate.
wordDur * 0.98 => wordDur;
0.01 +=> alpha;
}
// wind down
backgroundScary.exit();
typeWriter(" \n\n - an anxious stream of consciousness.", 50);
// release the chord
for (0 => int i; i < 4; i++){
pad[i].noteOff(0.2);
}
8::second => now;
}