Collaborative Music App-Part 1: playing sounds

Me, making some music

Recently, I came across PLINK!, a collaborative music app by DinahMoe Labs, and decided to recreate a simple version of it. This short tutorial implements a single player version.

In Part 2, we will refactor it to be multiplayer, and update the look to be closer to the original.

Here is a list of basic features we will tackle in Part 1:

• The pitch of the notes you play is determined by the y-coordinates of your mouse position. With lower y-coordinates (higher on the screen) being used for higher notes.
• In order to increase the chance that the notes played sound good together, a pentatonic scale is used, comprising of the notes ‘C’, ‘E’ , ‘G’, ‘A’, ‘D’. This scale is used in a lot of jazz music.
• The user can play repeated notes with a beat by holding down the mouse.

To accomplish this, we will be using the Web Audio API.

If you are not familiar with the Web Audio API, here is a quick primer:
All sounds in run within an audio context. Within the audio context, you can create an oscillator, which is responsible for creating sound.

var audioCtx = new window.AudioContext 

// create Oscillator node
var oscillator = audioCtx.createOscillator();

The frequency can be set by either frequency.value, which changes the frequency immediately, or frequency.setValueAtTime, which can be used to change the frequency in the future.

oscillator.frequency.setValueAtTime(440, audioCtx.currentTime); // value in hertz

The oscillator then needs to be connected to the audio context’s “destination” node before the sound can be played. Calling the start method on the oscillator plays the sound.

oscillator.connect(audioCtx.destination);
oscillator.start();

Here it is all together:

var audioCtx = new window.AudioContext

// create Oscillator node
var oscillator = audioCtx.createOscillator();
oscillator.type = 'square'; // if not set, defaults to sine.
oscillator.frequency.setValueAtTime(440, audioCtx.currentTime); // value in hertz
oscillator.connect(audioCtx.destination);
oscillator.start();
// To stop the sound
oscillator.stop()

With that out of the way, let’s get cracking on our Plink! spinoff.

To jump-start us, I suggest starting with the following example code: https://github.com/mdn/violent-theremin.

The violent-theremin example is similar to what we’re making, in that the frequency is determined by the mouse coordinates.

// create web audio api context
var audioCtx = new (window.AudioContext || window.webkitAudioContext)();
// create Oscillator and gain node
var oscillator = audioCtx.createOscillator();
var gainNode = audioCtx.createGain();
// connect oscillator to gain node to speakers
oscillator.connect(gainNode);
gainNode.connect(audioCtx.destination);
// create initial theremin frequency and volumn values
var WIDTH = window.innerWidth;
var HEIGHT = window.innerHeight;
var maxFreq = 6000;
var maxVol = 1;
var initialFreq = 3000;
var initialVol = 0.5;
oscillator.type = 1; // square wave
oscillator.frequency.value = initialFreq; // value in hertz
oscillator.start();
gainNode.gain.value = initialVol;
// Mouse pointer coordinates
var CurX;
var CurY;
// Get new mouse pointer coordinates when mouse is moved
// then set new gain and pitch values
document.onmousemove = updatePage;
function updatePage(e) {
    KeyFlag = false;
CurX = e.pageX
    CurY = e.pageY
    oscillator.frequency.value = (CurX/WIDTH) * maxFreq;
    gainNode.gain.value = (CurY/HEIGHT) * maxVol;
canvasDraw();
}
...

All we have to do is use Y axis instead of the X axis, round the pitch to one of the notes on our scale, and we have our first feature done! But where are we going to get the values for the pitch? From the simple synth tutorial’s source code (click “open in codepen”), which has a handy list of frequencies. Using those numbers, and eliminating all notes from the table that are not A, C, D, E, or G, we can create the following array.

let freqs=[]
freqs[0]= 110.000000000000000;
   freqs[1]= 130.812782650299317;
   freqs[2]= 146.832383958703780;
   freqs[3]= 164.813778456434964;
   freqs[4]= 195.997717990874647;
   freqs[5] = 220.000000000000000;
   freqs[6] = 261.625565300598634;
   freqs[7] = 293.664767917407560;
   freqs[8] = 329.627556912869929;
   freqs[9]= 391.995435981749294;
   freqs[10] = 440.000000000000000;

We can then reference the array when we assign frequencies, in our theremin example. Let’s say we include 11 different notes. Our code could now look something like this:

/* We want the highest frequency to be the top of the screen,
The highest frequency being position 10 in the array.
 So, we find how far down the mouse is, as a percentage of HEIGHT:
CurY/HEIGHT
 Invert it, to find how far up the mouse is: 1-CurY/HEIGHT
Multiply it by 10, and round, to find the fraction out of 10:
Math.round(10*(1-CurY/HEIGHT))*/
function updatePage(e) {
CurX = e.pageX
CurY = e.pageY
oscillator.frequency.value =freqs[Math.round(10*(1-CurY/HEIGHT))];
gainNode.gain.value = Math.floor(CurX/HEIGHT) * maxVol;
canvasDraw();
}

Congratulations! You can now play a minor pentatonic scale!

Playing a note on mouse down

We want the notes to repeat when we hold down the mouse button. The thing is, oscillators can’t be reused once they are stopped. So we are going to create a fresh oscillator for each note. We’ll copy the part where we are creating the oscillator, starting the oscillator, and setting its frequency, and put it in a separate function. Let’s have each note play for 100ms, and play another note after 200ms. We’ll do that by using the setTargetAtTime method of the gainNode. The whole thing will happen on mouse down. Also, we’ll disable changing the frequency and volume when we move the mouse, and we won’t create the oscillator on init anymore.

function playSound(){
  var oscillator = audioCtx.createOscillator();
  var gainNode = audioCtx.createGain();
  oscillator.connect(gainNode);
oscillator.frequency.value =freqs[Math.round(10*(1-CurY/HEIGHT))];  gainNode.gain.setTargetAtTime(0, audioCtx.currentTime, 0.1);
oscillator.stop(audioCtx.currentTime+0.2)
}
canvas.addEventListener('mousedown', playSound)
  
function updatePage(e) {
....
//oscillator.frequency.value = (CurX/WIDTH) * maxFreq;
// gainNode.gain.value = (CurY/HEIGHT) * maxVol;  }

Repeating the note

Next, we’ll keep the note repeating until mouseUp:

let mousePressed=false
let isPlaying=false
function handleMousedown(){
 mousePressed=true;
if (!isPlaying){playSound()}
}
function handleMouseup() {
mousePressed=false;
}
function playSound(){ 
 if(mousePressed){
  var oscillator = audioCtx.createOscillator();
  var gainNode = audioCtx.createGain();
  oscillator.connect(gainNode);
oscillator.frequency.value =freqs[Math.round(10*(1-CurY/HEIGHT))];      gainNode.gain.setTargetAtTime(0, audioCtx.currentTime, 0.1);
  oscillator.stop(audioCtx.currentTime+0.2)
  setTimeout(playSound,200)
  }
else {isPlaying:false}
}
canvas.addEventListener('mousedown', handleMousedown)

Here, we’re looping playSound, so each time we play a sound, we schedule another sound to play in 200ms… but it will only if the mouse is still down 200ms later!

So, we now have the ability to play notes, and repeat them. That’s it for Part 1! Here’s a demo:

Part 2 will cover creating a multiplayer version. Stay Tuned.