How To Make A Real-Time Music Application Using WebSockets

I recently decided to become more familiar with WebSockets, in an effort to understand how to build interactive web applications using real-time data transfer between multiple clients and a server. The WebSockets API allows for this real time communication by facilitating the transfer of messages and event-driven responses without the need for HTTP long-polling.

I find this real-time interaction with multiple users to be one of the most exciting aspects of web applications, and as a musician exploring WebSockets for the first time, the most obvious use case for this type of interaction was the ability to make music with friends, or even complete strangers, without being in the same room.

So, on that note, I would like to walk through how we might set up a WebSocket connection for transferring sound so we can jam with our friends in real-time! In order to do this, we are going to use Node.js, some helpful JavaScript frameworks and libraries, and WebPack for bundling all of our JavaScript files.

First and foremost, we will need sound. WebSockets supports the transfer of mp3 data, but I am more interested in audio synthesis, so lets build a very basic synthesizer to be used as our sound source. In order to do this, we are going to use Tone.js, “a Web Audio library for making interactive music in the browser”. You might say I am a bit of a synth snob, so my initial expectations for the quality of sound produced by a JavaScript library were fairly low, but I found Tone to be really great in this regard, especially when utilizing the sound shaping tools such as LFOs, filters, and effects. For the purposes this project, I am going to keep the sound sculpting to a minimum, using Tone’s basic Synth without any additional processing.To create the synthesizer, we will need to install Tone.js using node package manager, and then import it into our javascript file. Once imported we can connect the synth to our master output:

const Tone = require('Tone')
const gain = new Tone.Gain(0.5).toMaster()
let synth = new Tone.Synth().connect(gain)
export default synth

Now that we have a sound source, we can do just about anything with it, but for now let’s keep it simple, and build a basic 12-key piano in the C-Major chromatic scale. One option for this is to build the keyboard from scratch using HTML and CSS, then add event handlers to each of the buttons which make up the individual ‘piano keys’. For general purposes this is fine, but to make things a bit easier, as well as more visually appealing, we will utilize NexusUI, a toolkit which provides HTML5 interfaces for building musical instruments. NexusUI also provides some helper functions for adding event handlers to our piano interface. Once again, we can install NexusUI using npm, and import it into our index.js file. In the html file inside our public folder, we can now create a <div> element which will be our synthesizer, with an id attribute to be referenced by NexusUI’s piano constructor back in our javascript file:

import Nexus from 'nexusui'
import Tone from 'tone'
import synth from './synth'
const piano = new Nexus.Piano('#piano-keyboard', {
    'size': [500,125],
    'mode': 'toggle',
    'lowNote': 72,
    'highNote': 84
})

We can then use the provided on change method to map our synthesizer to the piano’s keys, so that we can trigger the corresponding notes using the synth’s built-in “triggerAttackRelease” method.

const triggerKeyPress = note => {
  synth.triggerAttackRelease(note, '16n')
}
piano.on('change', event => {
  let note = event.note, on = event.state
  if(on && note === 84) triggerKeyPress('C5')  
  else if(on && note === 83) triggerKeyPress('B4')
  else if(on && note === 82) triggerKeyPress('Bb4')
  else if(on && note === 81) triggerKeyPress('A4')
  else if(on && note === 80) triggerKeyPress('Ab4')
  else if(on && note === 79) triggerKeyPress('G4')
  else if(on && note === 78) triggerKeyPress('Gb4')
  else if(on && note === 77) triggerKeyPress('F4')
  else if(on && note === 76) triggerKeyPress('E4')
  else if(on && note === 75) triggerKeyPress('Eb4')
  else if(on && note === 74) triggerKeyPress('D4')
  else if(on && note === 73) triggerKeyPress('Db4')
  else if(on && note === 72) triggerKeyPress('C4')
})

At this point we should be able to play the chromatic scale on our synthesizer — very cool indeed! So let’s see how we can get this funk machine to churn out some WebSocket sounds, and get a jam session going with our friends. For this we are going to use Socket.io, a JavaScript library that simplifies some of the aspects of WebSockets, while providing native support across browsers. There is a debate to be had regarding how much is actually simplified compared to just using WebSockets, but I am a bit more comfortable with Socket.io, so that’s what we will be using.

In order to get our sockets going, we first need to get our server running. I will be using an express server — however WebSockets and Socket.io provide solid documentation on how to set up connections with a standard HTTP server. First we will have to npm install socket.io, and then require it in our server.js file. We also need to bring socket.io into our html file inside a script element. We can then initialize our socket instance by passing it the express server, and setting up the connection event listener for all incoming sockets. We should also set up a a disconnect event listener to let us know when a client has disconnected:

const path = require('path')
const express = require('express')
const app = express()
const socketio = require('socket.io')
const server = app.listen(4000, () => {
  console.log(`Listening on port ${server.address().port}`)
})
const io = socketio(server);
io.on('connection', socket => {
  console.log('A new client has connected!');
  console.log(socket.id);
  socket.on('disconnect', () => {
    console.log('user disconnected')
  })
})
app.use(express.static(path.join(__dirname, 'public')))
app.get('/', function (req, res) {
  res.sendFile(path.join(__dirname, 'index.html'));
});

We now have global access our io() function, which will allow us to set up the socket connection in our index.js file. We can then use that connection to send and receive messages from our server to the clients. To show this, we can log our connection to the console:

const socket = io():
socket.on('connect', function() {
  console.log('I have made a two-way connection to the server!')
})

With our socket connection set up, it is time to see if we can get some sounds traveling through the connection in real-time. To do this, we will use our piano’s on-change handler method given to us by NexusUI. Inside the handler, we will need a series of if statements to check for which MIDI note ‘value’ matches the event target’s note with a state currently set to ‘on’. When it finds a match, we can use the handler to fire off the correlated note on our synthesizer, once again using “triggerAttackRelease”. Here is where our socket connection comes into play — we can use our socket to then “emit” the note event back to the “press-key” listener we have set up on our server side:

socket.on('press-key', payload => {
  socket.broadcast.emit('press-key', payload)
})

When our server receives the note payload, it then “broadcasts” that payload to all client-side listeners set to receive the payload through our ‘press-key’ event. We can then use this payload to update which key is being pressed for all of our clients in real-time. First add the socket emitter to the body of our “triggerKeyPress” function:

socket.emit('press-key', note)

Next, pass the MIDI note value to our the piano’s built-in “toggleKey” method, which allows us to toggle the correct key’s ‘pressed’ state on or off, effectively playing the corresponding note on our synthesizer for each client:

const notes = {
  'C5': 84,
  'B4': 83,
  'Bb4': 82,
  'A4': 81,
  'Ab4': 80,
  'G4': 79,
  'Gb4': 78,
  'F4': 77,
  'E4': 76,
  'Eb4': 75,
  'D4': 74,
  'Db4': 73,
  'C4': 72
}
socket.on('press-key', key => {
  let note = notes[key]
  let keyToPress = piano.keys.find(key => key.note === note)
  piano.toggleKey(keyToPress.note)
})

We should now be able to play the piano, sending the correct key-press to all of our connected clients. Nice! In order to test this, we would need to use to separate devices with individual audio outputs, as we wouldn’t be able to hear more than one sound from a single output. However, if you only have one device you can still open http://localhost:4000 in a separate window — viewing the two windows side-by-side — and you should notice that the piano interface changes for each window when a key is pressed. If this is the case, then you can feel pretty confident the sound is being sent as well.

We have only scratched the service of what’s possible when using WebSockets for real-time music making, but with this foundation, one should have the tools to build much of what is possible with a digital audio workstation (DAW) all in their web browser, in addition to countless options for creativity that isn’t possible in DAWs alone. When you throw in WebSockets and the ability for real-time jamming with people all over the world, the possibilities for what can be done grow exponentially, especially when considering the other cool things WebSockets can do as a companion feature to the jamming!

Since discovering just a few of these possibilities, there are almost too many ideas for what I might want to do with this combination, and I look forward to exploring them in the future. Hopefully building this ‘synth socket’ has offered some insight into what you might want to explore as well.

So go explore, and happy jammin!