Music reactive visualizations
Hello creative world, In this article I’ll describe a technique I came up with for making awesome audio reactive visualizations. Several well known concepts will be presented and a methodology to tie all these in a practical workflow.
First of all we need to define a context. We want our graphics to react to music in a meaningful way. Music is a common arrangement of audio frequencies in a way that can incite sentimental reactions to its listeners. The psychoacoustic effects can vary slightly by listener but most would agree that the sonic patterns form something meaningful and have a distinct feel.
These patterns are well described by music theory and that gives us a great starting point to base our processing on. The established format for grouping frequencies is called MIDI, a non linear, human friendly, segregation for audio.
Frequencies are grouped in octaves, which have the characteristic that every subsequent octave is twice as wide as the previous one. These are then also non linearly split into 12 notes each. The end result is that the first note(C) from the first octave would be approx 1/1000 the size of the first note(C) of the 11th octave. So the first notes would cover a handful of hz while the later ones would span hundreds and even thousands of hz.
This makes sense from a physical standpoint, as higher frequencies are more difficult to pick up with precision so our sensors(ears) resort to non linearity.
With this knowledge at hand an ideal first step would be to rearrange the linear output coming from an FFT to midi space and work with that. This gives us the advantage of measuring tonal properties of the sound in addition to the, usual, energy ones.
We may not get the exact same scoresheet used by the tracks composer but specific areas of the spectrum will stand out and the sound peaks will form distinctive patterns over time. These patterns are all we need as we’re just aiming for meaningful & good looking results, not an academic exercise.
Graphics, by their nature, reside on higher dimensions than audio. Sound analysis will generally provide us with 1D signals, varying over time, so we need to come up with a strategy to tie them with the visuals which eventually get rasterized as a 2D grid but, as a process, go through lots of 2D/3D vectors.
Vectors, that’s it! We could express music as vectors and work with these. There are various ways to do that, some being more meaningful than others.
A good combination would be mapping the position of the most energetic midi note measured in an octave as the vector’s x axis and it’s actual energy as the y axis. Measuring the length of that vector produces distinct patterns and modulators that can be applied to properties of our graphic generation process. By doing this for several octaves and tying the modulators together we’ll end up with awesome music driven visuals that seem to dance stylishly.
There are various other combinations and tricks we can perform on top of that, and I will describe them in future articles, but for now I’ll just provide links to some examples so you can see and evaluate the results by yourself:
Sea, Sun & Zorba, Cubes & Roses, Electro Fractal Land,
Bello Electro, Boom Generators, ClubberToy
These examples are made with Clubber.js and ClubberTool, a js library and associated tooling I put together to ease up the abovementioned workflow.
The music vector manipulations are performed with glsl which seems like the most appropriate language for the task thanks to its builtin vector functions. It’s easy to reuse the modulators in any context though, just transpile the glsl to js with Clubberize and modulate whatever property you want.
That’s it for now, till next time have fun and keep those pixels pumping :)
