A Gift of Sound and Vision — Affecting images using the Web Audio API
What happens when you pass an image through a guitar pedal?
Don’t you wonder sometimes?
In 1977 David Bowie released the album called “Low”, one of the greatest pieces of art that humanity ever listened. Today I’ll prove that “Sound and Vision” was a prediction about the APIs that we have on browsers nowadays!
On a previous post I wrote how to apply audio filters on images. There is some important concepts there on how to load a picture and draw it on canvas. Check it for more details, but I’ll replicate the code below:
The main idea is: if we can grab colors from the getImageData method as arrays, we can use them as input to Web Audio API nodes! So, yes, we can hear a color. Go to this link and press right arrow on keyboard or click “next” on bottom right corner. Take care with the volume because the sound is far from pleasent! You will listen to the red color on Low’s cover.
I would like to thank the amazing Sam Bellen and his talk where he created a complete pedal board! I knew this other pedalboard.js project before but when I watched the JSConf Budapest I thought: “What if I apply other effects than filters to images?”. So I took a look on his repo for the implementations on the most complex cases!
Here are some nodes that we’ll use to create the effects.
We’ll start by adding a delay between the media input and the output, not really useful as a guitar effect because it’ll only make the sound be delayed (as the name clearly says!) without to have the imediate feedback on what you’re playing but it’ll help to explain the basic concept.
Plotting the input (a simple 300 Hz sin wave, in orange) with the output (in white) we have the graph below. Well, that’s what we expect from a delay: a time displacement.
Now if we split the colors from an image and then use it individually through the delay configuration above we can note a displacement on the image:
This happens because when we “rebuild” it from the splitted colors, the rgba array will be displaced. Let’s add a big delay value to visualize that the image is being “pushed” away and that’s why we have the gray zone on top. If we look on the graph above we can see that on the response beggining (white signal) there are zeroed values that correspond to the gray area. You are probably asking yourself why is gray and not black (if the values are zero). That’s because the process to transform amplitudes between -1 and 1 to 0 and 255.
And here is a gif with the final image being generated color by color:
To achieve this I created a Delay class and some util classes (creative name, huh?). The code below is how I put it together the image loading and the effect applying
And below is the method that get the colors and plot them on canvas after applying the effects:
A better delay
Now let’s add a direct connection between the input and output and a delay in another one.
With a direct connection we’ll have the original image preserved and the delay will add a exact copy.
It’s interesting to see how the colors are more intense because the overlap of the signals. Depending of the delay value the second image will be placed in a different spot:
Retro Feeded Delay
Now let’s add a gain node on the delay path and connect it back to it. This will cause a kind of looping.
On the graph below there is an input (a 300 Hz sin wave with two oscillations, in orange) and the output (in white) shows how this looping structure keeps the signal alive even after the input is zeroed.
Passing the image through this configuration we can see the multiple copies of it being placed!
Depending of the delay and gain values the copies are placed different:
Let’s use a different pedal! The flanger has a quite complex structure:
Note a gain connected to the delay, then the delay is connected to a node, which is connected to the first again, this builds a looping structure. There is also an oscillator connected to the same delay.
Looking at the response graph we can see two different outputs in white and yellow (using different values of the oscillator frequency and delay).
The result shows multiple images being placed because the looping and how the oscillator make all the copies curved. Also, the original is preserved because there is a connection that is not being passed through the delay (and neither being affected by the oscillator).
And changing some parameters:
Reverb — Convolver
Let’s add a convolver node which is used mostly to create the reverb effect:
One way to achieve the desired output is to load an audio file of the impulse response that will dictate the behaviour. In this implementation is a clap sound being recorded on a big hall (stolen from Sam Bellen examples, thank you again, sir).
The output (in white) shows how it affects phase and keeps the signal reverberating (accurate name, right?).
The result shows how amplitude and phase changes can affect things in an almost unrecognizable way. But let me tell you, I would totally buy a shirt with this theme!
Distortion — Waveshaper
To create a distortion we’ll add a waveshaper node.
On the graph below the curves goes from most distorted (orange) to less (white). As quick the transition happens on zero point the distortion will be more agressive.
Plotting the input with different outputs (please, to the graph below ignore the color convention of the curves above 😅) we can see how it affects a sin wave depending on the type of the sigmoid function. As more distorted, more close to a square wave.
Looking at the affected image we can see that colors are being rounded loosing the nuance from tones that are close to each other. See how the grass became a big piece of green almost impossible to note differences!
All the code for images being affected is here.
Don’t do this in production™! Seriously! But hey, code can be fun sometimes right? If we can use to do ambitious applications we can have some silly uses too!
I want to thank every person working to improve browser APIs and make it so flexible that I can mess around like I did ❤️!
If you have any question I would love to help! Reach me on twitter!