Earthling cover affected by a retro feeded delay

A Gift of Sound and Vision — Affecting images using the Web Audio API

What happens when you pass an image through a guitar pedal?

Don’t you wonder sometimes?

In 1977 David Bowie released the album called “Low”, one of the greatest pieces of art that humanity ever listened. Today I’ll prove that “Sound and Vision” was a prediction about the APIs that we have on browsers nowadays!

On a previous post I wrote how to apply audio filters on images. There is some important concepts there on how to load a picture and draw it on canvas. Check it for more details, but I’ll replicate the code below:

Snippet demonstrating how to load image to a canvas and extract information from it

The main idea is: if we can grab colors from the getImageData method as arrays, we can use them as input to Web Audio API nodes! So, yes, we can hear a color. Go to this link and press right arrow on keyboard or click “next” on bottom right corner. Take care with the volume because the sound is far from pleasent! You will listen to the red color on Low’s cover.

I would like to thank the amazing Sam Bellen and his talk where he created a complete pedal board! I knew this other pedalboard.js project before but when I watched the JSConf Budapest I thought: “What if I apply other effects than filters to images?”. So I took a look on his repo for the implementations on the most complex cases!

Sam’s talk on JSConf Budapest!

Here are some nodes that we’ll use to create the effects.

List of nodes that will be used on the demos (Delay, Media, Oscillator, etc…)

Delay

We’ll start by adding a delay between the media input and the output, not really useful as a guitar effect because it’ll only make the sound be delayed (as the name clearly says!) without to have the imediate feedback on what you’re playing but it’ll help to explain the basic concept.

Adding a delay node between the input and output

Plotting the input (a simple 300 Hz sin wave, in orange) with the output (in white) we have the graph below. Well, that’s what we expect from a delay: a time displacement.

Graph 1 — Delay response (white) to a 300 Hz sin (orange)

Now if we split the colors from an image and then use it individually through the delay configuration above we can note a displacement on the image:

Original image (left) and result (right) after pass the colors through the delay node

This happens because when we “rebuild” it from the splitted colors, the rgba array will be displaced. Let’s add a big delay value to visualize that the image is being “pushed” away and that’s why we have the gray zone on top. If we look on the graph above we can see that on the response beggining (white signal) there are zeroed values that correspond to the gray area. You are probably asking yourself why is gray and not black (if the values are zero). That’s because the process to transform amplitudes between -1 and 1 to 0 and 255.

Result of a big delay value being added to the process. The gray part on top shows the zero values on the signal response.

And here is a gif with the final image being generated color by color:

Original image (left) and result (right) being created color by color

To achieve this I created a Delay class and some util classes (creative name, huh?). The code below is how I put it together the image loading and the effect applying

And below is the method that get the colors and plot them on canvas after applying the effects:

A better delay

Now let’s add a direct connection between the input and output and a delay in another one.

Diagram of the delay effect with a direct connection between input and output

With a direct connection we’ll have the original image preserved and the delay will add a exact copy.

Result of a delay with the configuration above

It’s interesting to see how the colors are more intense because the overlap of the signals. Depending of the delay value the second image will be placed in a different spot:

Result using the same configuration with a different delay value

Retro Feeded Delay

Now let’s add a gain node on the delay path and connect it back to it. This will cause a kind of looping.

Diagram of the retro feeded delay

On the graph below there is an input (a 300 Hz sin wave with two oscillations, in orange) and the output (in white) shows how this looping structure keeps the signal alive even after the input is zeroed.

Graph 2— Retro feeded delay response (white) to a 300 Hz sin (orange)

Passing the image through this configuration we can see the multiple copies of it being placed!

Result of a retro feeded delay

Depending of the delay and gain values the copies are placed different:

Same configuration with different delay and gain values

Flanger

Let’s use a different pedal! The flanger has a quite complex structure:

Diagram of the flanger effect

Note a gain connected to the delay, then the delay is connected to a node, which is connected to the first again, this builds a looping structure. There is also an oscillator connected to the same delay.

Looking at the response graph we can see two different outputs in white and yellow (using different values of the oscillator frequency and delay).

Graph 3 — Flanger responses (white and yellow) to a 300 Hz sin (orange)

The result shows multiple images being placed because the looping and how the oscillator make all the copies curved. Also, the original is preserved because there is a connection that is not being passed through the delay (and neither being affected by the oscillator).

Result of the flanger effect

And changing some parameters:

Result of the flanger with different values of delay and oscillator

Reverb — Convolver

Let’s add a convolver node which is used mostly to create the reverb effect:

Convolver node diagram

One way to achieve the desired output is to load an audio file of the impulse response that will dictate the behaviour. In this implementation is a clap sound being recorded on a big hall (stolen from Sam Bellen examples, thank you again, sir).

The output (in white) shows how it affects phase and keeps the signal reverberating (accurate name, right?).

Graph 4— Reverb response (white) to a 300 Hz sin (orange)

The result shows how amplitude and phase changes can affect things in an almost unrecognizable way. But let me tell you, I would totally buy a shirt with this theme!

Result of the reverb effect

Distortion — Waveshaper

To create a distortion we’ll add a waveshaper node.

Waveshaper node diagram

In order to adjust how distorted the signal should be it needs a distortion curve. I saw many examples using sigmoid functions.

On the graph below the curves goes from most distorted (orange) to less (white). As quick the transition happens on zero point the distortion will be more agressive.

Multiple sigmoid functions plotted together

Plotting the input with different outputs (please, to the graph below ignore the color convention of the curves above 😅) we can see how it affects a sin wave depending on the type of the sigmoid function. As more distorted, more close to a square wave.

Graph 5— Waveshaper responses (white and yellow) to a 300 Hz sin (orange)

Looking at the affected image we can see that colors are being rounded loosing the nuance from tones that are close to each other. See how the grass became a big piece of green almost impossible to note differences!

Result of the distortion effect

All the code for images being affected is here.

Conclusion(?)

Don’t do this in production™! Seriously! But hey, code can be fun sometimes right? If we can use to do ambitious applications we can have some silly uses too!

I want to thank every person working to improve browser APIs and make it so flexible that I can mess around like I did ❤️!

If you have any question I would love to help! Reach me on twitter!