Is Audio Two-Dimensional?

When we represent uncompressed audio digitally, we often represent as a one-dimensional array of values, corresponding to the amplitude of the waveform over time.

Image for post
Image for post
A zoomed-in graph of an audio waveform

However, psycho-acoustically, audio does not feel like it is described by a single real number for each moment of time. Rather, when he hear audio, we notice all sorts of qualities of the audio like pitch and timbre. These qualities are best described as “spectral properties”. They are properties most readily identified from the Short Time Fourier Transform (STFT) of the signal. Here is an example of plotting the STFT of someone speaking the word “one”.

Image for post
Image for post

Although not totally intuitive to read, such two-dimensional plots of audio tend to give a better (more interpretable) representation of what an audio clip sounds like than the waveform.

The importance of this observation comes in when we are trying to do further processing on audio, for example running machine learning algorithms on audio. Performing a STFT to audio before feeding it into an algorithm can make it easier for that algorithm to pick up on the features that our ear hears. This approach can be used, for example, in using a CNN (Convolutional Neural Net) to detect features in audio. First we perform the STFT, then we can treat the plot (known as a spectrogram) as an image, and apply 2D convolutional layers to it to isolate features. This will often yield better results than trying to use 1D convolution on the raw audio waveform.

Hope this gives you some ideas when working with audio!

Written by

Logic PhD transitioning into Data Science

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store