EMCT: COMPUTING MAJOR PROJECT
Categorising the underlying principles of audio effects into three areas from a technical/coding perspective.
I’m assuming some prerequisite knowledge of plugins here. I’d imagine you at least know what a plugin is and how to use one, but if I asked you to describe a plugin or audio effect more scientifically, that’s where difficulties start to occur. Would you know what to say if I asked for a description of an audio effects component parts? When describing an audio effect as its parts (including VSTs, Plugins and any digital hardware), it can be difficult to pinpoint what we’re trying to explain. This is where my initial research comes in. I can’t do a deep dive into audio effects if I can’t describe what they are. Hence:
Digital Audio Effects — The Three Component Pillars:
After some initial internet scouring, I came across the excellent work of Dr Andrew McPherson and Prof. Joshua Reiss. Their book on Audio Effect Theory and Implementation was a useful resource for my research. They described audio effects within the digital domain as ‘systems that take a discrete, uniformly sampled audio signal, process it, and produce a discrete uniformly sampled output audio signal’¹. My full paper, to be released alongside the project’s final output, goes into this in more depth and directly points to how this definition supports my classification of audio components. However, I’ll talk about it quickly below.
I break audio effects into three parts, the three pillars of audio effects. These pillars can encompass many different things, but I’ve found you can categorise pretty much any element of an audio effect into one of these 3. So here they are:
- The System
— First, the system itself includes anything related to the actual behaviour of the audio effect. This could be handling the audio processing, defining how user interaction is processed within the plugin, and defining the audio effect as a piece of software within the operating system or host environment the program is running in.
— Audio plugin programming is normally done in a compiled, low-overhead systems language with some or complete direct memory management. Traditionally this has been C or C++.
— Digital system design requires reasonable programming skills and comfortability in your chosen programming language. You need to be able to define and create a real-time software system within a digital context; one must also follow best practices and maintain efficiency to ensure real-time audio processing. The system you build defines the vast majority of what the audio effect is and how it behaves. This is the first part of a plugin you must understand, - The DSP
— Second is DSP, which stands for Digital Signal Processing and is an offshoot of signal processing mathematics. DSP refers to any mathematical manipulation of discrete audio samples to achieve a desired audio output within audio effects. This could be a distortion algorithm or something more complicated like a dynamic-level compressor.
— DSP is complicated and requires reasonable mathematical skills to even understand a DSP algorithm. You could likely apply DSP algorithms with a moderate mathematical understanding, but writing your own would require real expertise. Understanding how to use DSP algorithms in code and apply them to an audio buffer or stream of audio samples is the second thing you must come to know if you wish to embark on this journey of making audio plugins. - The Audio Data and Constants:
— Finally, there’s the Audio Data and Constants, which refers to the audio itself in its digitally sampled form and how you manage this data across the audio effect. As an audio programmer, you should know the answer to these questions. What is numerical audio data? What does an audio sample value look like? What is an interweaved audio buffer? What range does a sample value fall between? What sample rate was your audio sampled at? How many bits of data does each audio sample have? Do you understand what an audio buffer is? Audio data management is critical, and mismanaged audio data will lead to unexpected behaviour or may even make your audio effect completely unusable.
— I’d say this last pillar mostly hinges on a good understanding of audio. If you’ve spent some time working in audio, you likely understand why sample rates are important or what sort of value range you’d expect the audio samples to be in. If you don’t, I’d start with understanding what a waveform is (what is a sine wave?) and the difference between digital and analogue audio. They’re both a good place to start. Though, ask questions below if you’d want more direction on this one.
— It’s also worth noting that inside certain environments (DAWs, for example), constants like the sample rate, bit depth or buffer size tend to be controlled by the DAW. So you likely won’t define these yourself but instead manage your data in a way that allows the DAW to determine these factors for you.
By understanding these three categories, we can see how an audio effect works. The audio data is our audio; we make sure to handle it properly and ensure we maintain the proper rules for digital audio processing. The system is our coded environment; it defines the audio effect and decides how our audio data and user interaction are sent around inside the audio effect to achieve our desired goals. The DSP is our processing; we apply different algorithms to the audio to manipulate it and achieve the desired effect.
Based on this, the simple audio effect we could make would be a basic system that could take in an audio input, manage the audio data appropriately based on characteristics like the sample rate, apply something basic to it like a gain multiplication to said audio and then spit the audio back out to wherever it came from.
So what does this mean?
Well, now we understand an audio effect at the absolute bare-bones level. Modern and professionally developed plugins are huge and have many steps to allow for their crazy advanced features. However, they would still employ these basic ideas and a simple plugin utilising these ideas, even in the most basic manner, would still resemble an audio effect.
Next Steps:
- Start building systems
— The first step will be to build something that can handle audio input and return audio output. Some extensive research will need to be done on this. I know that connecting to drivers to run physical things like speakers is not a process for the faint of heart.
— I’ll likely experiment with various ways to do this and consider different methods for the foundational system of my project output. - Start learning DSP
— Spending time manipulating low-level audio signals in any regard will help with this. I have some units for university which will help me learn more about direct signal manipulation though admittedly, I’m already comfortable with understanding.
Written: Jan 20, 2023
