How to Use Unity’s Native Audio Plugin SDK on macOS
+ Integrating with STK
This tutorial will show you how you can get the Native Audio SDK to compile your own plugin that can be loaded into the engine. It’ll allow you to use custom filters and synthesize sounds. Additionally, we’ll integrate with the STK library to simplify common signal processing tasks and get Unity to play a simple sine wave. Although this was written for OSX, the same process works for Windows and Linux, using that platform’s C++ compilers and tools.
This tutorial was written using Xcode 8.2.1 and is meant to supplement the guide provided by Unity.
Note: Native plugins are written in C++ and not C#.
We’ll start by creating a very simple plugin that just generates noise.
1. Setup the Unity and Xcode Projects
The first thing we need to do is download the SDK files from the Unity repository here. There’s a lot of example code in the download, but the main files of interest are AudioPluginInterface.h, AudioPluginUtil.cpp, AudioPluginUtil.h, and PluginList.h.
Once we have the SDK, let’s start a new Unity project. I put mine on the Desktop and called it UnityAudioPluginNoise. With the project created, make a new folder within the project called NativeCode. This will hold our Xcode project files.
Now we need to create an Xcode Bundle project and place it in the sub-folder we just made. In Xcode, hit File -> New -> Project and select Bundle:
Then name the project something like AudioPluginNoise:
When the Save dialog opens up, save the project in the NativeCode folder we made earlier.
IMPORTANT: The .bundle that Xcode builds must use a filename that has the prefix AudioPlugin in order for Unity to load the plugin. Therefore, it’s a good idea to also name the Xcode project with the prefix (but it’s not necessary). You can change the name of the bundle from the target settings (click the project name in the left sidebar).
Once the project’s created, go ahead and copy AudioPluginInterface.h, AudioPluginUtil.cpp, AudioPluginUtil.h, and PluginList.h from the SDK into the project. Make sure to check “Copy items if needed” and the bundle-target in “Add to targets”.
For non-OSX users: this effectively places the AudioPlugin*.h and PluginList.h files within the compiler’s header-search path and adds AudioPluginUtil.cpp as a compilation unit. With gcc/g++, you can use the -I compiler flag to add the directory holding the SDK header files as an additional include. With Visual Studio, you can edit the Additional Include Directories entry in Project Properties -> Configuration Properties -> C/C++ -> General to also point to a folder holding the SDK headers.
Now let’s configure deployment settings to automatically move the.bundle to the right folder after building. In Xcode’s project navigator, click on the project name to view its settings. Click on the bundle target and then the Build Settings tab and do a search for “deployment” settings.
Find the entry for Installation Build Products Location. We need to change this to point to the Unity Assets folder. If you followed the directory structure suggested earlier in the tutorial, the Xcode project should be located within the Unity project in a sub-folder named NativeCode.
We can use the Xcode variable $(PROJECT_DIR) to get the directory in which the .xcodeproj is located. From this location, the Assets folder is just two levels up, so the entry we need to use for Installation Build Products Location is $(PROJECT_DIR)/../../Assets.
We also need to change Installation Directory to /Plugins so Xcode will drop the .bundle into the Plugins sub-folder.
Plugins`Finally, change Deployment Location to Yes, and Skip Install to No.
At this point, everything should be configured, but building now will fail because PluginList.h tries to declare the example plugins from the SDK. So first delete the contents of that file, then with no more declared plugins, the build should succeed and place a .bundle in Assets/Plugins.
Make sure of the following things before continuing on:
- The build succeeds
- It generates the
.bundleinAssets/Plugins - The bundle is named with the prefix
AudioPlugin
Here’s an archive of the Unity + Xcode project files up to this point:
2. Generate Noise
Now that the Xcode project is building, let’s start implementing the necessary callbacks so that Unity can do some processing with the plugin. Make a new C++ class file and name it something like Plugin_Noise, we don’t need a header file so deselect the option.
Overwrite the file’s content with the following (feel free to remove the comments):
Although we’ve added some of the plugin’s logic, we still need to register the plugin with the SDK so switch over to PluginList.h (should be empty) and add:
You’ll notice that the effect declaration is guarded with the UNITY_OSX macro. Native plugins require building on all platforms that they’ll be deployed on so you can use theif guard to prevent your plugin from being built on incompatible platforms. I haven’t tested this noise plugin on Windows, but as far as I know, it doesn’t make any OSX specific calls, so theoretically it should compile on Windows with msvc.
Because we’ve now included AudioPluginUtil.h in a compilation unit, the project won’t successfully build until we’ve added all the necessary callbacks for the plugin.
Go back to Plugin_Noise.cpp and add callbacks for plugin creation and deletion:
UnityAudioEffectState* state is a pointer created by the SDK that gets passed into all the callbacks. You should store any of the state that the plugin relies on within the void* effectdata member.
Next, let’s add the callback that does the actual processing for the plugin:
ProcessCallback is continuously invoked by Unity with the effectstate, an incoming audio buffer, and an outgoing buffer that needs to be filled.
The audio buffers interleave channels: for the nth sample in the buffer the signal amplitude for channel c (zero-indexed) is found at buffer index [n*numChannels + c]. For example, with a stereo buffer (2 channels), the left channel for the nth sample is at index [n*2+0] and the right at [n*2+1]. Signal amplitudes should be normalized to values within the range [-1.0, 1.0].
We’re almost finished, all we need to do is add callbacks to handle parameter manipulation from the editor:
Putting all these snippets together, the final Plugin_Noise.cpp file should be:
And PluginList.h should have:
Once the files are correct, the build should succeed and place a .bundle in Assets/Plugins. If the build fails with a Code Sign Error, try cleaning (Product -> Clean [cmd+shift+k]) and rebuilding.
Here’s an archive of the Unity + Xcode project files up to this point:
3. Test The Plugin
Open up the Unity project in the editor.
Add an Audio Mixer to the project and name it “Noise” Right-Click -> Create -> Audio Mixer:
Double-click the audio mixer to edit it, then add the Demo Noise plugin we just built to the master channel (notice with Demo Noise selected, the inspector shows a slider for the parameter we exposed):
Create an empty GameObject, add an Audio Source component to it, and set the Output channel to the master channel of the Noise mixer we just made:
Now hit the play button and hear our noise!
Congratulations! If you were able to get similar results as shown in the video, then you have everything you need to start developing native audio plugins for Unity. The following sections will show you how you can speed up development time by running a standalone application for testing and how to use the STK library to simplify signal processing tasks.
Here’s an archive of the Unity + Xcode project files up to this point:
4. Debugging with a Standalone Application
Having the plugin working in Unity is great, but we run into a problem if we want to rapidly make changes and see their results as we develop. Of course changing the code would require recompiling the plugin and this is unavoidable, but what hurts us more is that Unity won’t load an updated version of a plugin if the old one is already loaded. This means a complete restart of Unity is required to hear even the smallest change in a plugin.
In this section we’ll see how we can reuse the plugin’s code within a separate application we create outside of Unity so we can test the plugin without Unity and avoid frequent restarts. For plugin components that are tightly coupled with Unity-specific behaviour, testing with Unity may be unavoidable.
Let’s first create a new command-line target for the Xcode project. Name it whatever you please, I named mine “AudioPluginNoiseStandalone”.
Xcode also creates a new file main.cpp alongside the new target. If you check the new target’s Build Phases, you’ll see that the new .cpp file was automatically added as a compile source for AudioPluginNoiseStandalone.
Let’s also add AudioPluginUtil.cpp and our Plugin_Noise.cpp as compile sources. You can do this by dragging them from the project tree into the area for “Compile Sources”.
Building AudioPluginNoiseStandalone now should work (it worked before adding the sources but let’s check now to make sure it’s still working). Switch the active scheme over to the command-line target:
In order for our standalone application to work with the unaltered plugin code meant for use within Unity, we need to make the plugin think our application really is Unity. We’ll do this by mocking up our own UnityAudioEffectState object and calling our plugin’s CreateCallback and ProcessCallback (and ReleaseCallback for good measure) functions.
Open up main.cpp and add AudioPluginUtil.h as an include, then get rid of anything that’s already in the main method.
In main(), create the mock UnityAudioEffectState and call CreateCallback to initialize the plugin. In this case, since we’re not using a header file for our Plugin_Noise.cpp, we need to forward declare the plugin’s namespace and callback functions so the compiler is aware they exist.
Once the plugin’s initialized, we can call ProcessCallback with the same mock state, an input buffer (in this case nullptr since we don’t care about input), and an output buffer. We can do this however we like. Since Unity must incorporate real-time user interactions into audio, the engine will continually invoke ProcessCallback with a relatively small buffer size, but in our standalone application, we don’t require real-time alteration. Nothing is stopping us from using a single, larger out buffer and only calling ProcessCallback once, so let’s do that.
We’ve used the plugin’s code to render a 2-second signal …but we can’t hear it because the code to actually playback the signal with the speakers doesn’t exist yet. For this, we’ll use STK’s RtWvOut class.
Here’s an archive of the Unity + Xcode project files up to this point:
5. Integrating STK
Download the STK distribution from this page https://ccrma.stanford.edu/software/stk/download.htm and extract the contents from the .tar.gz archive to a location where Xcode will be able to find it. A location might be /Developer/sdks.
We need to first compile the distribution into a static library that we’ll use for linking. Since the source files are given in the distribution, you can also just add the .cpp/.h files to the Xcode project for the classes you’re interested in and forego compiling the library, but that isn’t covered in this tutorial.
Let’s build STK.
If you take a look at INSTALL.md included in the root of the distribution, you’ll find more detailed instructions and options regarding how to build.
On macOS, it should suffice to just open up a terminal, cd (change directory) to the STK distribution, and run ./configure --with-core. The --with-core option let’s us build the library with macOS’s Core-Audio framework. On other systems, you should specify the option corresponding to that OS (see INSTALL.md).
It will generate a Makefile that we can use to compile STK. Once the prompt returns, go ahead and run make.
In the image above, I extracted STK into my downloads folder and built it there because I already have it built in /Developer/sdks/stk-4.5.1.
If everything was successful, you’ll find a libstk.a file in the src directory. This is the static library we’ll use when linking.
Now we need to point Xcode to our STK distribution so it knows where the headers and library live.
In Xcode, go to the standalone target’s (not the project’s) Build Settings and do a search for “compiler flags”. Add an entry for -D__MACOSX_CORE__. Then, using the location of the include directory that comes with the STK distribution, add the location to the C++ compiler’s header-search path with the -I flag. For me, this location is /Developer/sdks/stk-4.5.1/include. Xcode also has an entry for “Preprocessor Macros” and “Header Search Paths”, so you can specify the macro and path using those instead of the compiler flags too.
If there are spaces in your location’s path, you’ll have to surround the entry with quotes. We can find out about more compiler flags and what they do by running gcc --help in the terminal. On macOS, the default gcc compiler is actually clang.
After adding the include directory to the search path, you may now #include the various headers that STK provides and Xcode will know how to resolve them. However, we haven’t yet directed Xcode to link the build with the actual static library we compiled earlier, so although Xcode won’t be displaying any errors in the editor, any .cpp sources that include STK headers will cause linker errors during the build.
I think of the headers as an outline of the various classes and functions that make up STK, while the static .a library provides the actual implementation for those methods. In other words, the headers tell the program how to call the functions that are implemented in the static library.
Now we need to add the static library we built earlier to Xcode’s build phases. Navigate to the Build Phases tab of the standalone target and expand the “Link Binary With Libraries” phase. Hit the + button, select the option to “Add Other…”, and locate the libstk.a file (for me it’s at /Developer/sdks/stk-4.5.1/src/libstk.a). You can also just drag and drop.
Even though we’ve told Xcode where to find the library through the Build Phases, Xcode still fails to link with the library unless we specify a library search path. I’m not sure why this is the case (since we just gave it its exact location), but let’s tell the linker where to find the library now:
Go back to Build Settings and do a search for “linker flags”. Add the path to the folder containing libstk.a with the -L flag (for me this is /Developer/sdks/stk-4.5.1/src). Xcode also has an entry for “Library Search Paths”, so you can specify the path there instead of using a compiler flag too.
In addition to libstk.a, real-time STK programs have to also link with the CoreAudio, CoreMIDI, and CoreFoundation frameworks on macOS:
NOTE: we only configured Build Settings for the standalone target. If you want the plugin itself to use STK as well, configure the Build Settings exactly the same for the bundle target.
At this point, the project should know where to find STK, so now we can finally use it to play the audio signal with our computer’s speakers.
Open up main.cpp and #include “RtWvOut.h". Then add the following just under Plugin_Noise::ProcessCallback():
This is what main.cpp should look like:
We now have a standalone app that uses the plugin to generate sound.
Here’s an archive of the Unity + Xcode project files up to this point: