Unity Playground’s audio blind spot
My experience building an audio package and toolkit for Unity Playground to help junior developers create immersive audio.
Recently the cross-platform game engine Unity released Unity Playground, a collection of scripts and assets empowering users to create small, simple games without any prior coding knowledge. If you haven’t already, check it out — it’s exciting for newcomers, as well as a fun little exercise for those with more than a few late-night coding sessions behind them.
As a sound designer for games and interactive media, I’ve had to learn basic coding skills to implement my ideas and systems in games. I distinctly remember the first time I opened Unity — feeling utterly overwhelmed by the plethora of options and features, like a confused kid in a candy store.
Unity Playground gives you an easy way in. It’s the smooth introduction to Unity you always wanted, without the intimidating lines of coloured text. That said, Unity haven’t provided any tools for easily adding basic audio features, such as basic randomisation of audio, triggering mixer changes, or changing music loops from one cue to another musically. Of course, Unity Playground is a very basic entry into video-game creation, so more advanced animations or other complex systems would be beyond its scope. But creating small random containers or basic mixer snapshot triggering from the conditional scripts (if ‘this’ happens then do ‘that’) is a fairly simple task for the Unity developers. And it would open newcomers up to the yet untapped world of adaptive audio.
That’s why I decided to make some small scripts to help aspiring game developers use more dynamic audio, as well as understand the power that music and sound effects can bring to even the smallest, do-it-yourself projects. (You can see how it works here; when you’ve got 14 minutes to spare).
It’s all in the variations
In Unity Playground it’s possible to load up ‘audiosources’ that play audio on game start or object creation, and loop that audio until the game stops. This means you can have a basic audio loop running, like an icy wind howling through a forest, as background ambience, and this loop would play continuously throughout the game level. But since games are interactive they also require audio that changes with the player’s actions. This means if we want to trigger a sound effect at a specific time, or add a number of different shooting sounds to our character, it falls short (can you imagine Star Wars without those iconic blaster sounds).
If we want to create a more ‘believable world’ for a game it’s important to be a more aware of how our ‘real world’ sounds. Next time you pour yourself a fresh cup of your favourite steaming beverage, or grab last night’s leftovers from the fridge, try to carefully tune into those usually unnoticeable sounds. And I mean all of them; the whole sequence. From the cup’s swift slide across the shelf, to its steady landing on the kitchen table, the clothes gently rustling when you move your arm and the satisfying sound of that first sip. You’ll soon realise that the sound we’re almost always utterly oblivious to contains vast amounts of information.
This helps us orientate ourselves without knowing or needing to know. For example, you could close your eyes and still know if you set the coffee mug on the table, just by the sound of ceramic against the wooden tabletop (I don’t recommend trying). Sure you can feel it, but if you didn’t hear the two objects touching, you’d start wondering if you were deaf or the world was out of order. Now consider the sound of the same mug being placed on the table, and repeat the process. Listen to all the small variations each time you place it down, and you’ll also realise that a lot of the vibrancy of ‘life’ itself comes from these subtle variations.
This starts to matter when you want to create an immersive world. That’s why randomly triggering different footstep sounds makes for a more believable character. You can then add random volume and pitch randomisation to create further variations from the same set of audio files. That’s how you get closer to recreating that ‘real world’ feeling.
Obviously we don’t have continually changing music throughout our daily lives to score every decision, action and conversation (imagine the day when a sweet violin concerto accompanies your first Valentine’s Day date). But we know from a our long film history that music adds emotional layers, deepening the understanding of the story for the viewer. Since games aren’t linear like film, playing the same music repeatedly doesn’t make sense. Also, we can’t always be sure that the player’s hard-earned, climactic boss battle defeat will be accompanied by suitably glorious music.
It’s fairly easy to instantly switch the music to a different track, but then the continuity is lost and it breaks the music and game flow (known as immersion in games). Having several music loops and a way to transition dynamically and musically therefore, is a must. Not all games need evolving dynamic music, but should at least reflect the ups and downs of your gameplay experience, or appropriately transition out of or between music loops. A while back we conducted an experiment revealing that adaptive music increases gameplay time by up to 30%.
You might’ve noticed the incredible adaptive soundtracks in AAA titles such as Uncharted and Red Dead Redemption. However, this seems less common in small games found at game jams, and certainly isn’t included at all in Unity Playground, which is aimed at even smaller games. They’ve might’ve missed the opportunity to introduce junior game developers to adaptive music, as well as basic audio features like multiple, randomised sounds per object, at a stage early enough for them to benefit from. In the same way that Star Wars blasters sounds make the shooting memorable, a blood-curdling scream dramatically increases the satisfaction of slaying monsters. In more comprehensive audio tools such as Wwise, FMOD and Fabric, these features are built in from the start. Unity does allow these features, although somewhat ‘hidden’ behind a few lines of code that need to be written to accomplish this.
The tools
As discussed earlier, in order to create more variation we can use randomisation of audio to create a more dynamic game world soundscape. This is why I built specific tools that I’ve included in the downloadable package. These trigger random audio files in slightly different ways, ensuring you never hear the exact same audio clip twice in a row, as long as there’s more than one audio file. The scripts I put together for Unity Playground are all (except for one) developed to use random audio clips and random pitch, so that same sound played again will always differ slightly.
The scripts can be split into two different groups: automatic scripts and triggered scripts. The triggered scripts can be used together with the conditional scripts in Unity Playground (for example, if the player enters this area then trigger a sound). These work the same way as the other Action scripts provided by Unity. There are three triggered scripts — one that triggers random audio, one that triggers a “mixer snapshot” (a way of recalling a mixer setting saved in the audio mixer, like different channel levels or effect settings), and one that triggers the transition to a new music loop (or cue).
This works together with the Music Container script, which can hold multiple music loops. A project can contain as many Music Containers as your computer can handle, and the “change cue script” changes cues across all music players for them to stay in sync. It’s possible to select how fast the transition happens, say, on the next beat or next bar, while always in time with the music. It’s possible to have multiple music containers playing different stems of a music loop. This means you could have a drum or guitar part playing separately, and then fade in one or the other by adjusting the volume using the mixer snapshot. This creates the illusion that the music is progressively changing, even though it’s been playing throughout.
The other scripts don’t require a trigger, and can therefore be considered automatic. Like the triggered scripts these also pick random sounds from a list, and apply random pitch every time a sound is played. For example, there’s an automatic footsteps script that can be heard in the tutorial video, that plays random sounds when the object it’s attached to moves. Then there are two scripts that can be used to play audio continuously. One plays with a time delay in between sounds, while the other picks a new sound when the current one stops playing. Using all these different tools, one can create adaptive soundscapes and evolving music throughout a game without touching any code.
Educating the newcomers
Of course, Unity Playground isn’t meant for massive video game titles and the fully immersive virtual experiences competing with them. Because this development package is primarily aimed at beginners and others who might one day continue making their own games, it becomes more important to think about what functionality to include.
The tools used for a particular project, and their limitations, will ultimately dictate what they create and how. If aspiring developers aren’t given options, they might miss out on adding music loops or varied shooting sounds before further development, unless someone else teaches it to them. For sound designers or composers it’s even harder, because they aren’t usually developers and would need to learn how to use more complicated middleware (Wwise, FMOD etc.) to access basic functionalities such as randomisation and dynamic music.
Sure, Unity Playground doesn’t offer animation tools or more complex features for deeper gameplay, but the simply effective audio features offer some easy wins and opportunities for depth or even humor (fart sounds, anyone?). The sound of an opening door is a great example of this. In Unity Playground it’s possible to turn objects on or off, basically hiding them. If the door is just hidden, you’ll get visual feedback. But by adding the sound of a opening door, you can compensate for the lack of animation and provide the extra feedback necessary for a player to understand that the door is open. Footsteps are a great example, because you believe that a character is actually moving and ‘walking’. Wind howling, a distant monster’s growl and echoing drops of water tell you the dungeon you see is wet and scary, even though there aren’t water or ghosts on the screen. As discussed earlier, nothing really ‘happens’ without the associated sound.
As I see it, Unity Playground is all about an easy introduction to game development, but only provides tools for game design and simple programming concepts. But with the tools I’ve put together at Melodrive, it’s also possible for aspiring sound designers and game composers to get into interactive/adaptive audio and experiment with ideas.
In the end it’s not about creating amazing AAA games, or advanced audio systems — it’s about giving users audio tools to make projects more fun and alive. By providing these functionalities from the onset rather than needing to code, it’s more likely that adaptive audio will be explored. Simultaneously this also helps raise the level of what the expected minimum for interactive audio and music should be, and why it matters.
Hopefully this means that future composers and sound designers won’t have to explain to developers why they need multiple footstep sounds. Ultimately, I hope other people find these tools useful for their own small projects, and that it will inspire and spread an ounce of the incredible passion I feel for adaptive audio.
The package is free to use and comes with bunch of audio assets, too. Go get them from the Unity Asset Store or our Github, and up your Unity Playground audio game.
