How To Turn Gameplay Into Music
Why video games can expand the concept of ambient music
If an orchestra were transformed into a vast forest during a piano concert, the pianist would no longer be accompanied by strings, percussions or wind instruments, but by the sounds of wind, foliage and crickets. But what if you could walk and explore this forest, and then reach valleys, deserts, mountains, changing the rhythm and harmony of this concert?
I know, this may just seem like a dream…for theater performances. But you can have exactly that experience in The Legend of Zelda: Breath of the Wild. Do you want to understand how this is possible and why video games are the definitive means for the expression of ambient music? Follow me in this essay.
The power of natural sounds and sound effects
Sound effects when opening a door or walking in a desert are not part of the soundtrack of a game or a movie, but they can be just as important as suggesting fear, suspense, loneliness, among other sensations.
In the classic loading moments of the first titles in the Resident Evil series, a sound effect of a door opening through the darkness was more than enough to incite fear and suspense. And the sound of Wander’s footsteps echoing in the desert was also enough to echo loneliness in Shadow of the Colossus.
In addition to the sensations, the sound effects may suggest the presence of someone approaching, as in Minecraft when hearing the zombies’ moans. And they even have the power to imitate sounds from other objects. This is the case, for example, of a brilliant scene of the film Wild Strawberries, by Ingmar Bergman.
At the beginning of the film, the protagonist is immersed in a very strange and confused dream, and the silence together with the sound effects feed the sensation of strangeness and ignorance. In addition, there is always something making a monotonous sound and at a regular binary rhythm.
First you hear a bass sound from a piano key, then, as the scene changes, the source of sound, in the same binary rhythm, becomes that of a wooden wheel hitting against a pole, then the creak of a cart wooden. Finally, when the character wakes him up, it is discovered that those sounds were imitating the ticking of his clock in his dream.
We can go beyond that. The beginning of the film Solaris, by Andrei Tarkovsky, is proof of how the effects of the scenery and the actions of the characters can even suffice to create an incredibly beautiful and as memorable moment as a strong melody made by a composer.
Looking at this seashell, in which I seem to see evidence of “construction” and, as it were, the work of a hand not operating by “chance,” I wonder: Who made it? […] And now I strive to find out how we know that at a given object is or not made by a man? — Paul Valéry apud Pierre Schaeffer (In Search of a Concrete Music)
It is not enough to let the sounds flow naturally, it is also possible to edit them in a kind of montage. After all, “Natural sounds” like footsteps or ticking clocks have musical notes and, although they are not obtained from a musical instrument, they can be used to make music in a montage; more specifically, “concrete music”, as the French composer Pierre Schaeffer called it.
However, in many cases it may be desirable to color or complement natural sounds or sound effects. And there are two ways to do this.
There are two ways to compose with “natural sounds” like steps and clock ticks. One is to cover them up with music, another is to unite them with musical composition. In the first case, we have background music; in the second, ambient music.
These two types of music should not be exactly alike. The background music is produced by sounds of instruments, voice or synthesizers and can be enjoyed independently of natural sounds without prejudice to the purpose of the work. On the other hand, the composition of ambient music depends on natural sounds or sound effects to fulfill its purpose in a scene.
A good example to notice this difference is the beginning of Super Metroid gameplay. On the occasion, while Samus is searching for the last Metroid within the Ceres space colony, there is ambient music, the composition of synthetic sounds mixing with the sounds of the place itself.
After Ridley appears to take Samus to fight, there is a dramatic background music with a well-marked rhythm to generate more dynamism and anxiety during the confrontation and during the escape of the place, which will self-destruct. This musical contrast is also accompanied by a contrast of colors, from blue to red.
Background music
The “background music”, as we know it today, owes much to the idea behind some pieces by Erik Satie called “furniture music” (musique d’ameublement). Satie’s concept was reported by a friend, the painter Fernand Léger:
We were having lunch, friends and him, in a restaurant. Forced to endure a noisy and unbearable music, we leave the room and Satie says: “There is still to achieve a furniture music, that is to say a music that would be part of the background noise, which would take them into account. I think it would be melodious, it would soften the sound of knives and forks, without dominating them, without imposing itself. It would fill heavy silences sometimes occurring between the guests. It would spare them small talk. It would neutralize, at the same time, the street noises entering the game without discretion.” It would, he said, respond to a need.
More than a century later, background music is found everywhere: in elevators, radio and TV programs, gyms, blogs and, of course, video games. An interesting example of background music in games appears in chapter eight (The Itch) of Portal 2.
In almost all puzzles the player is immersed in ambient sound, but in the fifth test chamber of that chapter his reasoning will be accompanied by a background music, Machiavellian Bach. It is an interesting arrangement for harpsichord and synthetic sounds from one of Johann Sebastian Bach’s Six Little Preludes.
This part of the game is particularly interesting because the camera starts with a comment by GLaDOS (ally of the protagonist, at the time) about the song chosen by Wheatley:
“Ohhhh, now he’s playing classical music.”
GLaDOS, Portal 2
This excerpt reveals that background music may eventually exist within the fictional world of a game. It can be recognized and commented on by the fictional characters, so that it serves as a “background” for both the player’s actions and the characters’ actions.
This characteristic of some background music is no longer present in ambient music, since ambient music, in general, contains a meta-fictional aspect; it mixes with the scenery, it is as if it were part of the other sounds in the world where the characters live. Fez, Super Metroid, Portal 2 and so many other games do it very well, varying compositions according to the objects, the acoustics and the nature of the place. And in some games, such as Breath of the Wild, background music can appear and play simultaneously with ambient music, such as when Kass appears in some places playing the accordion.
Ambient music
Ambient music is not in the background of a scene, but is close to the scene, in direct dialogue with its sound effects, and in such a way that, in some contexts, it may even be difficult to distinguish whether a particular note belongs to a sound effect or to the ambient music.
To blend with sound effects or natural sounds, ambient music does not have much emphasis on rhythm. Instead, it focuses on tone and atmosphere.
It often uses textural layers of recorded and edited sounds that can reward both passive and active listening, and encourage a sense of calm or contemplation. In this sense, it is said that this genre metaphorically evokes an “atmospheric” quality.
The OST of the games we mentioned (Fez, Super Metroid and Portal 2), for example, clearly provide this atmospheric quality with synthetic sounds that mix with the environment. But this can also be seen in ambient music even without being made for a game or movie scene. In Solari, for instance, and other songs from Ryuichi Sakamoto’s async album.
On the other hand, ambient music can also use timbres of musical instruments (such as strings, voice, piano and flute). This is because the composition can be made in such a way as to leave “space” to be occupied by external sounds. A good example in this respect are the soundtracks composed by Austin Wintory for games from thatgamecompany, such as Journey, due to the way in which harmony, rhythm and melody mix with the sound of the creatures, wind and sand.
When playing becomes the same as listening
As noted by Johan Huizinga, in his book Homo Ludens, every game (digital or not) presupposes freedom for player to act in different ways, although limited by absolute rules in the fictional world.
For this reason, unlike other forms of artistic expression, such as cinema, video games tend to provide much more interaction and dynamism. This interactive potential of video games results in many other possibilities for music.
This may sound strange, since, over the past few decades, video games have almost always brought elements of other art forms into their fictional worlds. In some cases, they literally brought other types of art into the games, like a movie in The Witness and an opera in Final Fantasy VI.
The past few years, however, have shown much evidence that it is time for video games to be incorporated into other forms of art. One of these, music.
Unlike what happens in other media, both background music and ambient music in video games can be dynamic. They can have timbre, rhythm, harmony or melody modified as the players act.
The examples are abundant. In Flower, the melody (in ambient music) changes and is built as you control a burst of petals. In The Legend of Zelda: Skyward Sword, you change the harmonic tone of the same background music for different NPCs you interact with.
In Portal 2, the rhythm of the music can accelerate or become slower, among other changes, when interacting with portals, gels and other mechanisms. And in The Legend of Zelda: Spirit Tracks, as you step up the stairs of the Tower of Spirits, progressively new instruments appear in the background music, enriching its plurality of timbres, similar to the famous Bolero by Maurice Ravel.
And in addition to dynamism in timbre, rhythm, harmony and melody, some games can also make mathematical and performance parameters dynamic, in order to connect with the mechanics of the game itself.
So, when you go back in time in Braid, your background music also sounds backwards; and when you dive into Donkey Kong Country: Tropical Freeze your background music is muffled (something very common in platformers).
In this way, the possibilities for creating background music and ambient music are considerably expanded in video games. And this is because gameplay can interact with various aspects of composition and musical performance.
However, ambient music is even more benefited by the interaction with the gameplay. This is because, while the background music is an entirely independent composition, but made for a given scenario, the ambient music is made with a scenario, being dependent on its objects and sonorities.
Thus, the composition of ambient music can be vague and minimalist enough in-game so that it can change as the player acts within the fictional world. In this point, I believe that The Legend of Zelda: Breath of the Wild has a lot to teach for the future of ambient music.
The future of ambient music
In an article for Nintendo Blast, I argued that with the Hyrule Field Theme of The Legend of Zelda: Ocarina of Time, players will tend to feel more engaged in the predetermined course of their epic history. In addition, players can be impatient more easily if they stay in the same place for a long time. This is due to the repetition of strong and striking melodic lines as well as the well-marked rhythm and intensity of the sounds.
On the other hand, I argued that, in Breath of the Wild, the music invites the player to explore Hyrule Field in a more free, relaxed, peaceful way, making him more patient, curious and contemplative in his adventure.
Manaka Kataoka’s arrangements or the regions of the world of Breath of the Wild, such as Hyrule Field, do not usually have a marked rhythm or a basic instrumental sound, and the main melody is extremely minimalist with many gaps between the notes; gaps whose silence is filled by the sounds of the environment.
The sequence of notes, however, is not random. Manaka Kataoka arranged the melodies for Breath of the Wild using functional harmony over their respective harmonic fields. Between one gap and the other, the notes oscillate, within a cyclical pattern, between those that cause the most tension and those that cause the least tension until they return to a note of tonic function.
But the implementation in-game of Manaka Kataoka’s arrangements goes beyond minimalism and conventional ambient music. As with Portal 2, OST melodic lines from Breath of the Wild can become slower, accelerate, become more intense and gain new colors.
These changes will depend on the player being seated, running away from an enemy, walking, riding fast, among other factors. Finally, the music also varies organically from day to night and from one region to another.
For all these reasons, a substantial part of Breath of the Wild’s OST is not only dependent on the environment, but also on the player. The music was not only mixed with what’s on the scene, but also with the player’s own actions. And so the boundaries of ambient music are redefined by gameplay.
Evidently, much can still be done on this path. Video games are just the beginning of a long creative process of incorporating gameplay into ambient music.
As with natural sounds and sound effects, it is possible that, in the not-too-distant future, more composers will feel more compelled to use the most varied forms of interactivity (and not only in video games) as raw material as well.
