Deconstructing unpleasantness in videogame sound effects. Part 1: Theory

Denis Zlobin
Aug 30 · 12 min read
Image for post
Image for post

After I wrote a post about disruptive audio in games, a few people asked why didn’t I mention any known acoustic features of annoying or unpleasant sound effects. Back then, I did it on purpose. I think it is a vast topic that deserves a separate, in-depth exploration.

There are some universally hated sounds like the scratching of fingernails over a chalkboard. Nobody in the entire world perceives them as pleasant, disregarding the context. And I think it is fascinating! It implies the existence of certain acoustic features that trigger negative emotional reactions in every human being. And if such features exist, we can analyze, measure, and control them. Before we continue, I need to warn you. This two-part post will be much nerdier than anything I have written before or plan to write in the foreseeable future. It comes with a list of references, so you can follow either links or footnote numbers to see where the information came from.

Disclaimer: I am not an academic researcher, and I should not be considered a credible source of information about the majority of the topics I’m touching here. I’m linking all my sources for the sake of transparency, but remember: this blog is written by a curious idiot with no supervision. Doubt everything, and please reach out to me if you spot any mistakes. I’ll be happy to correct myself.

Similar neural mechanisms can trigger different emotions. There is evidence that the amygdala, the part of the brain that is frequently associated with threat detection and sensation of fear, takes part in processing acoustic information¹ ². Sometimes the line between negative emotional responses becomes too thin to separate them. So instead of speaking about sounds that make us scared, disgusted, irritated, or uneasy, I decided to stick with a more general concept of unpleasantness.

Of course, we can not reduce the entire spectrum of sound-induced negative emotions to a single universal pattern. Our reactions are very complex. Some sounds feel unpleasant because of external meanings that we associate with them. Vomiting noise is one of the most disgusting sounds you can hear³, but it doesn’t seem to have any acoustic properties to make it intrinsically aversive⁴. Other sounds only become unpleasant in specific contexts. You may love the sound of your car’s engine, but not if it wakes you up in the middle of the night. The unpleasantness of a sound may depend on other modalities. For instance, the sound can become more or less unpleasant when it plays on top of a video⁵. There is a variety of contextual factors: misplacement, unexpectedness, lack of synchrony with other sensory information, and so on.

That’s why I’m not attempting to build a universal framework to explain every possible case — to me, it sounds too ambitious and impractical. Instead, I’d love to discover a set of reliable, context-agnostic techniques to analyze and directly manipulate the perceived unpleasantness of sound effects.

Such techniques could be useful not only to instigate negative emotions in the player but also to monitor and reduce irritation from sounds that are supposed to be pleasant or unnoticeable. Ultimately, if proven to be reliable, they could become a foundation for a realtime processing tool or a procedural technology — something like a biofeedback-driven “emotioneering” system mentioned by Garner et al. back in 2010⁶.

To start, I explored some scientific and nearly scientific literature on the topic and tried to list the acoustic properties that make sounds unpleasant to our brains.

Loudness

Even though loudness is a factor, it might not be the most important one. At least in the videogame context, where we deal with a limited dynamic range. One experiment has shown that the volume level doesn’t significantly alter the intensity of the response of the players⁶. A larger-scale study by Ellermeier et al. has also demonstrated no significant effect of loudness on the auditory unpleasantness of the context-independent sounds⁹.

Sharpness

A study on textiles has shown that fabric feels less pleasant to interact with if it makes loud and sharp rustling noises¹⁰. In the videogame context, Garner and Grimshaw report a correlation between sharpness and biometric data they used to assess fear response in players¹¹. Another paper suggests that unpleasantness is a product of sharpness and roughness combined¹².

The study by Ellermeier et al⁹. comes to a similar conclusion but states that the role of sharpness is relatively small compared to roughness. Another study results suggest that the unpleasantness is “more level dependent but less frequency dependent than sharpness” ¹³.

Image for post
Image for post
Photo by Todd Cravens on Unsplash

Ambiguity

In the videogame context, ambiguity is a vital element of horror game soundscapes¹⁶. But the concept of ambiguity in sound is somewhat ambiguous on its own: there are several ways in which we might not get enough information from an auditory cue.

The sound may come from an unexpected place or point to an unusual source. Such ambiguity is contextual and goes beyond the scope of this post. There is also an awkward case of the inability to localize the sound source in space where sound can be ambiguous because of either contextual or acoustic factors. One study examined the effect localization has on the sensation of fear and terror in horror games. The researchers concluded that harder to localize sounds are perceived as more scary¹⁷. However, another experiment did not show any significant effect of binaural processing on the intensity of fear in the videogame⁶.

Noisiness

This statement is also supported by the idea that aesthetic pleasure is a function of processing fluency¹⁹, which on its own, aligns with Daniel Kahneman’s concept of cognitive ease²⁰.

Onset time or attack

In the natural environment, even gradually raising intensity can mean that something is approaching us, and the faster it gets louder, the faster it moves. So, even a slower attack time may put us into an alarm state²² ²³ ²⁴.

I doubt that onset time is a significant factor of unpleasantness. If on its own, it was enough to stress us, we would never leave this state, given how diverse the modern soundscape is. But subjectively, it can boost unpleasantness for the sounds that already have some other feature from this list.

Low pitch

Many believe that ultra-low sound beyond the range of human hearing (infrasound) makes us uncomfortable and frightened. Still, even if true, it has no practical implications in the videogame context. A recent study on low-frequency noise does not support either of those ideas²⁵.

Even though subjectively boomy sounds drag more attention, and we can easily get overwhelmed with prominent low frequencies, I could not find much evidence about the relation between low pitch and negative emotional responses.

Image for post
Image for post
Photo by Dušan Smetana on Unsplash

Roughness

In 2015 a group of researchers led by Luc H. Arnal found out that roughness is a distinct feature of human screams²⁶. Screams were crucial for the survival of our ancestors. And they remain an efficient way to inform others about a threat. That’s why Arnal et al. suggest that our auditory system has a mechanism to segregate screams from other communication signals.

According to them, rough temporal modulations between 30 and 150 Hz specifically communicate danger. In the same study, they found out that artificial alarms also follow the same pattern. And even more, a recent study demonstrated that the same acoustic property is typical for scary film music²⁷. It implies that horror movie composers have intentionally or unintentionally mimicked human screams and other alarm sounds to make their music more terrifying.

Opposing to all that, the study by Kurakata et al. didn’t show that roughness has a significant effect on unpleasantness¹³.

Image for post
Image for post
Photo by Jason Rosewell on Unsplash

Beating or Fluctuation strength

There is a study from 2009 that strongly links the unpleasantness with fluctuation strength, even though the specific term never appears in the text¹. The researchers selected famously unpleasant sounds like dragging fingernails down a blackboard, running an electric drill, scraping a steel fork over a glass, etc. and examined them for similar acoustical properties. The results show that all such sounds share two features: they have a lot of energy in the range from 2500 to 5500 Hz and a lot of temporal modulations in the range from 1 to 16 Hz. The authors hypothesize that those ranges are the ones our auditory system is the most sensible at for both temporal modulation and spectral content.

Andy Farnell describes a similar beating effect in his book Designing Sound²⁸, even though there he calls the effect roughness:

“Roughness as an unpleasant effect may be better understood as the doom tone. It is an interference rhythm composed of cycles in the 4Hz to 8Hz range resulting from dangerous high-energy phenomenon like stampeding animals and high speed wind. When it modulates higher frequency components, the resulting critical band ambiguity leads to an unsettling effect.”

At the same time, neither Kurukata et al.¹³ nor Ellermeier et al⁹. experiments that I’ve mentioned several times, didn’t show that fluctuation strength significantly influences the perceived unpleasantness of a sound.


In the book Psychoacoustics: Facts and Models⁷, Eberhard Zwicker has proposed a psychoacoustic annoyance model based on four parameters: loudness, sharpness, roughness, and fluctuation strength. The model is frequently referenced as reliable in other literature on psychoacoustics, but I failed to find any tool to test it on video game sounds.

In general, the tools allowing to measure the psychoacoustic properties described above are scarce and not very approachable for the game audio crowd. And still, I managed to find some. If you think this post was too short on game-audio relevant information, stay tuned! In the next part, I’m going to put some of this knowledge into action. Here is a sneak peek:

I selected 27 game sound effects that many people reported as unpleasant, annoying, disturbing, or scary. Then I asked 24 non-gamers to listen to those sounds and rate based on how unpleasant they feel. After that, I compared this “unpleasantness score” to measurements of some features from this post to look for correlations. Stay tuned! Update: read it here.

References:

[2]: A. Pannese, D. Grandjean and S. Frühholz, “Amygdala and auditory cortex exhibit distinct sensitivity to relevant acoustic features of auditory emotions”, Cortex, vol. 85, pp. 116–125, 2016. Available: 10.1016/j.cortex.2016.10.013.

[3]: T. Cox, “Scraping sounds and disgusting noises”, Applied Acoustics, vol. 69, no. 12, pp. 1195–1204, 2008. Available: 10.1016/j.apacoust.2007.11.004.

[4]: C. Reuter, M. Oehler and J. Mühlhans, “Physiological and acoustical correlates of unpleasant sounds”, in Joint Conference ICMPC13-APSCOM5, Yonsei University, Seoul, Korea, 2014.

[5]: P. Samermit, J. Saal and N. Davidenko, “Cross-Sensory Stimuli Modulate Reactions to Aversive Sounds”, Multisensory Research, vol. 32, no. 3, pp. 197–213, 2019. Available: 10.1163/22134808–20191344

[6]: T. Garner, M. Grimshaw and D. Nabi, “A preliminary experiment to assess the fear value of preselected sound parameters in a survival horror game”, Proceedings of the 5th Audio Mostly Conference on A Conference on Interaction with Sound — AM ’10, 2010. Available: 10.1145/1859799.1859809.

[7]: H. Fastl and E. Zwicker, Psychoacoustics: Facts and Models.Berlin: Springer, 2007.

[8]: D. Steele and S. Chon, “A perceptual study of sound annoyance.”, in Audio Mostly 2007, 2007.

[9]: W. Ellermeier, M. Mader and P. Daniel, “Scaling the unpleasantness of sounds according to the BTL model : Ratio-scale representation and psychoacoustical analysis”, Acustica United with Acta Acustica, vol. 90, no. 1, pp. 101–107, 2004.

[10]: J. Cho, E. Yi and G. Cho, “Physiological Responses Evoked by Fabric Sounds and Related Mechanical and Acoustical Properties”, Textile Research Journal, vol. 71, no. 12, pp. 1068–1073, 2001. Available: 10.1177/004051750107101206.

[11]: T. Garner and M. Grimshaw, “Psychophysiological Assessment Of Fear Experience In Response To Sound During Computer Video Gameplay”, in International Conference Interfaces and Human Computer Interaction, 2013, pp. 45–53.

[12]: R. Weber and R. Eilers, Combined contribution of roughness and sharpness to the unpleasantness of modulated band pass noise. DAGA ’07, Stuttgart, Deutschland, 2007. ISBN: (978–3–9808659–3–7), p./pp. 565/566. Deutsche Gesellschaft für Akustik.

[13]: K. Kurakata, T. Mizunami and K. Matsushita, “Sensory unpleasantness of high-frequency sounds”, Acoustical Science and Technology, vol. 34, no. 1, pp. 26–33, 2013. Available: 10.1250/ast.34.26.

[14]: L. Gwilliams, T. Linzen, D. Poeppel and A. Marantz, “In Spoken Word Recognition, the Future Predicts the Past”, The Journal of Neuroscience, vol. 38, no. 35, pp. 7585–7599, 2018. Available: 10.1523/jneurosci.0065–18.2018.

[15]: P. Whalen, “The uncertainty of it all”, Trends in Cognitive Sciences, vol. 11, no. 12, pp. 499–500, 2007. Available: 10.1016/j.tics.2007.08.016.

[16]: G. Roux-Girard, “Listening to Fear”, Game Sound Technology and Player Interaction, pp. 192–212. Available: 10.4018/978–1–61692–828–5.ch010.

[17]: I. Ekman. & R. Kajastila, Localization Cues Affect Emotional Judgements–Results from a User Study on Scary Sound, in 35th AES International Conference: Audio for Games, 2009.

[18]: I. Ekman, “Psychologically motivated techniques for emotional sound in computer games”, in AudioMostly 2008, 3rd Conference on Interaction with Sound, Pitetextbackslashaa, Sweden, 2008, pp. 20–26.

[19]: R. Reber, N. Schwarz and P. Winkielman, “Processing Fluency and Aesthetic Pleasure: Is Beauty in the Perceiver’s Processing Experience?”, Personality and Social Psychology Review, vol. 8, no. 4, pp. 364–382, 2004. Available: 10.1207/s15327957pspr0804_3

[20]: D. Kahneman, Thinking, fast and slow. New York: Farrar, Straus and Giroux, 2013.

[21]: S. Horowitz, The universal sense. New York: Bloomsbury, 2013.

[22]: J. Edworthy, S. Loxley and I. Dennis, “Improving Auditory Warning Design: Relationship between Warning Sound Parameters and Perceived Urgency”, Human Factors: The Journal of the Human Factors and Ergonomics Society, vol. 33, no. 2, pp. 205–231, 1991. Available: 10.1177/001872089103300206.

[23]: D. Bach et al., “Rising Sound Intensity: An Intrinsic Warning Cue Activating the Amygdala”, Cerebral Cortex, vol. 18, no. 1, pp. 145–150, 2007. Available: 10.1093/cercor/bhm040.

[24]: D. Bach, J. Neuhoff, W. Perrig and E. Seifritz, “Looming sounds as warning signals: The function of motion cues”, International Journal of Psychophysiology, vol. 74, no. 1, pp. 28–33, 2009. Available: 10.1016/j.ijpsycho.2009.06.004.

[25]: O. Behler and S. Uppenkamp, “Activation in human auditory cortex in relation to the loudness and unpleasantness of low-frequency and infrasound stimuli”, PLOS ONE, vol. 15, no. 2, p. e0229088, 2020. Available: 10.1371/journal.pone.0229088.

[26]: L. Arnal, A. Flinker, A. Kleinschmidt, A. Giraud and D. Poeppel, “Human Screams Occupy a Privileged Niche in the Communication Soundscape”, Current Biology, vol. 25, no. 15, pp. 2051–2056, 2015. Available: 10.1016/j.cub.2015.06.043.

[27]: C. Trevor, L. Arnal and S. Frühholz, “Terrifying film music mimics alarming acoustic feature of human screams”, The Journal of the Acoustical Society of America, vol. 147, no. 6, pp. EL540-EL545, 2020. Available: 10.1121/10.0001459.

[28]: A. Farnell, Designing sound. Cambridge, Mass.: MIT Press, 2010.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store