LIT REVIEW & WHITE PAPER
Changing Brains with Music Video Games
The potential for music in digital therapeutics
Neurodiversity is the term we use to describe the fact that all of our brains are slightly different. Most of us fall along an average, resembling one another in most of the ways that matter, but small variations in something as complex as the brain can have life-altering consequences. Such is the case with dyslexia, a genetic disorder which interferes with the encryption and decryption processes involved in reading.
Our brains are malleable and they can be changed through repetitive neural activity. This adaptable trait is called neuroplasticity and it is the reason we get better at things with practice. Since its discovery, scientists have been trying to figure out how to leverage neuroplasticity to help people who were born with special neurologic conditions, or acquired them through trauma (like a stroke).
Queue the music. Literally. A large section of the scientific literature around brain plasticity is dedicated to investigating how the practice of music can activate plasticity by leveraging the entire nervous system, not just the brain. Today, neurologic music therapy is used to treat a number of neurologic ailments including dyslexia.
Unfortunately, practicing music in a therapist’s office or finding someone qualified and available can still be fairly challenging. This is where video games come in.
Recent studies have begun to investigate how the immersive experiences afforded by digital games can alter the composition of our brains. The field that studies this is called cognitive consequence gaming, and the application of digital technology to treat neurologic disorders is referred to as digital therapeutics.
For its part, music motivates, in every sense of the word. Its role in both digital media and physical therapy cannot be overstated. This shared strength has, at its core, the fact that music is naturally moving. Rhythm activates muscles reflexively and links people to an external harmony both physically and emotionally.
The purpose of this article is to summarize the evidence that explains how music may be used to usher in a new generation of therapeutic video games that can make effective treatment accessible and affordable.
This literature review is the result of my work at Mila (mila-learn.com), a company developing games that combine neurologic music therapy and cognitive gaming science to train cognitive skills that treat learning disabilities. The article is split up into three short literature reviews:
- First, a review of the field of cognitive consequence gaming.
- Second, a review of the neuroscience of music and reading.
- Third, a review of the applications of neurologic music therapy for reading disabilities including dyslexia.
The final section of this article will draw on each of these scientific fields to expand on the potential for music in digital therapeutics.
Music motivates, in every sense of the word
Cognitive Consequence Gaming
Today, it is common knowledge that games are capable of keeping users engaged in all sorts of activities frequently and consistently. However, these are only useful in therapeutic contexts when players are able to transfer new skills outside of gaming environments.
The theory of the specific transfer of general skills suggests that the best way to transfer skills out of a training environment is with repeated exercise of that skill, immersion in a variety of contexts, and increasingly difficult challenges.¹ The field of cognitive consequence gaming seeks to empirically determine how theories of skill transfer apply to gaming.²
Most recently, studies have examined games that claim to train executive functions and other cognitive skills. A 2018 study that looked at the self-described “brain training” application Lumosity ran two experiments, one short-term study where 72 participants, aged 18 to 30, played for 3 hours over 2 weeks, and one long-term study where 51 participants played for 20 hours over 16 weeks. They found that players improved in their application of the targeted cognitive skills within the Lumosity games, but that they were unable to transfer these levels of performance to outside tests.³
Cogmed, a game (suite of games, actually) that claimed to train executive functions, was found to produce short-term positive effects that did not generalize. It was able to train visuo-spatial working memory but had no effect on verbal working memory.⁴ In the area of popular consumer games, Tetris was found to improve the spatial skills involved in mental rotation with 2-d but not 3-d shapes, and with Tetris shapes only.⁵
Authors concluded that despite continuous training, skill improvements did not transfer because the games failed to provide diverse contexts which would have forced players’ to develop the cognitive resources to make their skills generalizable.
The most extensively studied games have been first person shooters. These types of games were found to be capable of improving players’ perceptual attention skills.⁶ In a review, it was highlighted that their core mechanic endlessly rewarded and punished players based on their perceptual attention skills, and did so in many contexts (e.g. weapons, rules, environments, stories, social).⁷ Some authors also noted the importance of these games’ intuitive, discovery based systems which afforded players the ability to avoid overwhelming frustration and stay engaged longer, a desirable trait for any potentially therapeutic solution.⁸
Researchers used lessons from this field to build games with the explicit purpose of inducing cognitive changes.
Teams at the University of California, Santa Barbara and New York University were able to significantly improve children’s’ mental shifting ability with a game called “All You Can ET” in which players fed aliens based on rules that constantly changed.⁹ A game called Neuroracer dynamically adapted its difficulty to players’ skill levels and reduced multitasking costs in older adults (aged 60 to 85) below the levels of those achieved by untrained 20 year old’s. The effect lasted for up to six months.¹⁰
These authors highlighted the importance of strategically balancing focus on the cognitive task at hand with diverse and engaging contextual elements.
A game called Neuroracer… reduced multitasking costs in older adults below the levels of those achieved by untrained 20 year olds
Cognitive Consequences of Music in Gaming
Music in video games has been found to affect players’ levels of stress ¹¹ and even their perception of time ¹² but a particularly relevant 2016 paper by Dr. Amanda Pasinski compared the music perception skills of musicians to people who played instrumental video games (Guitar Hero and Rockband) and a control group who did neither. The sample size was 15 per group. Both musicians and video game players each had to have years of experience in one and not the other form of play.
The results showed that musicians and video game players achieved the same levels of musical perception and that both outperformed the control group.¹³ Dr. Pasinski’s work was among the first to show that including physical activities in music based games had the potential to affect significant changes in players’ perceptual abilities.
Musicians and [music] video game players achieved the same levels of musical perception, and both outperformed the control group.
The Neuroscience of Music
The application of music to therapeutic practice was, in part, inspired by studies that compared the brain and cognitive development of musicians and non-musicians. One early study identified above average development in musicians’ corpus callosum, the mass of white fibers uniting the left and right hemispheres of the brain. The authors speculated that the intensive two hand coordination exercise of playing an instrument caused a massive flow of information between the two hemispheres.¹⁴ A few years later another study discovered that areas in the left hemisphere of the brain involved in auditory perception and the assignment of meaning to sound were more developed in musicians.¹⁵
More recently, diffusion tensor imaging or tractography, which measures the diffusion of water molecules, has allowed neuroscientists to visualize the path and directionality of fibers in the brain. Using this technology, one group of researchers found that cortical thickness between musicians and non-musicians differed most in the frontal-temporal areas, and that activity in the two regions were more highly correlated among musicians.¹⁶ Such stark contrasts in brain development resonated with research that compared cognitive abilities between the two groups.
Studies conducted with large groups of children concluded that musical training improved verbal IQ, motor learning, non-verbal reasoning, and executive functions.¹⁷ ¹⁸ The importance of music in verbal cognition was highlighted in research by Dr. Nina Kraus which identified the common processes underlying verbal and “musical” language and highlighted the effect of musical learning on the auditory characteristics of speech.¹⁹ Dr. Mireille Besson expanded on this and focused on the idea that improving the perception of music could improve the perception of speech.²⁰ ²¹
To fully appreciate how the brain development caused by musical practice can affect peoples’ interactions with written language, an understanding of the neuroscience of reading is indispensable.
Reading & The Brain
Speech is natural, reading is not. One is not taught to speak, it is learned from listening to others. The brain hears a word, stores its sounds in a specific order (the inner ear) and plays it back (the inner voice) on an auditory rehearsal loop until it can enter the short or long-term memory where it begins to form associations.²² Writing and reading are encryption and decryption processes that allow for the quick recognition of a word and all of its associations, these processes must be taught. The encryption process can be characterized in three steps:
- Processing the sequence of sounds associated with speech which takes place in the parieto-temporal area of the left hemisphere.
- Forming associations between the processed sounds and other contextual stimuli which occurs in Broca’s area located in the frontal lobe.
- Development of the “word form,” a neural model of the word and its associations, takes place in the occipital-temporal area close to the visual cortex region.
Once the “word form” is created, the experience of recognizing a word approximates the experience of recognizing a familiar face. Activation of the word-form area allows for instant access to all features of a word which is essential for fluent reading. Poor readers make little use of the word-form area.
Our understanding of reading has its origins in the Connecticut Longitudinal Study which sought to investigate the nature of reading difficulties. Dyslexia was found to be the most common diagnosis underlying reading difficulties. It was revealed that the ability to break apart the smallest sounds in language, phonemes, was essential to learning how to read fluently and that dyslexic readers struggled significantly with this task.²³ For example, a child with dyslexia may be unable to yield the word “at” when asked to subtract the “c” sound from the word “cat.”
The ability to break apart the smallest sounds in language, phonemes, was essential to learning how to read fluently and dyslexic readers struggled significantly with this task
There are only two other causes, besides a lack of phonemic sensitivity, that have been identified as potential causes of dyslexia. 1) A deficit in processing speed ²⁴ and 2) abnormal development of cells in the visual system sometimes referred to as “visual dyslexia.” ²⁵ While tests for dyslexia include differentiation between “types” these reflect markers of severity in the phonetic type and deficits in executive functioning rather than alternative causes. ²⁶
Studies went on to characterize the perceptive skills involved in phonemic awareness (the perception of phonemes). Early work on the subject found that speech perception was the best predictor of later reading skills.²⁷ In 2002, a large-scale study with 4 to 5-year old’s found evidence to reinforce this conclusion.²⁸ A 2003 study linked poor perception of the temporal aspects of sound, such as duration and rhythm, to deficits in phonemic awareness.²⁹ Another study in the same year found that the ability to identify the contours of a sequence of pitches was strongly correlated with the phonological aptitudes involved in reading.³⁰
By 2012 researchers had linked perception of voice onset time (the audible feature that distinguishes spoken consonants such as “d” and “b”) to phonemic awareness ³¹ and in 2015 one study linked rhythmic abilities to grammar.³²
Eventually, neuroscientific studies revealed two cerebral particularities associated with phonemic insensitivity in developmental dyslexia:
- The first was a deficit in inter-hemispheric communication due to structural differences in the corpus callosum (the mass of white fibers that connects the hemispheres of the brain).³³ ³⁴
- The second was a communication failure of phonemic representations between the frontal lobe and the audio-processing regions in the left hemisphere.³⁵ ³⁶
Note that the same areas that were found to be overly developed in the brains of musicians were underdeveloped in dyslexic individuals. In 2016 a team of researchers connected neuroscientific and cognitive findings when they used brain imaging results to suggest that dyslexic brains had difficulty aligning fluctuations in auditory neurons with fluctuations in the amplitude of audible speech.³⁷ In other words, the neurons and audio were out of sync.
This deeper understanding of reading difficulties began to explain other cognitive observations that had been correlated with development dyslexia. For example, dyslexic individuals also had trouble with naming speed and often struggled with having words “on the tip of their tongues.” Naming speed refers to tasks that test verbal working memory (the ability to embed and retrieve codes from memory) which is reinforced by the auditory rehearsal loop described earlier. Intuitively, inner ear and inner voice are directly affected by auditory processing problems.³⁸
The same areas that were overdeveloped in the brains of musicians were underdeveloped in dyslexic individuals
Neurologic Music Therapy for Dyslexia
The direct link between rhythmic skills and reading was first explored in a 1951 paper by Mira Stamback.⁴⁰ In 2009, a study of 1,028 children aged 5 and 6 years old used the rhythmic patterns established by Stamback to show that the simple reproduction of rhythm was predictive of reading in early grades.⁴¹ Another study that looked specifically at dyslexic children found that they improved their reading skills, phonological processing and written transcription skills after a 15 week program that trained rhythm and timing skills (once again, pitch skills were inconsequential).⁴²
In 2008, Dr. Mireille Besson’s team assembled two groups of non-dyslexic 8-year old’s and assigned each group to a music or painting program (each designed to be engaging and motivating). The results showed that 6 months of musical learning increased children’s ability to discriminate between height variations in language and read phonologically complex words.³⁹
As a follow-up to their 2008 experiment with non-dyslexic children, Dr. Besson’s team placed dyslexic and non-dyslexic children into either a musical or visual arts training program. They measured the impact of these programs on sensitivity to voice onset time by having the children watch cartoons on mute while they were played a series of syllables of different durations. They measured the brains’ responses to the audible stimulus (the cartoons were meant take focus away from the sound).
Prior to the training programs the team had found that the brains of the dyslexic children were less sensitive to variations in the duration and timing of stimuli but did not differ from controls in detecting height (pitch) variations.⁴³ After the six month program dyslexic children’s perception of voice onset time had normalized and their perception of sound duration had improved.⁴⁴
Prior to the training programs the team had found that the brains of the dyslexic children were less sensitive to variations in the duration and timing of stimuli
In one other study that compared systematic rhythmic training to visual arts practice, musical training was found to have a more significant effect on auditory attention and several auditory perception and reproduction skills.⁴⁸ The visual arts program was designed to promote visuo-spatial skills, dexterity and creativity through painting whereas the music program emphasized use of percussion, syllabic rhythms (ta..ta..ti.ti.ti.) and rhythmic body movements accompanied by music. In the end, rhythmic production ability was the best predictor of performance in phonemic fusion and segmentation tasks which are used in assessments of dyslexia to provide measures of phonemic awareness.
A study by Dr. Nina Kraus’ team found that one year of musical instruction for 8 year old children benefited the rhythmic skills of those considered at risk of a learning disability significantly more than controls.⁴⁵ Another two studies examined these effects in a shorter time-frame. They looked at dyslexic and dysphasic children’s ability to identify incorrect syntax in sentences before and immediately after listening to a rhythmic primer and found a significant improvement.⁴⁶ ⁴⁷
Using this scientific insight, Dr. Michel Habib and his colleagues in France proposed new methods for using music to rehabilitate learning differences. The core of their method was the integration and consistent practice of multi-modal musical interactions that could activate brain plasticity and achieve the anatomical changes observed in the brains of musicians.⁴⁹ This also allowed them to target cognitive and executive functions influenced by the same type of practice. The method was intended as a complement and not a substitute to any requisite speech therapy work. Dr. Habib and his team compared their method to a visual arts program over 6-weeks and found improvements in phonemic awareness and working memory.⁵⁰
A year later they tested the method in two different settings, an intensive 3-day program that met 6 hours per day, and a 6-week program that met 4 times a week for a total of 3 hours per week.⁵¹ The children who participated were severely dyslexic. In both settings, those who practiced music improved their abilities to perceive the duration of sounds and categorize phonemes. The longer setting also measured reading, phonemic awareness and auditory attention. They saw improvements in all three and found that these effects were maintained up to 6 weeks after the program had ended. Following these findings, Dr. Habib’s team met with the founders of Mila with the vision of scaling musical therapy techniques through digital media.
Those who practiced music improved their abilities to perceive the duration of sounds and categorize phonemes
Digital Music Therapy
Reading disabilities are most effectively treated early in a child’s life, preferably while they are still learning to read. The promise of music therapy delivered through digital games is to frequently engage children in therapeutic activities unrestricted by standard treatment settings.
Ultimately, the elements that make digital cognitive training effective are the same ones that make neurologic music therapy effective. The Handbook of Neurologic Music Therapy written by Dr. Michael Thaut set the elementary rules for learning-oriented motor-therapy: ⁵²
- Repetition
- Feedback
- Cueing
- Task orientation (instruction)
- Active learning (requires effort and interaction)
- Ecological validity (functional relevance)
- Shaping (adaptive difficulty)
- Motivation
Target Cognitive Skill - Rhythmic Perception & Production
To combine these two technologies into a therapeutic solution that supports the treatment of reading disabilities, rehabilitative practice should target players’ perception of the temporal aspects of sound. The objective in rhythmic training is for a player achieve synchronization (entrainment) with a beat or its period (time between beats).
A player’s ability to entrain with complex rhythmic patterns reflects a higher ability to perceive things in time (temporal perception). However, to achieve long-term improvements in temporal perception the training must be capable of activating plastic changes in the brain. The earlier discussion around the impact of music and neurologic music therapy supports the use of multi-modal interaction to activate brain plasticity.⁵³
In one direction, repetitive physical movement helps refine anticipation by tying spatial distance to the period of a beat (this is why walking is a common way to teach children about rhythm). In the opposite direction, anticipation of audible cues primes motor neurons for more effective movement.⁵⁴ This approach leverages the reflexive link between auditory and motor neurons to gradually strengthen cortical connections between the relevant areas of the brain.⁵⁵
[Multi-modal interaction] leverages the reflexive link between auditory and motor neurons to gradually strengthen cortical connections between the relevant areas of the brain.
Thus, it is important that a therapeutic game elicit a physical response from players using audible cues. Simultaneously, they should allow for players to adjust or otherwise correct their physical interactions using audible feedback. This feedback process, otherwise known as sonification, is critical for the effective reinforcement of audio-motor skills⁵⁶ and should be facilitated by the careful cueing, feedback and instruction mechanisms that characterize high quality games and personalized therapy sessions.
Create a Diverse & Engaging Experience
The next step would be to design an experience that meets the criteria necessary for skill transfer in digital games. Digital platforms cannot engage the players’ senses as directly as instrumental play with another human being. Hence, the importance placed on context diversity by the cognitive consequence gaming literature.
In this regard, “context” in gaming is akin to functional relevance in Dr. Thaut’s handbook and should be understood to include everything from visual and audible aesthetics (or game aesthetics) to rules of play and interaction diversity (or game mechanics).⁵⁷
Diversity is important when considering game aesthetics and interactions. Engaging the senses through different visual, audible and tactile elements (i.e. vibration, device tilt, positioning, or movement) should avoid specialization to a limited set of gaming environments and interactions. However, aesthetic designs should also avoid providing players with alternative cognitive strategies to succeed.
For example, a 2017 study that looked at existing musical games and examined their potential for rhythmic training found most games relied heavily on reactions to visual cues rather than prediction from synchronization.⁵⁸ This afforded the players the opportunity to cheat themselves by avoiding strict exercise of auditory-temporal perception and audio-motor coordination.
Games that relied heavily on reactions to visual cues rather than prediction from synchronization afforded players the opportunity to cheat themselves
The use of diverse game mechanics is an opportunity to engage players in active learning and expose them to different applications of the target cognitive skills. Nevertheless, it is important that repetition remain a priority and diversity be appropriately sequenced with difficulty and mastery.
Prioritizing repetition simply means targeting the same cognitive skill over and over again (in this case temporal perception). This does not exclude changing the activities or rules of play. In fact, achieving entrainment while splitting focus with another cognitive task, or shifting focus to apply the target ability in different ways, reflects greater ability because it reduces the mental resources available to entrain to a rhythm.
Additional cognitive tasks may be considered part of a “diverse context” and contribute to functional relevance by allowing the player to synchronize in situations where they previously would not have been able to. The key is to make sure players are always figuring out better ways to perceive rhythm by challenging them with measurable and enjoyable tasks.
Adaptive Learning
Without getting into too much detail, there are important considerations in machine learning for any who intend to make this type of solution truly accessible. Diagnostics are an important first step. Plenty of digital reading tests exist online. For dyslexia please visit our friends at the AppRISE Project who are working with scientists all over the U.S. to develop a digital diagnostic test for Dyslexia.
A diagnostic baseline and a cognitive benchmark for the target perceptual skill allows developers to measure the training effect of each element discussed. Deploying them in diverse arrays that allow for intensive training and a consolidation period may prove superior to more gradual increases in difficulty. Machine learning affords neuroscientists and developers the opportunity to work together and maximize the well-being of each and every player.
Conclusion
This article has summarized a number of literary resources that describe the cognitive & neuroplastic effects of gaming, brain development in musical practice, the neurologic causes of dyslexia, musical treatments for dyslexia, and the foundations of strategies that may effectively deliver those musical treatments through games.
Technological development is rapidly converging towards a world in which high level ability is exceedingly valuable. The tools and systems we’ll need to educate the next generation will have to evolve to meet the challenge. Embracing those things that bring us joy is first step.
