neurons.mp3
What goes on in the brain when we listen to music, and what can we learn from monitoring brain activity while listening?
Music is a core component of human experience, often a daily part of our lives. A topic that interests auditory neuroscientists is that of music processing. Songs can be broken up into many smaller parts, each of which have their own complex details. Recent research from University of California, Berkeley has made incredible developments in understanding how we process music, as well as how to decode and reproduce sound from information that is communicated between neurons in the auditory cortex. Participants within the study listened freely to “Another Brick in the Wall (Part 1)” by Pink Floyd while their neural activity was recorded. The group at Berkeley were then able to take the neural recording and reconstruct them back into the original song, with impressive accuracy.
In previous research, music perception has been shown to occur across many parts of the brain, including both cortical and subcortical regions. Previous research has shown that while both brain hemispheres are involved, the right is more active than the left. Auditory processing is very complex and includes many factors, such as timbre, pitch, rhythm, harmony, and melody. This study aimed to investigate and explore the neural mechanisms engaged to process complex auditory stimuli, like a song₁ . When it comes to processing music, many different brain areas are involved at different levels. Melodies, for example, are not processed note by note but are processed by the relationships between pitches in the melody. Our prior experience with music also affects our processing, because we learn to expect certain notes with others. This is a reason why even non-musicians can usually tell when a sound is out of tune.₃
Neural recording is the process of measuring and analyzing the activity of individual or multiple neurons. When neurons communicate, electrical current flows into the dendrites of the neuron, through the cell body, and down the axon to talk to the next neuron by sending a postsynaptic potential to its neighbor. There’s several ways to measure this activity, but we’re going to talk about electrodes. Electroencephalography, or EEG, uses many electrodes placed onto the scalp to measure brain activity. The method used in this experiment is similar, but more invasive. ECoG, or electrocorticography, is just like EEG, but the electrodes are placed directly onto the surface of the patient’s brain.
In order to explore the underworkings of music perception, they wanted to see how accurately auditory input (a song, in this case) could be recreated using an auditory spectrogram of direct neural activity. An auditory spectrogram is like a music score, but made out of the frequencies of a song. The frequency activity collected from participants’ neurons can make something like this as well. Using ECoG, often used to map parts of the brain and make predictions about cognitive/motor functions and their origins₂ , neural activity was recorded and collected for analysis.
Recordings were collected from 29 neurosurgical patients passively listening to the song, each with 2,668 electrodes placed directly onto their cerebral cortex. Out of the 2,668 total electrodes, 347 of them had significantly similar frequencies to the original song’s frequencies. This means that the frequency activity from the neurons firing under the electrodes could be predicted from the audio spectrogram of the song.
There has been shown to be a significant overlap in neural activity between speech and music processing₁ . Considering this, the reconstruction of neural recording into Pink Floyd used a neural net to decode the auditory information, a strategy previously used in speech decoding. They also used a linear regression model, and compared the results. Not only was the group able to reconstruct the song with reasonable accuracy from direct neural recordings, but confirm a right-hemisphere preference for music perception.
Nonlinear models are more suited to capturing abstract and complex concepts compared to linear models, as well as have the ability to adapt to more variable changes like pitch or background noise₄. Additionally, nonlinear models have a nonlinear relationship between input and output (as the name suggests), which matches the nonlinearity that occurs in auditory stimuli and neural responses.
From both linear and non-linear reconstruction, the models classified the auditory data as highly correlated to parts of the original song spectrogram. Simply put, the difference between linear and nonlinear decoding is that a linear system has an output directly proportional to its input, whereas nonlinear systems do not. The nonlinear models overall had more accurate decoding, which reinforces the idea that methods for speech decoding work well in auditory decoding as well.
So, what does this mean for the future of auditory neuroscience? This strategy of auditory decoding could be used for stimulus reconstruction in clinical settings, like for patients with auditory processing disorder. Additionally, these developments could be exciting for the future of Brain-Computer Interfaces, or BCIs. The linear models involved with approximating spectrogram signals to neural activity could be a strategy to increase ease of interpretation. Removing electrodes had no effect on the spectrogram, but taking them from regions of the brain associated with music processing leads to decreased accuracy. Considering this, BCI applications should focus more on targeting areas in the brain that deal with auditory processing rather than just using thousands of electrodes. For example, focusing electrodes onto the superior temporal gyrus, confirmed to be heavily involved in music processing, would increase efficiency and accuracy. While there is still much to be done, this is an exciting step towards the future of auditory neuroscience.
Sources:
