3 (out of many…) Ways Neuroscience Inspired Machine Learning
By Brian Song
You might be already familiar with the connection between machine learning and neuroscience. Machine learning → smart machines → brain is why we are smart → neuroscience. The brain serves as a natural model for studying intelligence, which is why it’s intuitive to draw from real neuroscience principles to make your algorithm “smarter”. In this article, we’ll explore three instances where various areas of machine learning (just neural networks part) found inspiration from neuroscience. At the end of the article is an interview with a Purdue Neuroscience Professor that I found which I thought was interesting.
Artificial Neural Networks (ANN)
Note: Artificial neural networks are basically the most simple type of neural network. Looks like:
You may already know that artificial neural networks are inspired by real neurons. You may have seen this diagram:
Where the similarities are pretty clear. What might be new is a neuroscience principle called Hebbian Learning, often summarized by “neurons that fire together, wire together”. It states that when two neurons are activated simultaneously/consistently, the connection between them strengthens, which means that neuron B is more likely to fire in response to an input from neuron A. In neural networks, during training, when a pair of neurons frequently contribute to a successful output, the connection between them is strengthened (the weight value increases), mirroring the way synapses strengthen in response to repeated activation in real neurons.
Pruning
During puberty, a process known as synaptic pruning occurs, where connections between neurons in our brain are refined for more specialized and efficient communication. This natural process eliminates redundant connections, enhancing the overall efficiency of the neural network. In the above image, you can see that the synaptic density is a lot higher at 6 years vs. 14 years, which might seem counterintuitive, as you would expect higher density = better cognitive abilities.
In machine learning, pruning involves selectively removing connections between “neurons” in a neural network. This results in a reduction of the model’s parameters, leading to decreased memory usage and improved energy efficiency. Interestingly, it’s possible to prune a significant portion of the network while maintaining the same level of performance, revealing that many of these connections are redundant.
A guide to pruning if you are interested:
https://wandb.ai/authors/pruning/reports/Diving-Into-Model-Pruning-in-Deep-Learning--VmlldzoxMzcyMDg
Convolutional Neural Networks (CNN)
In the 1960s it was discovered that there are certain neurons in the visual cortex that respond to certain orientations. These neurons are described as “edge detector” neurons. This provided the insight that the visual system processes things in a hierarchical manner, where neurons in level 1 detect edges, neurons in level 2 detect more complex shapes, etc.
In Convolutional Neural Networks (CNNs), this hierarchical principle is evident. In the initial layers, the filters are designed to detect edges, representing low-level features. As we progress through the layers, the filters evolve to recognize more intricate shapes. In the provided image, the CNN appears to be specifically identifying tire shapes.
Other computer vision systems also followed the idea of hierarchy and the importance of edge detection. Before CNNs came along, earlier systems used manually designed features, especially for finding edges. But what sets CNNs apart is their ability for discovering the simplest features, then building on them to identify progressively more complex ones.
Other Resources
While there are many similarities between neuroscience and machine learning, they don’t align perfectly. Nonetheless, it’s intriguing to observe how these principles can serve as potential sources of inspiration for improving machine learning.
This brief article can’t cover all the intersections between neuroscience and machine learning, but here are some extra resources to explore:
- Review article by Deepmind founder (pretty comprehensive) https://www.cell.com/neuron/pdf/S0896-6273(17)30509-3.pdf
- Pretty cool look at the possible future of neuroscience + AI: https://www.nature.com/articles/s41467-023-37180-x
- Video of this neuroscience/ai/hardware PURDUE PROFESSOR!!!! https://youtu.be/wRhEPy-0i8k?si=uEK1mRyFBoFAmYNp
- TEDx Talk https://www.youtube.com/watch?v=97iYdJE9mQ4
