Is our brain a Bayesian or Deterministic NN
Before reading ahead I am in no way claiming that our brain is a Neural Network. But I ASSUME that a Neural network is the simplest approximation of neuronal connections in our brain.
This blog post tries to answer the question, whether our brain learns point estimates or it learns a distribution for computation.
Synapses
The synapse is a functional connection between a neuron and another neuron. The synaptic gathers the information from multiple stimuli, combines it and transmits it to other neurons. This is basically what a single node in MLP does.
It was believed in the neuroscience community that the synapses learn point estimates for computation. But the paper Probabilistic Synapses by Aitchison et.al. proposed for the first time that in high — noise regime keeping track of probability distributions not point estimates is the optimal way of learning.
Probabilistic Synapses
The synapses keep track of two variables-Mean and variance thus learning a log-normal distribution for computation. The paper gives two main facts -
- Bayesian Plasticity
- Synaptic Sampling
Bayesian Plasticity
They prove that the synapses keep track of the distributions(mean and variance) and not just the mean value of weights. The second concept they used is how the learning rate of a neuron is directly proportional to its uncertainty. If the synapse is more uncertain about its prediction then the update in that synapse should be more than the update in the weight of a synapse which has lower uncertainty. Here alpha is the learning rate of i neuron and S is the standard deviation, the denominator is just a normalizing term.
Synaptic Sampling
This is the theory of how weights are sampled from distribution and how the synapses uncertainty should fall as the presynaptic firing rate increases (If the previous synapse is more certain about its prediction then the certainty of the current synapse increase)
Relation to Bayesian Neural Networks
The whole field of Bayesian Deep Learning argues the importance of learning uncertainties along with the weights. This paper just validates the neuroscience aspect of it.
The paper relating the learning rate to uncertainty was recently accepted in ICLR-Uncertainty-guided Continual Learning with Bayesian Neural Networks. They also relate how uncertainty is a measure of importance and perform continual learning in this scenario by regularizing weight with low uncertainty.
If you want to know more about the experiments performed in detail please refer to my slides or the paper (link above).