An Introduction to Spiking Neural Networks (Part 1)
Recently, Elon Musk owned neurotech startup Neuralink announced its ambitious plans to enhance the human brain’s computational capabilities by implanting minuscule, flexible robotic electrodes onto the surface of the brain. These nanomachines would then effectively be able to function as a part of your brain, making us superhuman cyborgs if all goes according to plan!
That brings us to the question, how would these nanomachines be able to process the signals in our brain and further contribute additional signals to enhance the brain’s capabilities? In order to understand this, let’s first take a look at how the neurons in the brain are wired and how information is represented and transmitted by them.
We will also then see how the biological neural networks in our brain compared to the artificial neural networks(ANNs) that have led to the emergence of deep learning. Finally, we will explore whether modeling neural networks by using more biologically realistic neuron models and learning rules could be the next big step in the world of deep learning.
I am writing this article mainly for readers who are fascinated by the seemingly magical powers of deep learning and have little to no background in biology, so pardon me if some of the next information seems very elementary to some of you!
Understanding how biological neurons create and transmit information
Although there are hundreds of different types of neurons in the brain, the above diagram which can be found in most introductory textbooks can be considered a good functional representation of a neuron.
The biochemical conditions in our brain are such that concentrations of ions are unequal inside and outside the membrane of a neuron which leads to the development of a potential difference. There is a higher concentration of positively charged ions outside the cell membrane, which means that there is a negative potential difference between the inside and the outside.
Neurons communicate with each other through voltage spikes! That is, from an engineer’s point of view, a neuron is nothing but a battery with a switch which only can be closed for the tiniest instant, thus producing a voltage spike and then becoming open again!
Biologists will complain that this is a very crude way to describe neuron function, but it works. Although there are beautiful biochemical mechanisms at play behind all of this, essentially a biological neuron is a battery with a switch that stays closed only momentarily. If you are interested in understanding how the sodium-potassium pump works and how voltage spikes are generated, I would recommend watching Khan Academy's video series on this topic:
Khan Academy : Advanced nervous system physiology
You might be thinking,”Wait, that sounds easy enough but when does the switch close?” In order to answer that allow me to introduce you to the concept of ‘receptive field’ of a neuron.
Receptive field
Let us consider the neurons in your eyes (known as photoreceptors) to understand this concept. The photoreceptors are connected to the visual cortex(the part of your brain that processes visual input) through the optic nerve and some other neurons in the eye.
Now read this carefully! Each photorecepetor’s ‘switch’ closes when light is incident upon it in a particular manner.The exact way in which light has to be incident on a photoreceptor to close its switch is different for each photoreceptor.
For example, one photoreceptor may respond to red light falling on it at 60 degrees. Another photorecpetor may be receptive to the intensity of light. Its switch may close when light below a certain intensity level falls on it at any angle. The particular way in which light needs to fall upon a photoreceptor is called its ‘receptive field’ and we say that it fires an ‘action potential’ (fancy biologists’ way of saying spike of high voltage). If the manner in which light is incident on a neuron does not align with its receptive field, the neuron will not be as ‘activated’ and thus its switch is much less likely to close as compared to the case in which the light does align with its receptive field.
Each neuron in the nervous system has a receptive field! It is easy for us to figure out what the receptive field is for neurons that directly receive external input from the surroundings of the organism. However, as we go deeper inside the nervous system, the receptive fields of neurons become more complex. For example, there may be a neuron in your brain that fires an action potential when you see a picture of Cristiano Ronaldo or one that fires when you hear high pitched screaming as you are walking through the woods!
But how does that neuron deep inside your brain come to know that you are looking at a picture of Cristiano Ronaldo? Drawing a parallel to ConvNets, our neuron here has learned to detect a feature. But what learning rule is at play here? Does our brain actually carry out backpropagation?! Before I answer that, let me explain to you how action potentials are transmitted to other neurons.
When a neuron fires an action potential, the voltage spike travels along the axon of the neuron (which essentially acts like a conducting wire, for biochemical mechanism refer to Khanacademy). When the action potential reaches an axon terminal(the end part of an axon), it causes the release of neurotransmitters.
What are neurotransmitters?
Neurotransmitters are chemicals which are stored in bags called synaptic vesicles in the axon terminals. These bags are burst when an action potential reaches the terminal. The chemicals travel through the region between the axon terminal of the initial neuron (presynaptic neuron) and the next neuron’s dendrites (postsynaptic neuron). The connection between two neurons is called a synapse. The neurotransmitters attach to receptors present on the dendrites of the postsynaptic neuron. The effect of the neurotransmitters attaching to receptors depends on the type of neurotransmitter. Let’s see what the types are.
Types of neurotransmitters
Excitatory and inhibitory are the two types of neurotransmitters. When excitatory neurotransmitters attach onto receptors, they cause a net flow of positive ions into the neuron, which is called depolarization. The openings in the neuron’s cell membrane through which the positive ions flow are called ion channels.
Initially the voltage changes occuring due to opening of ion channels are linear and additive in nature. However once enough positive ions have come inside and the potential difference has reached a value called ‘threshold potential’ there is a sudden mass entry of positive ions which results in an action potential (ie. closing of the switch). After an action potential has occurred, the neuron returns to its initial stable state over a period of time.
On the other hand, if an inhibitory neurotransmitter that attaches to the receptors , it causes opening of ion channels that allow negatively charged ions to flow in or let positively charged ions flow out. Due to this, the potential difference between the exterior and interior becomes more negative and this process is called hyperpolarization.
How different types of neurotransmitters together determine neuron function
Almost all neurons in the brain have thousands of synapses(connections to other neurons). Neurons themselves can be classified as excitatory and inhibitory. Excitatory neurons contain only excitatory neurotransmitters in their synaptic vesicles whereas inhibitory neurons contain only inhibitory neurotransmitters. It is reasonable to assume that the average neuron in the brain is connected to tens or maybe hundreds of neurons of each type. As a result, at any point of time, a neuron in an active organism would be receiving inputs at multiple synapses. The net effect of this is that inhibitory and excitatory presynaptic neurons would compete to influence the activity of our postsynaptic neuron. Sometimes, the excitatory neurons win and the postsynaptic neuron fires a spike. Whereas sometimes the inhibitory neurons are able to shut it up for quite a long time!
Now here’s what makes it even more interesting. If the neuron that fired a spike is inhibitory, it will try to suppress the other neurons it is connected to. This even though its presynaptic excitatory neurons won the battle and managed to get it to spike, they might lose the war because our current neuron may suppress other excitatory neurons from firing and slow down the flow of information!
Strength of connections between neurons
The strength of a connection between two neurons decides to what extent the presynaptic neuron is able to influence the spiking behavior of the postsynaptic neuron. The strength depends upon the amount of neurotransmitter the presynaptic neuron releases and how receptive the postsynaptic neuron is to it. The strength of a connection between two neurons is not fixed. I will discuss how the magnitudes of connections are changed in the next part of this series :).
What happens after an action potential is fired?
Earlier in this post I said that after a neuron fires an action potential, it returns to its stable state. However, this is not an instantaneous process. The time period after the end on an action potential is called refractory period.
During the refractory period, it is difficult or in some cases impossible to trigger another action potential in the neuron no matter how strong the excitatory inputs to it are. This happens because the sodium ion channels which play a crucial role in the depolarization phase that causes the action potential are in an inactivated state and do not open at any membrane potential.
This is how it is ensured that a neuron receiving constant excitatory input does not go on a crazy firing spree and end up producing a lot more information for other neurons to process than necessary.
Computational models of neurons
How can we make computer models of biological neurons? Note that all information transmission related processes in biological neurons occur due to change in concentrations of ions. Could we model these biochemical processes using differential equations?
In 1952, two scientists Hodgkin and Huxley attempted to do so and ended up winning a Noble prize for their efforts! They modeled the cell membrane of and do not open at any membrane potential.the neuron as an electric circuit, similar to the one shown below:
They considered only the two most common ions involved in the generation of action potentials: Sodium(Na+) and Potassium(K+). They also incorporated a ‘leak current’ into their model, which basically accounts for the cell membrane not being completely impermeable to ions at all times. Using Ohm’s Law(V = IR) they wrote down the following circuit equations:
We know that the current flowing through the circuit is the sum of these three currents and the current through the capacitor. Using the current-voltage relation for a capacitor we get:
V is the displacement from equilibrium potential across the capacitor. Thus we can say that:
where Ek is the equilibrium potential and E is the current potential inside the membrane.
gNa, gK and gL are the conductances of the resistors. gL is a fixed constant value whereas gNa and gK are dependent on the number of ion channels that are currently open.
Hodgkin and Huxley introduced parameters n, m and h to represent what fraction of ion channels of the corresponding ion are open. Their values lie between 0 and 1.These relations were determined based on experimental behavior of neurons.
gKbar and gNabar are constant values. The parameters n, m and h however are not constant and vary according to the following differential equations:
alpha is the rate constant of opening and beta is the rate constant of closing of the ion channels
The Hodgkin Huxley model was a great achievement of 20th century biophysics. However, it was not suitable for large scale simulations of spiking neural networks on computers.
There are two ways the Hodgkin Huxley model can be improved/modified. Either you can include a higher level of detail which could involve adding more types of ion channels, factors that influence conductance, as well as taking the shape of the neuron into account because the location of the synapse on the neuron body influences spiking behavior too.
A more detailed model is likely to produce more realistic behavior of neurons. However, adding more parameters makes it more computationally expensive to simulate the neuron. As a result the simulations are likely to become very slow, which is undesirable for data scientists and engineers.
A model with fewer parameters is likely to be less biologically accurate, which means it will not be able to behave like a real neuron. However, in 2003 Eugene Izhikevich proposed a very elegant model that was able to reproduce most spiking patterns found in biological neurons.
Spiking Patterns
Until now, we have only focused on understanding how single spikes are generated and transmitted by neurons. But a single spike by itself cannot do much in a spiking neural network. A ‘spike train’ which is basically a two dimensional plot of time and membrane voltage with multiple ‘spikes’ indicating that the neuron fired at that particular point in time is able to hold much more information.
In the above image, various types of spiking patterns are shown.
You may notice a,b,c and d given below each firing pattern in the above image. These are parameters for Izhikevich model neurons. Let’s find out more about them.
Izhikevich Neurons
You can find his original paper by clicking on the above link.
By various techniques, he was able to reduce the equations of the Hodgkin Huxley model to the following two differential equations:
v’ = 0.04v² + 5v + 140 -u + I
u’ = a(bv — u)
with auxiliary after spike resetting,
if v ≥ 30mV then reset v = c and u = u + d.
Here, v and u are dimensionless variables, and a , b , c , and d are dimensionless parameters, and the derivative is with respect to time t.
v is the variable for membrane potential while u which is known as the membrane recovery variable corresponds to how active the sodium and potassium ion channels are.
Synaptic or injected dc currents are delivered through the variable I.
- The parameter a describes the time scale of the recovery variable
u . Smaller values result in slower recovery. A typical value is
a = 0 .02 . - The parameter b describes the sensitivity of the recovery variable
u to the subthreshold fluctuations(ie. below threshold potential) of the membrane potential v .
Greater values couple v and u more strongly resulting in possible
subthreshold oscillations and low-threshold spiking dynamics.A
typical value is b = 0.2 . - The parameter c describes the after-spike reset value of the membrane potential v. A typical value is c = -65 mV
- The parameter d describes after-spike reset of the recovery variable u.
A typical value is d = 2 .
I understand that the above might be a lot to process. The differential equations and parameters that govern neuron models have very complex behavior.
You can go into more detail if you have studied nonlinear systems of ordinary differential equations. Using numerical methods, it is possible to analyze the behavior of such neuron models in depth and get a good understanding about them.
However, you do not need to be an expert on how nonlinear systems of equations behave in order to start using these neurons to build spiking neural networks! Let us get started with building networks in the next article.Until then cheers!
For further discussion, you can contact me at chinmayvc@gmail.com.
