What Makes Spiking Neural Network Tick?

This is the second part (find part 1 here: What is Spiking Neural Network?. This article is the first in a series… | by NeuroCortex.AI | Oct, 2023 | Medium ) of the five-part series on spiking neural networks. Here we discuss the mathematical workings of spiking neural networks and the various models used to describe its operations.

Artistic visualization of neurons and their activity in the brain

The neural networks based on their computational unit can be distinguished into three generations:

1. First Generation: This generation of neural networks included the Perceptron, Hopfield networks, Boltzmann machines etc.
2. Second Generation: This generation of neural networks included multi-layer neural nets, recurrent networks etc.
3. Third Generation: This generation of neural networks includes the Spiking Neural Networks (SNNs)which is a more realistic representation of the working and information flow within the brain.

How does a Spiking Neural Network work?

Transmission of neuroreceptors between two biological neurons

Illustration of Optical Microcavity as pulsating neuron

The key difference between a traditional ANN and SNN is the information propagation approach. SNN tries to more closely mimic a biological neural network. This is why instead of working with continuously changing in time values used in ANN, SNN operates with discrete events that occur at certain points of time. SNN receives a series of spikes as input and produces a series of spikes as the output (a series of spikes is usually referred to as spike trains).

The general idea is as follows:

1. At every moment of time each neuron has some value that is analogous to the electrical potential of biological neurons;
2. The value in a neuron can change based on the mathematical model of a neuron, for example, if a neuron receives a spike from the upstream neuron, the value might increase or decrease;
3. If the value in a neuron exceeds some threshold, the neuron will send a single impulse to each downstream neuron connected to the initial one;
4. After this, the value of the neuron will instantly drop below its average. Thus, the neuron will experience the analog of a biological neuron’s refractory period. Over time the value of the neuron will smoothly return to its average.

SNN models

SNN neurons are actually built on the mathematical descriptions of biological neurons. There are two basic groups of methods used to model an SNN neuron.

Conductance-based models describe how action potentials in neurons are initiated and propagated

1. Hodgkin-Huxley model
2. FitzHugh–Nagumo model
3. Morris–Lecar model
4. Hindmarsh–Rose model
5. Izhikevich model
6. Cable theory

Threshold models generate an impulse at a certain threshold

1. Perfect Integrate-and-fire
2. Leaky Integrate-and-fire
3. Adaptive Integrate-and-fire

Although all these methods try to describe biological neurons, so SNN neurons built based on these models might slightly differ.

A comparison between different SNN models and their computational costs

Here we will explain three most widely used SNN models from the above two categories:

1. Hodgkin-Huxley model

The Hodgkin-Huxley model is a mathematical model that describes the generation and propagation of action potentials in neurons. It was developed by Sir Alan Hodgkin and Sir Andrew Huxley in 1952 and represents a significant advancement in our understanding of the electrical activity of nerve cells.

The model specifically focuses on the squid giant axon, a long and relatively large nerve fiber, making it suitable for experimental studies. Hodgkin and Huxley conducted a series of experiments on the squid giant axon and used the data to develop a set of differential equations that accurately described the ionic currents and membrane potential changes observed during an action potential.

The key components of the Hodgkin-Huxley model include:

1. Membrane Capacitance (Cₘ): Represents the ability of the neuronal membrane to store charge.
2. Sodium Conductance (gₙₐ): Describes the permeability of the membrane to sodium ions.
3. Potassium Conductance (gₖ): Describes the permeability of the membrane to potassium ions.
4. Leak Conductance (gₗ): Represents a small, constant conductance for other ions.

The model’s equations describe how conductance changes over time in response to changes in membrane potential, as well as the influence of ion concentrations inside and outside the cell. The Hodgkin-Huxley model provides insights into the mechanisms underlying the initiation and propagation of action potentials in neurons.

Illustration of mechanism of ion transfer inside soma of a neuron. One of the main observation is that ions of chlorine the charge cancels out but for potassium there is slightly more pressure to exit

An equivalent circuit diagram approximating the processes going on in the dendrites based on Hodgkin-Huxley

The Hodgkin-Huxley model is described by a set of four ordinary differential equations that govern the dynamics of the membrane potential and the conductance of sodium (Na⁺), potassium (K⁺), and leakage (L) currents. The model was developed based on experimental data from the squid giant axon.

The membrane equation is as follows

Membrane equation as defined in Hodgkin-Huxley Model

Equation for the sodium activation variable (m)

Equation for the sodium inactivation variable (h)

Equation for the potassium activation variable (n)

Here:

• V is the membrane potential.
• C​ₘ is the membrane capacitance.
• Iₑₓₜ is the external current
• gˉ​Na​, gˉ​K​, and gˉ​L​ are the maximum conductance for sodium, potassium, and leakage currents, respectively.
• m, h, and n are dimensionless gating variables representing the activation of sodium channels, inactivation of sodium channels, and activation of potassium channels, respectively.
• ENa​, EK​, and EL​ are the reversal potentials for sodium, potassium, and leakage currents, respectively.
• αₘ,βₘ​,αₕ​,βₕ​,αₙ​,βₙ​ are voltage-dependent rate constants that control the opening and closing of ion channels.

Gating eqn in the Hodgkin huxley models

These equations describe the dynamics of the membrane potential and the opening and closing of voltage-gated ion channels, providing a quantitative understanding of the generation and propagation of action potentials in excitable cells.

These equations collectively describe how the neuron’s membrane potential changes in response to external stimuli and how this change leads to the generation of an action potential. The Hodgkin-Huxley model is highly influential in neuroscience and continues to be a fundamental model for understanding neuronal behavior.

One of the key findings of the Hodgkin-Huxley model is the concept of voltage-gated ion channels. According to the model, the opening and closing of these channels are voltage-dependent, allowing the neuron to regulate ion flow based on changes in membrane potential. This model laid the groundwork for our understanding of the basic principles of electrical signaling in neurons, and it has been influential in the field of neuroscience.

While the original Hodgkin-Huxley model was developed for the squid giant axon, variations of the model have been adapted to describe the electrical activity in different types of neurons and other excitable cells. The model has been instrumental in shaping our understanding of the biophysics of neuronal excitability and remains a fundamental concept in neuroscience.

2. Izhikevich model

The Izhikevich model is a simplified mathematical model of spiking neurons developed by Eugene M. Izhikevich in 2003. It provides a computationally efficient representation of the dynamics of neuronal spiking behavior while maintaining a level of complexity that captures key features of biological neurons. The Izhikevich model is particularly attractive for simulations and computational studies due to its simplicity and ability to replicate a variety of firing patterns observed in real neurons.

The model is expressed by two coupled differential equations, one for the membrane potential (v) and the other for a recovery variable (u). The equations are as follows:

Membrane potential equation

When membrane potential reaches a threshold the spikes are produced and variables are updated using the reset condition as shown

Here:

• v is the membrane potential.
• u is a recovery variable.
• I is the input current.
• a,b,c, and d are parameters that control the dynamics of the neuron.

The spiking model and the formation of spikes

The Izhikevich model is versatile and can replicate a range of spiking patterns observed in real neurons, such as regular spiking, fast spiking, bursting, and chattering. The model’s flexibility makes it useful for studying the computational properties of different types of neurons and neural networks.

One of the strengths of the Izhikevich model is its simplicity, which allows for efficient numerical simulations and analysis. Researchers and computational neuroscientists often use this model to explore neural network behavior, synaptic plasticity, and other aspects of neural computation. Additionally, it has been deployed in the development of neuromorphic hardware, which aims to mimic the computational principles of the brain in electronic circuits.

3. Leaky Integrate-and-fire

The Leaky Integrate-and-Fire (LIF) model is a simple mathematical model used to describe the behavior of a neuron in response to input stimuli. It is widely used in computational neuroscience for its simplicity and computational efficiency. The model captures the basic principles of neuronal integration and firing, although it is a simplification of the biophysical complexity found in real neurons.

Schematic diagram of Integrate and fire model

The key idea behind the LIF model is that the neuron’s membrane potential integrates incoming synaptic currents over time. When the membrane potential reaches a certain threshold, the neuron fires, generating an action potential (spike). Additionally, the model incorporates a leak term, representing the passive leak of ions through the cell membrane, which causes the membrane potential to decay toward a resting level in the absence of input.

Schematic diagram of Leaky integrate and fire model

The basic equation for the LIF model is given by:

Here:

• V is the membrane potential.
• τₘ is the membrane time constant, representing the time it takes for the membrane potential to reach approximately 63.2% of its final value in response to a step input.
• I(t) is the input current.
• Vᵣₑₛₜ​ is the resting membrane potential.
• Cₘ is the membrane capacitance.

When the membrane potential v reaches a predefined threshold value Vₜₕᵣₑₛₕ​, the neuron fires a spike, and the membrane potential is reset to a resting value Vᵣₑₛₑₜ. The reset mechanism is described by:

The LIF model provides a computationally efficient way to simulate the firing behavior of neurons and is often used in large-scale neural network simulations. While it lacks the detailed biophysical realism of more complex neuron models like the Hodgkin-Huxley model, the LIF model is valuable for studying network-level properties and emergent behaviors in neural systems. It serves as a building block for more sophisticated models and is a useful tool for understanding the basic principles of neural information processing.

Neuron spike response based on Izhikevich Model

Neuron spike response based on leaky Integrate and fire model

Summarizing Leaky Integrate-and-fire, Izhikevich model, Hodgkin-Huxley model

1. Leaky Integrate-and-Fire (LIF) Model:

• Concept: A simple mathematical model that describes the behavior of a neuron as it integrates incoming currents over time.

Key Features:

• Membrane potential (v) integrates input until it reaches a threshold.
• Upon reaching the threshold, a spike is generated, and the membrane potential is reset to a resting value.
• Includes a leak term representing the passive decay of the membrane potential in the absence of input.

2. Izhikevich Model:

• Concept: A neuron model designed for computational efficiency, capable of replicating a variety of spiking patterns observed in real neurons.

Key Features:

• Flexible and able to mimic different firing patterns, such as regular spiking, fast spiking, bursting, etc.
• Uses a simplified form compared to the Hodgkin-Huxley model, making it computationally efficient.
• Particularly useful for simulations and computational studies of neural networks.

3. Hodgkin-Huxley Model:

• Concept: A detailed biophysical model describing the generation and propagation of action potentials in neurons.

Key Features:

• Based on experiments with the squid giant axon.
• Includes voltage-gated ion channels for sodium and potassium, as well as a leakage current.
• Provides a detailed representation of the ionic currents and their role in action potential generation.
• Fundamental for understanding the biophysics of neuronal excitability and widely used in neuroscience.

In conclusion, the mathematics of spiking neural networks involves a combination of differential equations, algebraic equations, and learning rules to capture the spiking behavior of individual neurons and their interactions within a network. These mathematical models help researchers and engineers understand the computational properties of SNNs and their potential applications in areas such as neuromorphic computing, pattern recognition, and cognitive systems.