The future of AI will not be AI, but NI (part III: natural thinking)

AI’s applications have made spectacular success stories recently. Almost everyone use at least one, if not a lot of, AI applications in daily life. Even though AI is still based on Neumann-architect computer as talked in the 1st part, its return is still higher than it’s computational cost. It is the ability of AI systems to mimic human’s senses, in particular the vision and hearing, that made AI so fantastic and people almost ignore its computation inefficiency:

Image: Machine can “see-recognize-react” and “hear-understand-reply”

However, as long as AI becomes more complex, one would see the limit of the current AI workflow and look for an alternative. But where should we change? Let’s look at where AI deviate from a biological system:

Image: AI start with NN architect, then Deep Learning. But the information propagation is not the same

This capability of sensing was only made possible when the ML model try to replicate somehow the way our brain process information: modify the Artificial Neural Network (ANN- an old Machine Learning model) to adapt to the sensing problem. For example, by using multi-layer, the ANN can process data sequentially with increasing complexity (or Deep ANN (DNN)). Also, the convolution technique integrate the intensity of neighborhood points in an image, while traditional ANN model treat image’s pixels independently. However, at the implementation level, these are only marginal similarity between brain-like computation and DNN.One of the most important differences is the way the information is propagating between neurons. In the brain, information is transferred by short potentials peak, also known as spike (or a train of spikes).

It is this observation that suggests a new framework, the spiking neural networks (SNNs). Surprisingly, the spikes are found to be uniform, of 100mV in amplitude and 1msec width [2]. This is radically contrasted to the ANN/ DNN where the potential has a continuous float-point value that contains information. In a SNNs: information is contained in the time direction.

Image: SNNs transfer information by time

Furthermore, in SNNs, because spikes (or trains of spikes) are data-coded, the architect of the network is almost the same for different problems. In contrast, in DNN, one has to design the appropriate network for each recognition problem, like the CNN, RNN, LSTM,… There are two immediate opportunities from this fact: firstly the SNN structure will more universal. As we can see, in our brain, different regions do different sensing work, but if one of the sensors fails, the corresponding brain part can be used for others sensing. Secondly, as spikes are event-encoded, spikes are generated only when data changes, the spike frequency is data-dependent and sparser. That reduces significantly the number of spikes generated (make spike train sparse) and optimize energy consumption.

Image: SNN is more universal than current DNN architects (CNN, RNN, LSTM,…). The brain can be used for many sensing purposes. Brain can put into stand-by mode when there is no change

As a consequence, the SNN will be very relevant for temporal AI applications such as gesture detection, object targeting, self-driving car,…

Image: people surveil, people image recognition

SNN architecture: in the simplest form, a SNN consists of spiking neurons and interconnecting synapses. Spiking neurons can be modeled with a simple threshold dynamics: they generate a spike when its membrane potential crosses a threshold. The dynamic of synapses are more complicated. Neuroscientist have discovered that their weight can be modified depending on the relative time difference between the spike’s fire of its pre- and post-synaptic neurons. If the pre-synaptic neuron fire a before the post-synaptic neuron, the weight of their connecting synapse is strengthened, and vice versa. This rule allows synapses to “learn” through the flow of spikes going through and is called the spike-timing-dependent plasticity (STDP), while the threshold firing rule of a neuron is called the Integrated and Fire (I&F) effect. I&F and STDP are the two most important features of a SNN. Both of them are biologically inspired, and therefore, are expected to make the SNN closer to achieving natural intelligence.

Image: I&F and STDP are the key features of a SNN

Besides the biological inspired effects of I&F and STDP, one also needs learning rules for SNN for both unsupervised- and supervised-learning. Recent works have shown progress on this subject. However, the hardware implementation is more difficult. At a high level, one can use Neumann machine and CMOS technology to simulate SNN, for example:

1. SpiNNaker (the University of Manchester) use ARM9 cores to simulate neurons, which communicate with each other by spikes, in an event-driven protocol. The largest system consists of 1 million cores/processors and demonstrates some fundamental concepts of the brain.
2. TrueNorth (IBM): neurosynaptic cores are integrated on-chip with local SRAM memory to store the synapse states, neuron states and parameters,… with some level of discretization. It implements I&F dynamic and asynchronously communication. The chip provides an energy efficiency of about three levels of magnitude higher than typical CPUs.
3. Loihi (Intel): use similar architect as TrueNorth with 14-nm fin field-effect transistor process. It supports 130,000 neurons and 130 million synapses, allows several forms of STDP learning rules.

Using CMOS technology, a large number of circuits (transistors, gates) are required to realize SNN features such as I&F, STDP, asynchronously communication. Recently, memristive devices have emerged as alternative nanotechnology to complement to the CMOS technology. It is novel two terminal devices resistor whose resistance can be changed by applying appropriate electrical pulses. The “memory” part of a memristive is achieved through various physical mechanisms such as phase transition, ionic drift, spintronic effects,… which can be used to exhibit an accumulative behavior (for I&F) or plasticity behavior (for STDP). This device becomes particularly suitable for SNNs implementation because it allows a single-device neuron and synapses (or with just one or two more resistance/capacitor) architect. A single memristor chip that contains, for example, 1 million memristors, which is the size of a typical CPU/GPU, can support up to 1 million neurons and synapses. Such performance is at least a thousand times higher than current technology, and is very convenient for heavy AI computation (training) or data centers to reduce the cost of cloud neural networks applications.

Furthermore, thanks to its two terminal concepts, memristors can be fabricated down to feature sizes below 10nm and switch with nanosecond timescales, thus allow for denser hardware. It makes them an ideal candidate for embedded devices subject to power constraints, e.g. mobile phones, mobile and aerial robots, internet of things (IoT) devices.

In conclusion, the future of AI will be based on novel technology other than the current Neumann machine, with different features:
- Large-scale, on-device computation and local memory
- No (or very little) binary digitization
- Computation is more natural: based on physical/biological effect.

Of which, nanoelectronic devices such as memristors become a potential candidate. The memristive memory, Resistive Random Access Memory (ReRAM), has been commercialized by HP, Fujitsu Semiconductor, Hynix Semiconductor,… AI memristive chips are developed in several labs including IBM, the University of Massachusetts,… with accuracy approaching the current DNN architect. Last but not least, AI chip using non-electronic technology are being developed recently. Lightelligence, Luminous and Lightmatter use light to carry information, and computes by optical devices such as interferometer, attenuator, … AnotherBrain even try to use biological matter for AI. Those start-ups are funded by well-known investors including Bill Gates và Travis Kalanick (Luminous).

Example: Imagin a black-box containing of mirrors, optical fiber, attenuators, interferometer, splitter, …) that can detect dog or cat: if a dog goes through, the light comes out at the “dog-hole” is more intense. In contrast, if a cat goes through, that at the “cat-hole” will be more intense.

References:

1. Tavanaei, A., Ghodrati, M., Kheradpisheh, S. R., Masquelier, T., and Maida, A. (2018). Deep learning in spiking neural networks. Neural Networks
2. W. Gerstner, W. M. Kistler, R. Naud, and L. Paninski, Neuronal dynamics: From single neurons to networks and models of cognition. Cambridge University Press, 2014.
3. Wang, Z., Joshi, S., Savel’ev, S. et al. Fully memristive neural networks for pattern classification with unsupervised learning. Nat Electron 1, 137–145 (2018) doi:10.1038/s41928–018–0023–2
4. Bipin Rajendran, Abu Sebastian, Michael Schmuker, Narayan Srinivasa, Evangelos Eleftheriou, Low-Power Neuromorphic Hardware for Signal Processing Applications, Special Issue on Learning Algorithms and Signal Processing for Brain-Inspired Computing in the IEEE Signal Processing Magazine
5. Pfeiffer M and Pfeil T (2018) Deep Learning With Spiking Neurons: Opportunities and Challenges. Front. Neurosci. 12:774. doi: 10.3389/fnins.2018.00774
6. Camuñas-Mesa, L.A.; Linares-Barranco, B.; Serrano-Gotarredona, T. Neuromorphic Spiking Neural Networks and Their Memristor-CMOS Hardware Implementations. Materials 2019, 12, 2745.