The future of AI will not be AI, but NI (part I: the ALU bottleneck)

Because the current AI architect is really artificial and extremely inefficient, compared to natural (biological) systems like our brain. This is attributed to three main drawbacks: 1) the serial computation (the Neumann-architect computer), 2) the binary digitization and 3) the synchronizing temporal coding schema.

Firstly, almost all AI systems are based on Neumann-architect machines, like our PC or Mac computers. Although our computers do great jobs and satisfy most of our computational needs, its architect has a great limit because computation is done only in the Arithmetic Logic Unit (ALU), separated from the memory. We do not sense this limit in most of our works because this ALU runs extremely fast, a few billion switches per second. It can compress image, edit image’s shape, show it on screen, play music, connect to the Internet,… in a serial way and yet you still feel like it does all simultaneously.

Figure 1: Thousands of apps are done serially in our computers, either through a single or a few cores (2, 4, 8)

But AI work does not have this luxury because it has a huge work-load to do. For example, a network of 10 million weights (a normal one) with 1 million observation (images), results in 10,000,000,000,000 (ten thousand billions) objects to be computed. More seriously, each object needs to be computed many times in order to be in good shape. Imagine ten thousand billion patients line in several queues of doctors (ALU/cores), each patient must pass thousands of times to completely treated:

Figure 2: Network’s weights are like a patient who needs to be treated many thousand times. There are billions of patients but only a few doctors (core, either CPU or GPU)

In consequence, AI training jobs run days and nights, and scientists have to compromise their architect (thus performance) for computing time. Neumann architect appears to be a bottle-neck in AI time.

Future AI, therefore, does need a non-Neumann architect where computation can take place in many more places, up to the scale of the same number of components in the systems. For example, if a processor has 1 billion transistors, there must be 1 billion computing units or at least a few hundreds of millions ones. Ideally, it should be the transistors themselves that do the computation. Such an architect will be more natural, as in our brain, which not only merges computation and memory but also software (the network) and hardware (the transistors for example):

Figure 3: Comparision between Neumann and non-Neumann systems, where computation can take place in almost all components, results in a much faster action, such as pattern recognition, language translation,…

A non-Neumann system must be designed closer to the material/physical level, and therefore, make AI more NI (natural intelligence).

Part II: Beyond Digitalization

References:
1. Boybat, I., Le Gallo, M., Nandakumar, S.R. et al. Neuromorphic computing with multi-memristive synapses. Nat Commun 9, 2514 (2018) doi:10.1038/s41467–018–04933-y
2. Camuñas-Mesa, L.A.; Linares-Barranco, B.; Serrano-Gotarredona, T. Neuromorphic Spiking Neural Networks and Their Memristor-CMOS Hardware Implementations. Materials 2019, 12, 2745.
3. Pfeiffer M and Pfeil T (2018) Deep Learning With Spiking Neurons: Opportunities and Challenges. Front. Neurosci. 12:774. doi: 10.3389/fnins.2018.00774