Intelligence as a “Learning Formula”
The technological evolution of artificial intelligence is attracting attention.
Artificial intelligence is realized through programs, but the fundamental difference from conventional information processing technology is the approach of machine learning.
A program can be seen as a collection of formulas.
In conventional programs, human engineers design the rules of information processing. In other words, they design all the formulas within the program.
On the other hand, in programs that use machine learning, human engineers do not design the rules for information processing. Instead, through a process called training or learning, the program itself changes the rules of information processing, gradually approaching the optimal rules.
This means that the program modifies some of the internal formulas. Machine learning is about guiding these changes toward desirable information processing rules.
Therefore, this means there are formulas that learn. In this article, we will explore intelligence as a “learning formula.”
The World of Evolving Formulas
In the previous article, I explained that the natural world is a world of increasingly complex formulas, and within it is a world of evolving formulas.
To briefly recap, while physical laws can be expressed as simple, immutable formulas, we discussed how these laws generate complex and ordered macro phenomena.
In cases where elements interact, such as in the three-body problem, it has been proven that while the system can be described with unchanging formulas, it cannot be analytically solved.
Therefore, when considering solving such systems analytically, the equations are represented as algorithms that iterate over time, such as loop(x=f(x,y), y=f(y,x), Δt).
If we consider the expanded formula of this loop as the actual formula, it becomes clear that the formula is not static but becomes more complex over time. This explains why complexity appears in the natural world.
Furthermore, when applying this loop formula to numerous elements, even though predicting the result may be difficult, the outcomes tend to converge into certain statistical distributions, and periodic patterns longer than Δt may emerge. This explains why macro-level order appears in relation to the number of elements and the passage of time.
Neural Networks: Learning Formulas
A representative method of machine learning is the neural network.
When considering neural networks as formulas, there are similarities and differences compared to physical laws expressed as loop(x=f(x,y), y=f(y,x), Δt).
The similarities lie in the fact that they are both composed of repeated applications of simple formulas, and the input-output relationships of each simple formula involve an N-to-M relation.
The intermediate values in the iterations are calculated based on many preceding intermediate values as inputs. Furthermore, other intermediate values at the same computational stage share the same inputs.
The differences are that a simple neural network does not iterate indefinitely over time Δt, and each simple formula can have unique parameters for each intermediate value.
These unique parameters are the key to achieving formulas that change through the program. Even though the form of the formula remains constant, adjusting the parameters can bring the system closer to desired information processing rules.
Iterations Within a Neural Network
Let’s consider a basic neural network in more detail.
A basic neural network is structured such that within each unit called a layer, there are multiple elements called nodes. A neural network also contains multiple layers.
The number of nodes can vary by layer, but for simplicity, let’s assume the number of nodes per layer is constant. If the number of nodes per layer is N and the number of layers is L, the total number of nodes in the neural network would be N×L.
The value of each node is determined by a formula that takes all the nodes from the previous layer as inputs. Therefore, each node’s value is calculated through a simple iteration based on the number of nodes in the previous layer. This must be done for all nodes in the current layer, meaning we also have iterative calculations for each node.
Thus, to calculate all the nodes in a single layer, a simple formula must be repeated N×N times.
To perform this calculation across all layers, the basic neural network involves N×N×L iterations of a simple formula. Consequently, the number of parameters is also roughly proportional to this number of formulas.
Iteration of the Entire Neural Network
Furthermore, in recurrent neural networks and in large language models, which form the basis of chat AI, the entire basic neural network is iterated.
In the iterations within a neural network, each iteration has different parameters, and changing them allows for machine learning. However, in iterations of the entire neural network, the number of parameters does not increase, because the same neural network with the same parameters is being iterated.
If the iteration is repeated R times, the number of iterations of the simple formula would be N×N×L×R, but the number of parameters remains proportional to N×N×L.
In the iteration of the entire neural network, the iteration count is essentially unlimited. This is similar to the formulas that become more complex or evolve through repetition in the natural world. Thus, the iteration of the entire neural network also contains the properties of evolving and complexifying formulas.
The difference from the natural world is that the simple formulas are accompanied by many parameters, each being different, and these formulas are learning formulas.
Thus, an iterating neural network has the properties of complexifying, evolving, and learning formulas.
Learning the Natural World with Neural Networks
This interpretation provides a simple explanation of why neural networks can learn about the natural world by understanding it as a set of evolving and complexifying formulas.
Since neural networks themselves possess the nature of evolving and complexifying formulas, they share the same properties as the formulas of the natural world. On top of that, while the basic formulas of the natural world are immutable, the basic formulas of neural networks can be learned and adapted. In other words, they can be made to fit the formulas of the natural world.
Furthermore, as the natural world evolves, it not only follows basic formulas but also gives rise to macro laws. These macro laws can be expressed as other formulas or patterns, and the evolving and learning nature of neural networks can also be adapted to fit these macro laws.
Of course, due to limitations in the size of the parameters that a neural network can learn, it cannot fully fit all the complexity of the natural world. However, as the size of the parameters increases, this theoretically means that the neural network can fit the complexity of the natural world proportionally.
The Size of Neural Networks
This explanation accurately reflects the history of neural networks.
Neural networks have long been known in the field of artificial intelligence research. And the fundamental idea behind neural networks has not significantly changed from then to now.
Nevertheless, at a certain point, deep learning emerged, demonstrating that AI could recognize images, and more recently, large language models have shown the ability to handle natural language in communication and thinking, much like humans. Despite the unchanged basic principles, AI’s abilities have improved dramatically over time. The reason for this improvement is the increase in computational power. There are two components to computational power: the ability to calculate formulas quickly and cost-effectively and the ability to store large amounts of data, both of which have significantly improved over time.
This means an increase in the number of iterations of formulas within neural networks, N×N×L×R, and the ability to retain a number of parameters proportional to N×N×L. In other words, the dramatic increase in computational power directly translates to improvements in AI’s ability to learn from the world.
Behind this fact lies the principle that the nature of neural networks as complexifying, evolving, and learning formulas can grasp the complexifying and evolving formulas of the natural world. This principle explains why the increasing computational power of computers has been the main driving force behind the growing range of achievements realized through neural networks.
Similarly, the increase in the size of the human brain can be explained in the same way.
Within the human brain, neural networks are at work. While other mechanisms besides neural networks may also exist in the brain, it has long been thought that the brain’s intellectual abilities could not be entirely reduced to neural networks. However, as the scope of what neural networks can accomplish has expanded, it is now believed that the range of intellectual abilities realized by neural networks within the human brain is broader than previously assumed.
The fact that brain size has increased over the course of biological evolution can be explained by the same logic. The increase in brain size due to evolution can be seen as an increase in the size of neural networks, enhancing the ability to fit the complexifying and evolving formulas of the natural world through complexifying, evolving, and learning formulas.
In Conclusion
By understanding a basic neural network as a “learning formula” and adding the properties of complexity and evolution through iteration, it becomes possible to realize an intelligence that can learn about the natural world, which also has these properties of complexity and evolution.
This understanding allows us to view both the natural world and intelligence through the common element of formulas. With statistical distributions and periodic patterns emerging from the repetition of formulas and the mechanism of parameter fitting, we can begin to approach phenomena such as emergence and self-organization from a mathematical perspective.
While this article does not yet provide a precise theoretical explanation of these mechanisms, the possibility of modeling emergent and self-organizing phenomena from a mathematical standpoint has begun to emerge.
This indicates a shift in perspective from the view that scientific phenomena are fully understood once expressed by immutable formulas, to a standpoint that requires understanding how analytically calculable formulas unfold over time. When it comes to intelligence, an additional consideration of parameter adjustment in relation to these unfolding formulas must be included.
Furthermore, exploring the theoretical models of iteratively unfolding formulas could open up new realms in mathematics. This means that even if a formula appears to have the same properties when viewed as a simple analytical formula, its properties may change when expanded into a calculable formula.
