Towards a lifelike model of computing with an application to neural networks

What if we had a hypergraph-like model of computing but for neural networks instead of spacetime?

Ever since I learned about the Wolfram Physics project — especially as it relates to multicomputation — I have been wondering about how we could apply it to biology. Unlike our physically rigid, static, concrete, fixed, dead computers, Nature knows how to make living machines. What will it take to get there?

Interestingly enough, it is the scientific breakthrough that is ChatGPT — that human thinking in the form of language can be roughly mimicked by a 400-layer, 175B-connection artificial neural network with millions of neurons — especially as recounted in Wolfram’s book that made me put the pieces together: what if we took a model of a computing that worked like hypergraphs in the Wolfram Physics project, and applied it to neural networks? More specifically, I mean neural networks that can adapt like real brains. For the lack of a better name, let us tentatively call them adaptive neural networks as opposed to the inanimate neural networks of today. As Feynman might have said:

“Brains aren’t inanimate, dammit, and if you want to make a simulation of one, you’d better make it adaptive, and by golly it’s a wonderful problem, because it doesn’t look so easy.”

What must some of the properties of adaptive neural networks be? Let’s see:

• As Wolfram argues in his book, each neuron may be both a computing and memory unit (unlike our separation of CPUs, RAM, and GPUs).
• Each neuron must be able to make another neuron (using materials from its environment).
• Each neuron must eventually die. (Similar to dilution in inanimate neural networks, I think, when all the incoming and outgoing weights are reduced to zero.)
• Each neuron must be able to find and connect with other neurons.
• Each neuron is free to adjust its connections with its neighbours.
• The network can grow exponentially in the beginning, but it must eventually obey the S-curve.
• The network must be able to adapt to new inputs, and even its own outputs. (Dynamic neural networks may be close to what I have in mind.)
• All of this must be happening all at once.

Of course, just like with inanimate neural networks, the idea is to keep the abstraction without copying all the specific details from Nature. So, for example, we could cheat and simply pop another neuron into existence instead of trying to actually reproduce cell division.

Now, critics might say, what is the use of adaptive neural networks? Aren’t inanimate neural networks good enough? Why can’t we just fix the number of layers, neurons, weights, and so on, and go from there? Well, there are more than a few:

• No need for separation of logic from memory.
• Long-term memory. ChatGPT doesn’t permanently remember what it said before (unless you retrain it with some of its own sayings). There are some studies of memory in inanimate neural networks, such as LSTM and Hopfield networks, but I’m not sure whether there is any unification of memory with computing units.
• Surviving despite (minor) catastrophes (including regular churn), and continuting to function at least partially, like Phineas Gage. ChatGPT probably can’t survive a stroke right now.
• The rulial space of individual brains.

Thus, many of the same ideas of the Wolfram Physics project also apply to adaptive neural networks. For example, how do memories persist in our brains despite the Ship of Theseus problem? What would be the rulial distance of two different adaptive neural networks with different inputs and/or rules?

Of course, such a model of computing doesn’t exist yet, at least as far as I am aware of. It’s likely that some but not all of these features already exist in inanimate neural networks. But I bet it would be a massively interesting thing to study from at least a theoretical point of view. It is amenable to what I call the Wolfram modus operandi: take a very simple idea, and explore it to its logical extremes. For example, one simple experiment is to study the rulial space of even inanimate neural networks (subject to some constraints). In fact, it could be a nice Wolfram summer or winter school project. Hopefully, at the end of it, we get some answers for questions like how some brain structures arise.

Could adaptive neural networks explain how some brain structures arise, the same way the Wolfram physics project explains the rise of dimensions?

Postscript: This essay was actually written exactly around a year ago (June 1st 2023), and I held off publishing it so that I could find and acknowledge all previous technical work in a field I am not an expert in, although I think it’s safe to say that while something may fit a few properties I’m looking for, nothing has all of them yet. Meanwhile, that one-man-Bourbaki-committee called Stephen Wolfram has written yet another computational essay that may hold some clues...

Acknowledgements: Thanks to Sean McClure for his detailed review and comments. Any mistake or omission is mine.