Deconstructing Randomness as Chaos and Entanglement in Disguise
Here’s a question the perhaps needs to be asked, but hasn’t been asked enough. What is randomness and where does it come from?
This is one scary place to venture in. We take for granted the randomness in our reality. We compensate for that randomness with probability theory. However, is randomness even real or is it just a figment of our lack of intelligence? That is, does what we describe as randomness just a substitute for our uncertainty about reality? Is randomness just a manifestation of something else?
What is the motivation for examining something seemingly as fundamental as the notion of randomness and ultimately probability? The motivation is that given Artificial General Intelligence (AGI) has been a problem that is unsolved in decades of attempts, that we must at a minimum perform the due diligence to see if there are any flaws in the tools that we employ. I’ve written about the flaws of Bayesian inference. In this piece I will explore in greater detail our understanding of randomness and ultimately probability.
A really illuminating paper about this comes from Andrei Khrennikov who writes “Introduction to foundations of probability and randomness” where he explores the different interpretations of randomness and explores in more detail Kolmogorov’s proposal on this subject. He summarizes in excruciating detail the three different interpretations of randomness:
Randomness as unpredictability.
Randomness as typicality:
Randomness as complexity.
Where Khrennikov comes with the observation that:
As we have seen, none of the three basic mathematical approaches to the notion of randomness (based on unpredictability, typicality, and algorithmic complexity) led to the consistent and commonly accepted theory of randomness.
Of the three interpretations of randomness, only Kolmogorov’s interpretation is of an objective nature. This is indeed surprising in that Kolmogorov’s viewpoint isn’t actually the predominant viewpoint in our understanding of probability. Furthermore, Kolmogorov’s complexity isn’t even computable. So, randomness is either a subjective measure or a non-computable objective measure. The author is then resigned to conclude the following insightful idea:
It might be that randomness is not mathematical, but physical notion. Thus one has to elaborate physical procedures guarantying randomness of experimentally produced sequences and not simply try to construct such procedures mathematically.
Randomness cannot be understood in mathematical terms. I’ve written previously about the limits of mathematical thought. The gist of this is that, there are a class of systems that embody universal computation. These systems are impenetrable by present mathematics. Stephen Wolfram conjectures that it in fact is not possible. This is a similar problem to the “Halting Problem” in computability.
My opinion is that randomness is a manifestation of complex information processing. This view is shared by Stephen Wolfram who uses a simple celluar automata (Rule 30):
as the basis of the random number generator in Mathematica.
If perhaps what is perceived as randomness is just an exceedingly complex computation, then can it be possible to discover some accidental hidden structure in that randomness?
Take for example the recent discovery in the digits of prime numbers. Quanta describes one bizarre discovery:
a prime ending in 9 is almost 65 percent more likely to be followed by a prime ending in 1 than another prime ending in 9.
This seems to violate a longstanding assumption held by mathematicians, that prime numbers should behave much like random numbers. Of course, prime numbers however are generated in a deterministic algorithm, albeit with a very strange unexplained computational bias.
Let’s explore this further by avoiding the use a deterministic algorithm and instead use one we explicitly rig with a random number generator. Will structure arise despite built-in randomness? Quanta magazine described research in this question in “A Unified Theory of Randomness”.
Sheffield and Miller have shown that these random shapes can be categorized into various classes, that these classes have distinct properties of their own, and that some kinds of random objects have surprisingly clear connections with other kinds of random objects.
There are theories in physics that assume randomness in conjunction with a few constraints that can lead to phenomena that is surprisingly universal. The Tracy-Widom distribution is one of these universal phenomena that seems to be present in many different contexts that involve the correlation of interacting elements (i.e. an entanglement). Systems can exhibit universal structure despite its constituents being originally random. The Gaussian distribution assumes that variable are disentangled, however the moment you add an entanglement assumption, a different kind of distribution emerges. Random variables that are entangled in some way lead to structure. Randomness with entanglement begets structure. This is shown experimentally as well as in Ramsey’s theorem.
To emphasize this point, let’s put it in a pseudo equation:
R * R’ -> Structure
Where * is an entanglement operator. R and R’ are sets of random variables. How does entanglement of random variables lead to emergent structure? Shouldn’t random variables mixed with random variables also be random variables? Apparently, not when you have a correlation between them!
Curiously, shared weights in deep learning architectures ensures entanglement (See: “The Holographic Principle” ). There has been some recent studies exploring random matrix theory in the area of deep learning. Jeffrey Pennington and Yasaman Bahri have written a paper “Geometry of Neural Network Loss Surfaces via Random Matrix Theory” that explores this, concluding with the discovery of some ‘magic numbers’.
In the two classes of systems described above, we find that we cannot avoid finding structure in randomness. One with deterministic computation seemingly creating randomness but not violating its rules (i.e. in the case of Pi) and the other kind a computation driven by randomness seemingly creating structure despite randomness.
I will make the following statement, “creating perfect randomness is incomputable”. It sounds counterintuitive, however you can justify this argument in that to create a computation that generates randomness would imply a fixed sized procedure and therefore that procedure describes the order in that randomness. This of is in similar spirit to Kolmogorov, where he showed that infinitely long incompressible sequences have the properties of random sequences.
In an earlier article, I explored “Chaos and Complexity” as being two mechanisms that are in play to arrive at ‘emergent complexity’. Chaos can be characterized as a positive feedback mechanism. Iterated function systems are examples of a chaotic system. The mechanism that leads to chaos is transformations applied recursively (or iteratively) on a current state. One can think of iterations as a process of transformation in the time dimension. That is, each transformation occurs in an interval of time. Think of this as “iterative complexity”.
As an example, there are many iterative algorithms to calculate the number Pi. The Indian mathematician Ramanujan, conjured up this version from his intuition:
How Ramanujan was able to intuit order in randomness is beyond my grasp. How it is able to discern patterns in complexity is one of the big mysteries of intuition. Why an intuition machine like Deep Learning works is related to this mystery.
The other mechanism, is the effect of the law of large numbers of diverse interacting components. Emergent macroscopic phenomena arises from the interaction of microscopic phenomena. Entanglement arises when there is a rich diversity of interacting parts. Diversity requires a composition mechanism for parts to organize themselves in different configurations. So as an example, in evolution, life’s diversity is created through the interchange of DNA components and in mutations. In chemistry, one has the elements that are able to combine with other elements to create more complex molecules. One can think of entanglement as existing diversity in space. That is, each diverse entity occupies different intervals of space. Think of this as “combinatorial complexity”. There are recent theories that explains gravity as an ‘entropic force’ that is due to quantum entanglement. This entanglement manifests itself also in Deep Learning in the form of ensembles and in that these networks way too many parameters.
However, macroscopic phenomena is also governed by the Second Law of Thermodynamics, that is, of the tendency to move towards increasing entropy. However, what does maximum entropy imply, it implies uniformity, that is a university without diversity. The fact that the universe has structure and has evolved complex life goes against the Second Law of Thermodynamics. That’s because, despite maximum entropy, structure will emerge as seen in the Tracy-Widom distribution. The physicist call this break from uniformity, “symmetry breaking”.
Quantum entanglement is something that is real despite Einstein’s protests. Classical entanglement has not been proven to exist. In this post, I explore something that I call ‘entanglement’ that relates to the complexity of interactions of a diverse set of components. These components evolve in parallel and interact asynchronously. We don’t have mathematical notions that describe these two concepts well enough. In general, we simplify them by thinking sequentially (i.e. Turing machine) and synchronously. Classically, there is also no ‘time-travel’, so its not as easy to formulate toy problems of entanglement.
Giulio Tononi’s Integrated Information Theory of consciousness posits an axiom for consciousness that is analogous to entanglement. Tononi’s integration axiom states that:
Consciousness is unified: each experience is irreducible to non-interdependent, disjoint subsets of phenomenal distinctions.
Tononi’s integration axiom implies a system that is highly correlated where cause and effect cannot be deduced. The entanglement that I describe may not exists in the classical physics sense however it certainly is probable for an informational perspective. I am not going to make the leap like Tononi that entanglement is a measure of consciousness. Rather, I am making that more conservative statement that is a contributor to our subjective perception of randomness.
Entanglement is an important property of complex systems, however I complete disagree with Tononi’s assessment that higher entanglement (or integration) leads to consciousness. Consciousness appears somewhere in a spectrum of chaos and entanglement. Intelligence is likely to exists at the edge of chaos, but never in the chaotic regime, that would imply a level of insanity. Similarly, intelligence will reside in a regime of entanglement that has properties similar to a fluid. Not in the phase of minimal entanglement like a gas and not in the phase of high entanglement like a solid.
From a computer science perspective, one can think of these two complexities as a consequence of sequential and parallel computation. The abstraction known as Turing machines are purely sequential computers. Parallel computation are notoriously difficult to analyze mathematically so theoretical computer scientists use just one form out of convenience. In theory, a sequential computer (i.e. a Turing machine) can simulate a parallel computer. A Turing machine can generate a chaotic computation.
I wonder though if a Turing machine can generate an entropic computation. Can a grossly entangled parallel computer program be made sequential and proven equivalent like the Church-Turing thesis? The way deep learning networks make predictions is based on the execution of many predictions in parallel (i.e. using multiple ensembles). Deep learning networks are known to be extremely entangled (as opposed to being sparse). The issue with theories of induction (i.e. Solomonoff induction) is that it does not take into account multiple parallel predictions. Rather it assumes that the best prediction is the one that is simplest (based on some complexity measurement). The best prediction does not necessarily follow Occam’s razor, the best prediction is the one that is most accurate and that can only be determined by a measurement in the future and not in the present.
Chaos and Entanglement, acting both in time and space leads to what we perceive as randomness. This randomness is the effect of emergent complexity and not some mathematical notion of intractability. There is no such thing as something intrinsically random (See the Khrennikov paper above with regards to the intrinsic randomness of quantum mechanics). Randomness is an abstract concept like infinity that exists only in concept and has no physical basis. True randomness is in fact achieved only with maximum entropy, which perhaps only exists when time is at infinity (the same as the venerated Central Limit Theory). In short, never. Ramsey’s Theorem in fact is an even more elegant proof of why true randomness is impossible in any interconnected structure.
Perceived randomness is an appearance of complexity and is created through deterministic automation, that is, deterministic information processing components (capable of computation, memory and signaling). Memory is computation that is deferred in time. Signaling is computation transferred in space. Life and intelligence also emerges from information processing, albeit in a form that is adaptive and has self-preserving behavior. How do these ideas of randomness and probability translate to the idea of adaptive and ultimately intelligent systems?
I will be remiss if I did not mention Marcus Hutter’s essay on this matter of randomness and AI: “Algorithmic Randomness as Foundation of Inductive Reasoning and Artificial Intelligence”. Let me highlight the difference with my thinking and that of Hutter’s. (1) I don’t subscribe to a universal simple measure that identifies an optimal decision, (2) randomness is due to a physical process, (3) the solution will not be found in mathematical abstractions but in computation that is restricted to the laws of physics, and (4) complexity arises from chaos and entanglement.
If randomness is a consequence of a physical process then perhaps our solution to the AGI problem should also take in consideration the constraints of a physical process. Our mathematics may be introducing too many assumptions that may be detrimental to finding a solution. It’s like a case of “leaky abstractions”. Our abstractions are hiding the essential details and in the process we are unable to see what is in front of our eyes.
Related reading:
https://arxiv.org/pdf/0708.1362.pdf
https://arxiv.org/pdf/1701.01107.pdf
http://csc.ucdavis.edu/~cmg/papers/idep.pdf
https://www.quantamagazine.org/machine-learnings-amazing-ability-to-predict-chaos-20180418/
