ML Day #8 — Microscopic Instability in RNNs

Unknown terms:
 — “McCulloch–Pitts binary neurons”
 — “zero-temperature relaxation dynamics for a spin glass” [0]

What’s interesting about this essay is that it’s approaching the problem of understanding a deterministic computational system through the lens of non-deterministic science (thermodynamics) [1]

The question of microscopic instability is explained thus: you have a RNN — you put some input in and let it run recursively for a large number of iterations, N — basically until the system attains some sort of stable state (either statically or dynamically — meaning that either the distribution of neuronal activations is fixed or that it is changing in some periodic, fixed way). Then, you change the state of a single neuron. Apparently, whether or not the state of the entire system diverges or not is based on what type of macroscopic stability was attained after the first N iterations.

Interesting Quote:

“It should be noted that a neural network expresses microscopic instability in the entire parameter region in the limit of a large number of neurons, which corresponds to the thermodynamic limit of gases. Thus the coexistence of microscopic instability with macroscopic stability is expected to play an important role in the information processing in the real neuronal circuitry consisting of a huge number of neurons.”


[0] Why do people reference shit like this? Does anyone know what the “zero-temperature relaxation dynamics of a spin glass” are? Is this a GOOD metaphor!?! I assume that the authors are appealing to some deep thermodynamics, but then who are they writing the paper for? I feel like part of the reason that research in NNs and other fields goes so damn slowly is that researchers feel pressure to complexify their thoughts to make them seem less accessible, thereby justifying their own academic positions. Then again, this paper was written by Japanese researchers, and perhaps such relaxation dynamics are fairly commonly known in Japan? Or perhaps it’s a mistaken or awkward translation?

[1] One interesting aside is that the chaos arising inherently from the floating-point arithmetic used in NNs and especially in the cyclical computations of RNNs might actually render the system non-deterministic.