Nietzsche, History, and Reinforcement Learning
In “On the advantages and disadvantages of history for life,” Nietzsche inquires into what kind of history is most advantageous for life. In other words, as a society, what should we be recording to ensure our survival?
In this post, I will show how a lot of Machine Learning — in particular Reinforcement Learning (RL) — aligns well with Nietzsche’s conception of “useful history” or “history for life.” My hope is to continue inquiring into what insights from Nietzsche we can apply to computational problems in learning and cognition.
Before I go into the specifics, I’d like to make a point that Nietzsche does not really make explicit in his book. Everything Nietzsche says about history applies to memory when we move to the level of the individual. Another way of saying this is that history is a form of collective memory. So as philosophical or scientific problems, history and how it’s organized within a society is analogous to memory and how it’s organized within a mind. Like Riemann’s geometry, a lot of what Nietzsche wrote provides insights into problems their creator could not foresee. In particular, Nietzsche’s book on history seems to have a lot to teach us about how to build “machines that learn and think like people.”
Whether we are talking about human or AI agents, these agents are constrained by memory capacity and the amount of information they can “integrate” or query at any time. Given these constraints, we might ask these questions from the perspective of an agent: “What kind of things should I remember? How much of them should I store? How and where should I store them, given how I’m going to use them and retrieve them (frequency, purpose, which other memories should be summoned with it, etc.)? What things should I discard and when?”
I was talking to Vishal about an Einstein quote: “Never memorize something that you can look up.” A seemingly good piece of advice until we started taking it too seriously:
“- Is that person coming, the one you invited?
- I don’t know; look it up!
- Where do you lookup where to look things up?”
Going back to Nietzsche, in his thin book (one of the main reasons why I picked it up in the first place), he writes that there are 3 types of history:
- Monumental
- Antiquarian
- Critical
The first one — monumental — is about big achievements that are often done from within a “cloud” that ignores a lot of things about truth, history, the past.
The second one — antiquarian — is about preserving things for their own sake, for the sake of saving random bits and pieces of the past.
The third one — critical — is centered around evaluating and criticizing the past.
An excess of monumental history blinds us to errors or unnecessary risks; a lack of it makes us take initiative less often.
An excess of antiquarian history turns history into a meaningless list of facts; a lack of it makes it too easy to invent pasts that never existed.
An excess of critical history prevents us from acting; a lack of it makes it too easy to repeat our mistakes.
A very Nietzschean problem poses itself very quickly: we often have to decide between truth and life. In other words, some true information is detrimental to our survival. Easy example: if the thought of the inevatibility of one’s death was constantly on one’s mind, their life would probably be harder to live. We find many examples in the Economics literature about how a lot of our biases are the product of evolution. This is another way of saying that life ends up dominating truth.
Going back to monumental, antiquarian, and critical history, I’ve mentioned that we can apply these histories to individuals. I’m sure you can think of a great achievement of yours (monumental), a memory or object you’ve kept for no particular reason (antiquarian), a mistake you made that you will never repeat (critical).
For instance, monumental history is memory of reward. Within a RL framework, once a child learns how to recognize different digits, the encouragement she gets when she gets it right (or the negative stimuli that comes with getting it wrong) “propagates back” and “builds-up” the representation of different digits. Remembering more intense rewards or punishments is important because they tend to be more important.
Another feature of monumental history that Nietzsche highlights is that the effect of a reward (aka achievement) can propagate between domains that can look very different — for example, from surviving a storm to winning a marathon, or from impressing a friend to asking someone out. This is in spite (or because) of the fact that memories surrounding rewards are carried through very noisy and corrupting (turn subtitles on) processes (emotions, aggrandizement, time, “competition” from other memories, etc.). This is true because of the importance of confidence, but also because high-dimensional problems probably share common structure. This is connected to the promising domain of transfer learning.
Moving on to antiquarian history. We can think of the antiquarian as a process that decides to store (old) things at random. In other words, you’re sampling at random from the distribution of the world/your senses/your thoughts and “recording” these samples somehow. This is what allows you to “feel” that someone’s been in your room. This is what allows certain seemingly useless memories and thoughts to save you under pressure. Antiquarian history does not have a clear analog in RL. However, the importance of antiquarian history teaches us that we shouldn’t only strive to remember what we think is going to be useful. Even though memory and processing are constrained, it’s still worth it to leave some room for a more random and more exhaustive kind of sampling. Another way of saying this is that if you forget all the things that don’t matter, you will find out that they matter!
Finally, critical history — our ability to notice our mistakes and learn from them — is very important. This type of history has a clear analog in RL that’s called actor-critic. In this framework, the agent charged with making decisions is “split” into an actor and a critic. The actor figures out the policy according to which it chooses its actions; the critic evaluates the action and gets better at predicting the value of actions from how the world reacts to the actor’s actions. Unless you’re double-majoring in Philosophy and Computer Science, this is probably not how you live your life. But you do find your approach to morality and decisions to be evolving over time after each failure and success (broadly understood).
Emma pointed me to Generative Adversatial Networks (GANs), which is an unsupervised Machine Learning technique. In GANs, there are two networks learning: a generative neural network and a discriminative neural network. The generative network learns how to “imitate” a given data distribution, and the discriminative network learns how to discriminate between real instances of the distribution and ones that are synthesized by the generative network. The model is done learning when none of the networks can make progress against the other one. For example, one network can learn to generate images of Jennifer Lawrence while the other one learns how to discriminate real images of J Law from generated images. The promising successes of GANs indicate that Nietzcsche was right: it’s important to have a “critic within.” More generally, splitting a single system into two or more separate entities that collaborate and compete with each other sometimes leads to more efficient solutions. I’m not a Freudian, but it seems obvious to me that there’s a lot of “unconscious” processing that happens in our heads and our body — information that circulates but that we are never “consciously” aware of. This seems like a feature, not a bug.
How we choose to remember our history is a difficult problem that touches into a lot of disciplines, from History to Communication Theory. It also raises many important practical and political questions around education, media, and urban planning (which are powerful memory transmission mechanisms). One of the most lucid thinkers we know, Friedrich Nietzsche, grappled with this problem under many different forms. Because of this, I believe that Nietzsche has a lot to teach us about how our perceptual apparatus and our thought-process function. In learning about these questions, we will gain insight into how to build more intelligent artificial agents.
Another interesting question is: how do we remember history? It seems like Nietzsche leaves out a lot of the historical transmission process, in particular things that are associated with women and older people: education on social norms, emotional development, storytelling…
Since we’re on the topic of history, I’ll leave you with a Karl Marx quote I really like: “Men make their own history, but they do not make it as they please.” I think this is true for both meanings of make: enact and record/transmit.
—
Reading Recommendations: please comment or send me any kind of work that this makes you think of! In exchange, I’ll keep you updated about where this goes.
