Peli Grietzer finished his PhD in mathematically informed literary theory at Harvard Comparative Literature and the HUJI Einstein Institute of Mathematics. He is an on-and-off contributor to the ambient/archival literature collective Gauss PDF, and working on a feature film with video artist Tzion Hazan.
Over email, we held a conversation on Peli’s work on the Theory of Vibe, summarized here. We spoke about how to relate mathematical analogies with literary theory, and bringing the idea of knowledge back into poetics.
BN: Would you give a summary of your work on the Theory of Vibe, or how machine learning autoencoders can help us think about literary theory and modernism?
PG: I argue that the most outrageous excess of Modernist, Symbolist, and romantic thought about art — roughly, the doctrine that the poet’s (or whatever artist’s) work embodies the truths of a deeper stratum of reality as pure aesthetic form — turns out to make excellent scientific sense, once we have spent some time thinking about autoencoders.
An autoencoder is a neural network tasked with learning to make facsimiles through a compression bottleneck. The point is for the neural network to learn how to leverage rich recursive patterns that holistically structure its training set, developing a kind of gestalt fluency that ambiently models its training set like a niche or a lifeworld. Importantly, what an autoencoder algorithm learns, instead of making perfect reconstructions, is to build structurally motivated approximations of the objects of the training set. In fact, the difference between an object in the training set and its reconstruction demonstrates what we might think of, rather literally, as the excess of material reality over the gestalt-systemic logic of autoencoding. I call the set of all possible inputs for which a given trained autoencoder has a zero reconstruction error, in this spirit, the autoencoder’s ‘canon’ — the set of concreta that materially embodies the autoencoder’s gestalt for the training set.
Heidegger famously talks about ‘Stimmung’ (mood), a ‘presumed view of the total picture’ that conditions any specific attitude toward any particular thing. Autoencoding, by extrapolating a space of possibilities from a collection of phenomena, gives individual phenomena meaning by relating them to a totality. In this sense we can see autoencoding as the sensate cognition of a kind of Stimmung of a system. How it vibes.
Learning vibes is crucial for all kinds of knowledge-how, but it’s a slow, long process, and communicating learned vibes to each other is a problem — a vibe has the structure of a trained autoencoder, which is mathematically and conceptually intractable. One mathematical fact about neural nets that neural-netty creatures like us can easily use, however, is the effective identity between a trained autoencoder and its canon: as I show in (minor) formal work with the mathematician Tomer Schlank, a sample from the canon of a trained autoencoder acts as an optimal intersubjective articulation of the trained autoencoder. I propose that the imaginative or material assemblage associated with a Modernist literary work is analogous to a sample from the canon of a trained autoencoder attuned to some worldly vibe.
I leverage this conceptual machinery to build a surprisingly mathematically concrete interface between three important ‘theories’ about the cognitive-aesthetic object of Modernist or avant-garde literary works, which I’ll describe here with a folksy paraphrase since they get pretty weedy:
A) A Modernist work aims to demonstrate a way of looking at the world (cf. Sianne Ngai, Jonathan Flatley)
B) A Modernist work aims to demonstrate a weak affinity between the many heterogeneous worldly or textual materials it patches together (cf.Tan Lin, Hugh Kenner)
C) A Modernist work aims to demonstrate systemic forces underlying phenomena that appear unstructured (cf. Charles Baudelaire, Gertrude Stein)
As it turns out, all three of these are good informal technical descriptions of the meaning of the geometrical structure associated with a trained autoencoder, and the details of how they relate to one another in that case are edifying and delightful.
BN: This is fascinating: I think it takes huge steps towards an exciting yet formal and rigorous vocabulary in both machine learning and poetics.
The thing I’m skeptical about is over, ultimately, how any of the formal aspects of the autoencoder analogy which you painstakingly set up translates over to a useful view of aesthetics. You preempt this at several points: in neural networks “we are always crudely approximating a process too ‘soft’ to correspond to any description in [language].” But what would the theory lose, say, if it only expounded on a theory of mimesis as compression?
PG: I’m not sure there’s anything to the concept of an ‘autoencoder’ other than ‘mimesis as compression,’ except for the stipulation that the compression algorithm is differentiable. Working through the technical details of the analogy is fundamentally a way to verify that the analogy means what we think it means. I think of the formal or mathematical disciplines as essentially characterized by the possibility of descent from intuitive meaning to a syntactical or mechanical substrate, so I think a mathematical analogy has to include at least some proof of concept for descent into mechanics or it’s not really a mathematical analogy. As to the choice of autoencoders over non-differentiable compression algorithms, this has to do with what I see as the general promise of artificial neural networks for critical and literary theory, and maybe with my odd mixture of Humean deflationary instincts and repressed Hegelianism.
The very-online empiricist philosopher Liam Kofi Bright once posted: ‘One of my firmest convictions is that the world is a deep sense boring. Astrology, Romantic social theories favoured in the humanities, Freudian or Jungian psychology, any kind of animism; they’re all false. Because if they were true the world would be interesting. And it’s not.’ I take this hypothesis seriously! Artificial neural networks are in some sense the most boring model of thought possible. They’re basically ‘big blobs of compute’ that learn by trial and error. So if you can show that your unnervingly interesting critical-theoretic paradigm finds purchase in the realm of artificial neural networks, or in cognitive-theoretic models of mind, world, society and so on where processes analogous to artificial neural networks play a key role, you’ve really gone and upset the order of things.
BN: There’s something in that tweet. I’d posit that neural networks are uninteresting because of their lack of causality: it’s notoriously hard to introspect on what the weights of a neuron mean. Moreover, if we take what is mathematically elegant as something superficially simple revealing deep insight, many of the advances in neural networks are inelegant, fueled by incidental detail and well-funded compute.
On the division between the interesting and the useful: one thing that does frustrate me about Theory of Vibe is, because the symmetrical functions of compression / expansion are largely abstracted, the theory can’t yet be directly applied in the shape of a script. What do your experiments of training models on MNIST/CIFAR, which relay heuristics about how training loss changes with sample size, attempt to show?
PG: The purpose of these experiments is to fix a technical gap in the proof that a sample from a trained autoencoder’s canon is the optimal intersubjective form of a trained autoencoder. The proof, like most proofs about neural networks, assumes that the cost function reaches a global minimum during training, whereas real-life neural networks don’t. (It’s not that they get stuck in a local optimum per se, but rather that we finish training when further gradient descent becomes unmanageable or results on a validation set recommend early stopping.) We don’t have a good mathematical understanding of the exact relation of real-life trained neural nets to neural nets at the global minimum of their cost function, so we need experiments to show that what holds given the idealizing assumption of training to a global minimum approximately holds in classical real-life examples of the material computational process we’re idealizing.
I think that a fully developed framework of aesthetics as autoencoding *might* apply literally enough to art’s capacity to do Jamesonian ‘cognitive mapping’ of social-material reality for technicalities like these to have explanatory power, but I’m early in the process.
BN: The rise in usage of the autoencoder hasn’t been necessarily technological in the sense of Heidegger’s provocation; the effectiveness of autoencoders and other machine learning methods wasn’t brought through breakthroughs in math — the neural network architecture has existed since the 70’s — but because of the rise of cheap cloud computation of matrix algebra. What do you think could precipitate changes in aesthetic genres on the level that the “unreasonable effectiveness of neural nets” precipitated changes in autoencoders?
PG: It’s common to associate spikes in the rise of what became the ‘realist,’ ‘empiricist’ tradition of the novel with the emergence of paper and ink as affordable everyday household items in the 15th century, and later with the democratization of the printing press and the rise of the book as a consumer-product (18th century), followed by cheap serialized publication (19th century). At the same time, these information-technology booms are also recognized as critical moments in the evolution of abstraction-minded, Symbolist literary representations that deal with the chaotic world of ideas and affects and social dynamics born of these information booms. This is sometimes known as ‘the poetics of information overload’: Rabelais (15th century), Laurence Sterne (18th century), Goethe’s Faust, Flaubert’s Bouvard et Pecuchet (19th century). I think it’s plausible that — as far as literature aiming at what Frederic Jameson calls ‘cognitive mapping’ of the social-material world goes — the need to make use of aggressive artistic abstraction to communicate models of the world of ‘medium-sized dry goods’ decreases as communication grows cheap, but these same information booms also throw us into engagement with new worlds too complex to deliver with a manageable volume of untreated samples, pulling us back into Symbol-making. Personally I resent granting the realist, empiricist tradition of the novel this much dignity, but certain formal issues to do with optimization, sample size, and dataset complexity actually force us to ‘predict’ something like this on the aesthetics-as-autoencoding model, and facts don’t care about my feelings.
BN: I wonder if there are critical moves and thought experiments that autoencoders inspire. The first thing that came to mind was Vendler’s explication of Stevens’ “Emperor of Ice Cream,” a weird experiment where she rewrites the poem into a narrative, almost like visualizing a “narrative neuron” of an autoencoder trained on poems. What kind of work do you want Theory of Vibe to point towards?
PG: The Vendler thing is cool in that, I think, it demonstrates how little we gain from the addition of a narrative, rhetorical, and logical structure, holding the vibe-making assemblage constant. I think Patricia Lockwood’s recent essay in LRB about the cognitive-aesthetic texture of ‘the discourse’ as a textual mass on Twitter is a really good example of the kind of work you can do when you take vibe-making assemblages as a primary cognitive-aesthetic structure. Obviously you don’t need the formalism of autoencoding for this, but I think the formalism is a useful guide to giving this kind of analysis internal structure.
I like talking about vibe as a logically interdependent triplet comprising a worldview, a method of mimesis, and canon of privileged objects, corresponding to the encoder function, projection function, and input-space submanifold of a trained autoencoder. This relates to what I’m really excited about, which is undoing the distinction between ‘interpretation’ and ‘erotics’ (in the Sontag sense) or between ‘cognitive mapping’ and ‘listing’ (which Jameson is fixated on), or between mimesis on the one hand and curation, installation, and collage on the other. The fundamental drive is really to create a viewpoint where the ‘radically aesthetic’ — art as pure immanent form and artifice and so on — is also very, very epistemic. The late 19th and early 20th century were full of these great home-brew epistemologies by Modernist practitioners, like Aimé Césaire’s “Poésie et connaissance,” where the radically aesthetic grounds a crucial form of worldly knowledge. However, in modern literary theory, the radically aesthetic tends to figure either as non-epistemic or as dramatizing epistemic failure, like in Ngai. One might argue that this is because modern literary theory looks at artistic organon-making (or even at ‘knowledge’ in general) in terms of ideology and subversion, not in terms of epistemic achievement, but I think that’s a red herring: literary theory has a strong Marxist tradition — Lukács, Jameson, Moretti — that is totally invested in the thesis that Balzac, for instance, was a maker of importantly perspicuous representations of actual structures of the actual world. Why not Kathy Acker or Zukofsky too?
BN: How do we think about autoencoders that fail? For instance, how do we think about neural net models that fail to regularize?
In the dissertation you walk us through how an optimizer trains the symmetrical algorithms for compression and projection. But a big part of what machine learning research is about, aside from building new architectures and optimizers, is how to avoid the model from learning total facsimiles of the training set. One way by which regularization is done, to push model training out of local minima, is to include some representation of the complexity of the model in the loss function. I’d like to see how a literary theorist could think about an autoencoder that overfits.
PG: The paradigm where a literary work aspires to the condition of good autoencoding is in some sense a Modernist/Symbolist/romantic paradigm of aesthetic practice, rather than an avant-garde or post-Modernist paradigm of aesthetic practice. I don’t mean this chronologically, per se — I think this paradigm is still really important in contemporary aesthetic practice, including some of the most radical-in-every-sense contemporary work — but rather that it has a fundamental continuity with the mainstay of Modernist/Symbolist/romantic aesthetic practice. One of my favorite things about the aesthetics-as-autoencoding model is exactly that, as you’re sort of suggesting here, it stays effective outside of this paradigm. The mechanics of degenerate autoencoding and the aesthetics of degenerate transcendental subjects go beautifully together.
BN: I feel that a lot of Sianne Ngai’s examples around the stuplime, like Gertrude Stein, seem to me often like results of a generative technique from an algorithm whose hyperparameters are untuned, which could work as a human-to-computation analogy on obsession, attention, and boredom.
PG: I actually tend to read Stein as a Cubist, and therefore a Modernist very much in the business of articulating objective structure rather than demonstrating subject-object misalignment, but your mileage may vary.
BN: What do you think about bias and ill-posed questions? I understand the autoencoder model through your exposition to be a transcendental schema, with a broad conflation between “holdout set accuracy” and “success at its capacity to model systemic structures and relations.” There’s a really fun paper which has anecdotes about the creative ways machine learning algorithms minimized their loss functions by breaking rules that researchers didn’t anticipate — for example, exploiting approximations made in the physics engines of simulators.
PG: My unfinished chapter about pataphysical literature was going to get into this!
BN: Thanks for walking me through the questions I had!
PG: Thanks for the great questions, and for your attention and interest in my work!
