Augmented Imagination: Machine Learning Art as Automatism

Philipp Schmitt
Jan 15, 2018 · 6 min read

In the periphery of the landscape that is the current boom of machine learning (ML) sits a playground for tech-savvy creatives who re-appropriate the technology for their means.

In one corner, there are designers focussing on applying the strengths of neural networks to their field. They dream up new “intelligent”, generative tools that, for example, help them analyze data or produce a thousand variations of a design to select the best [1].

In another part of this playground are artists interested in the new medium’s own expressiveness: “[ML] is becoming a tool, just like painting”, artist Trevor Paglen told me at the opening of his recent exhibition [2]. Paglen and other artists like Mario Klingemann and Sascha Pohflepp have recently produced interesting visual work — let’s call it machine learning art (MLA) — that prompted the thoughts behind this essay.

Image for post
Image for post
Left: Trevor Paglen. A Man (Corpus: The Humans) Adversarially Evolved Hallucination
Right: Sascha Pohflepp. Spacewalk: Carnivores 3, Generation 320.

These images are similar in their their visceral evocative qualities, their organic “brushstrokes” and textures, their surreal creatures and objects. They are usually made to depict familiar objects and draw from visual material of this world, but have something alien to them that fascinates me.

The technology to make these works wasn’t invented for or by artists. The scientists and engineers around our playground use these images as well (but tend to discard them after). They use them to better understand how their neural network works — or doesn’t. In other words, the images — as debugging tool or artwork — often tell us more about the machine itself and the people who made it rather than about the depicted subject matter. [6]

MLA is interesting because of the network’s tendency to mis-represent or represent in alien ways. The glitches in the system make the fascination. As researchers are working on making the technology more accurate and explicable, its associative potential decreases. Any sufficiently advanced MLA system loses its magic, so to speak.

Machine Learning Art and Surrealism

Image for post
Image for post
Left: Max Ernst. Wizard Woman. 1941. United States
Right: Max Ernst. The Angel of the home or the Triumph of Surrealism. 1937. Paris, France

I will not proclaim Paglen, Klingemann, or their computers as contemporary Surrealists. Rather, I want to compare MLA with Surrealist technique to propose a different way of thinking about machine learning as a creative tool. Imagine looking over to the corner of the playground where the designers are playing. I want to explore the space in-between.

What do Surrealist Automatism and Machine Learning Art have in common?

There are many automatism techniques. Let us look at one that every kindergartener knows: Frottage. Frottage is a technique developed (not invented) by Max Ernst in 1925. This technique picks up textures from structured surfaces by placing a sheet of paper on top and rubbing over it with a pencil. [5]

In this video Max Ernst demonstrates his use of frottage.

Frottage starts with an object (fig. 1). By transferring the object’s textures to a canvas or piece of paper, the artist creates a two-dimensional abstraction of the object. Often, multiple frottages are combined on a single sheet. The textures form evocative images. They prompt connections to other, often unrelated objects, places, or creatures that the artists responds to by refining the image to bring out more the desired subjects.

Image for post
Image for post
fig.1 — Frottage process

MLA often follows a similar process (fig. 2). Leaving aside technical details, this process can be generalized as follows: It starts with a dataset of thousands or tens of thousands of objects (or digital representations thereof). Through training, the artificial neural network creates a high-dimensional abstraction of the object’s features: the model. A model can be made to visualize what it “sees”. It shows us textures and shapes somewhat characteristic to the object in ways that often look similar to frottage.

Image for post
Image for post
fig.2 — machine learning art process

I would argue that both processes create images with similar evocative qualities or potential. But there are, of course, also differences between the two. One is that neural networks inherently strive for representation whereas frottage and Surrealism in general seek to dissolve it.

Another striking difference is that frottage is generally used to launch an artistic process whereas MLA usually is not. And this, finally, brings me to the point of this essay.

Machine Learning as a tool for imaginaries

Image for post
Image for post
Max Ernst. The Fugitive (L’Évadé) from Natural History (Histoire Naturelle). C. 1925, published 1926 © 2017 Artists Rights Society (ARS), New York / ADAGP, Paris

As I have shown earlier, in current, early MLA the images mostly stand for themselves as artworks. The tool itself is the center of attention.[7] In a way, they stop where things get interesting.

What if we employed this means of making images neither as practical debugging nor as art in itself? But rather as an art and design tool for mind bending — like Surrealist frottage; one that caters to the subconscious, the associative, the imaginary rather than rationale?

This essay is part of my graduate research at The New School. Thanks to my advisor Fei Liu for her invaluable feedback.


[2] Paglen, Trevor. “A Study of Invisible Images.” Metro Pictures. September 8, 2017. Accessed December 03, 2017.

[3] Breton, André. “Manifesto of surrealism.” Manifestoes of Surrealism 15. 1924.

[4] MoMA. “MoMA | Surrealism.” Museum of Modern Art. Accessed December 03, 2017.

[5] Tate. “Frottage — Art Term.” Tate. Accessed December 03, 2017.

[6] Paglen, Trevor. “Invisible Images (Your Pictures Are Looking at You).” The New Inquiry. October 02, 2017. Accessed December 03, 2017.

[7] EyeEm. “Photography through the Eyes of a Machine — EyeEm Blog.” EyeEmƒ. September 08, 2017. Accessed December 03, 2017.

[8] MoMA. “Natural History (Histoire naturelle) | MoMA.” The Museum of Modern Art. Accessed December 03, 2017.

[9] Tate. “Automatism — Art Term.” Tate. Accessed December 06, 2017.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store