Generative AI was a helpful tool for an adventurous art exhibition. The art helped with AI model development, too.
By Emily Reif, with Camille Benech and Lucas Dixon (with special thanks to Shahryar Nashat, Sara Sadik, and Rachel Rose)
Currently on display through January 2024 at the LUMA Arts Center in Arles, France, is an exhibition that showcases the work of three contemporary artists: Shahryar Nashat, Sara Sadik, and Rachel Rose. Each uses generative AI as a new art medium; I was part of a lucky team of engineers from Google Research who had the opportunity to collaborate closely with them on the work, on behalf of our Experiments with Google initiative. Our shared goal was to “foster an artist-centric dialogue in fast-developing fields where original voices need to be amplified.”¹
While I’m now an engineer and researcher, I also have a background in visual art, and have used AI in my own practice. I illustrated PAIR’s Wordcraft Writers Workshop stories using Imagen, and co-created Waterfall of Meaning with others on PAIR, which was shown as part of the AI: More Than Human exhibit at the Barbican Center. However, these projects were focused on AI as not only the medium but also the theme of the art. What I love about the exhibit at LUMA is that it centers the artists’ creative visions themselves: the pieces address a wide spectrum of highly personal, human themes, ranging from the loss of a friend, to cultural alienation in a virtual universe, and the externalization of inner psychological states.
Artists may seem like unlikely partners in AI model development, just as engineers and researchers might seem like unlikely artistic partners. But these sorts of close collaborations are what allow us to develop the technology for a wider diversity of people — in this case, for people who want to use AI as a creative tool. Here, I share some lessons I learned by working closely with one of the artists, Shahryar Nashat.
A reverse pipeline for an inkblot test
I had the opportunity to work with Nashat on his piece, “Reverse Rorschach”. In this piece, Nashat reimagines the classic psychological assessment involving a subject’s interpretation of inkblot images, and flips the premise: can the artist’s internal state be externalized in real time with generative AI? Mahima Pushkarna, PAIR’s senior UX designer, and I helped with initial prototyping and ideation for the piece, translating the possibilities of the generative model (a variant of Imagen²) into what would be practical within the scope of the project. As Nashat describes it, “This work is like a real-time self-portrait that is constantly changing over the next six months.”³
I admit, before working on the project, I hadn’t known much about the Rorschach test other than that it involved showing inkblots to patients and interpreting what they see. I was surprised (and, to be honest, still somewhat skeptical) to learn that specific interpretations of images indicate specific internal states. For example, when a patient is calm and serene they tend to see human motion in the inkblot. Also, these are not random ink blots, there are ten specific images that have been deeply analyzed, and are always shown to patients.
In Reverse Rorschach, Nashat has created a pipeline to reverse the process. He tracks his bodily movements during typical everyday life for the duration of the piece, via biometric sensors. The final piece is a video installation that continually updates in response to Nashat’s body in real time. The sensor data is translated to continually changing semi-abstract, semi-representational inkblot images using a text-to-image model. For example, when Nashat has a low heart rate, the inkblot morphs to resemble an image of people running. Similarly, when Nashat’s vitals suggest a state of sadness, the image turns gray (according to Rorschach, when a patient is depressed, they tend to see inkblots as achromatic).
The output is beautiful, creepy, overwhelming, calming, and generally just enthralling — like the range of feelings one might experience on any given day. Watching the images morph in and out of abstraction, I felt like I was part of a new type of creative dialogue between the AI model, Nashat’s body, and as the viewer, my own subconscious thoughts. I wondered, as I observed the generated abstractions, is Nashat aware that we’re watching these images generated by his movements right now, and is that affecting the output? How much of what I’m seeing is my own attempt to read meaning into abstract images? Would a different model, trained or prompted differently, have the same outputs for Nashat’s emotions? Should I keep these interpretations to myself, because maybe they say more intimate things about me than the piece?
Generative takes media art to a new scale
To be fair, while generative AI as a technology was vital to making the piece come together to realize Nashat’s artistic vision, the AI model was actually a relatively small part overall. I’ve heard critiques of generative art being too simplistic: write a short description and make the model do the work. This was the opposite — Nashat was the one making the creative choices, and using the generative AI as a means to achieve his artistic vision.
As he states, “The automation and repetition of tasks like generating images is really new and fascinating. I don’t think artists had a similar tool in the past that could continuously generate images and be part of the creative process.”⁴
I was surprised at the ways in which the creative process was both similar and different to what I’ve seen with traditional (non-AI) art. As Nashat pointed out to me, the level of control was strikingly different: “with GenAI, I make core decisions that will affect the visual but ultimately I am not in control of the visual result.” On the other hand, I also saw some surprising similarities to traditional media as well: for example, an artist will choose to use oils, collage, or another medium based on their vision for a piece. When they are exposed to a new medium (e.g., watercolor), it broadens the range of the art they can create. Similarly, generative AI could simultaneously be used as a means to achieve what Nashat already had in mind, and it also shaped what was possible for the piece.
Overall, there were key aspects that could not have been possible without generative AI. The aesthetic itself was partially driven by initial generative experiments. Another aspect that generative AI made possible was scale: there were potentially hundreds of thousands of combinations of sensor data used to create the images. If there wasn’t a programmatic way to create the images, this would have been intractable.
Limitations — and the importance of interdisciplinary collaboration
However, like any other medium, the generative AI we used had unique challenges, too. There were frustrating quirks of the AI model, like how small variations of the input significantly changed the output. Compared to other media, Nashat worried that it would be less likely to produce happy accidents: “If the instructions are taken too ‘word for word’ then there will not be as many surprises,” as he says.
From a technical standpoint, the model was experimental. One significant problem about this fact was that the most useful model (one that could edit images using generation) ended up being deprecated halfway through the collaboration because it was an experiment, not a product. My largest contribution ended up being the engineering grunge work to launch and maintain it so it could be used not only for Reverse Rorschach, but for the other artists’ pieces as well. Going forward, to be helpful for artists in the wild, it’s clear we need to prioritize usability and productionization for experimental models for creative purposes, and label them as such and with guardrails such as gating for trusted creative users only.
Ultimately, the collaboration brought into focus both the potential of generative AI as a new artistic medium, and also the friction of actually getting cutting edge research into the hands of artists during both the artistic and AI development processes. If we want to make sure we’re building tools with affordances that are actually useful and empowering to our end users, we need to be working directly with them starting from day one. And while this is a difficult area to navigate and coordinate, it’s definitely worth the result.
You can read more about Nashat’s work as well as all of the pieces included in the show here [link]
Acknowledgements
The LUMA Arles & Google Research initiative was organized by Luma (Simon Castets, Vassilis Oikonomopoulos, Fabian Gröning) in partnership with Google ATAP (Camille Bénech-Badiou, Gabriel Vergara II), Google Arts & Culture, and Google Brain. Thanks to Dr. Joelle Barral, Dr. Olivier Bau, Dr. Lucas Dixon, Dr. Kelly Dobson, Laurent Gaveau, Sebastien Missoffe, Dr. Ivan Poupyrev, Amit Sood, and Jonathan Tanant.
