The Synthetic Language for Live Performance of Kat Mustatea
Explorations of AI Art — Episode 25
[This interview has been previously published on Cueva Gallery’s blog on August 12, 2020]
“ Machines don’t have goals when they produce text, and it shows in what’s produced — so without human intervention and rewriting, that kind of word salad, no matter how beautiful, just falls flat as dramatic speech.” — Kat Mustatea.
Over the years, technologist and playwright Kat Mustatea has built an impressive portfolio career. After studying philosophy and sculpture respectively at Columbia University and Pratt Institute, she transformed her early interest in math into a working position as a software engineer and product manager. Her passion for theater and live performance, however, never waned, and indeed led her to found a theater company in Berlin for which she wrote and directed plays.
Mustatea is a member of the Interactive Experiences track of NEW INC, the art and tech incubator at The New Museum of Art in New York City, and co-curator, together with Heidi Boisvert, of the performance series EdgeCut. The series, which investigates different aspects of our relationship with the digital, started in January 2020 at NEW INC and had a very warm and enthusiastic response. The public appreciated the live performance works that involve the use of a wide range of cutting edge technology. Due to the pandemic, unfortunately, the series has been shut down, but a new partnership with a cultural institution, the New York Live Arts, will bring a digital event series this September.
As a writer and public speaker, Mustatea focuses on the impact of technology on art and society. Her TED Talk raised questions on the value of art in the age of intelligent machines, shifting the attention from the output to the artistic process, and addressing the unsettling experience of machines’ success. Her essays appear in Forbes, The Week, and Hyperallergic and tackle several aspects related to AI, including those about marketplaces and social media. In her work, one point remains constant, and that is the concept of value: value in art, experience, and being humans in a time where machines could possibly threaten our creativity and deep essence.
Her plays, designed as cross-disciplinary works for the stage, have been performed in New York, Chicago, Berlin, and Oslo, and often take into account aspects of transformation and hybridity. As Mustatea explains, she has been largely fascinated and influenced by Metamorphoses, the work of Latin poet Ovid in 8 AD. The poem describes the history of the world from its creation to the deification of Roman general and statesman Julius Caesar within a loose mythico-historical framework. At the beginning of his work, Ovid declares the subject of his writing: “In nova fert animus mutatas dicere formas corpora” (“My inspiration brings me to speak of forms changed into new entities”). Mutation is the starting point of the work of Mustatea too: people transform into different animals, and with them their language changes.
The synthetic language becomes an opportunity to represent the uncanniness of the metamorphosis, as well as the one of a lived experience online due to the partial lock-down for Covid19. But there is more: this language made up of constraints, in works like the one staged on Instagram — and of which we talk with the artist-, becomes an expression of loss, bewilderment and entrapment in a timeless dimension in which New Yorkers have lived through the most terrible days of the epidemic.
In this interview, the playwright immerses us in the conception and creative process that led her to use machine learning to create a language of mutation. In this, a critical approach to technology drives the interaction with the machine, focusing on what it means to be human, on the possibilities and limits of using Artificial Intelligence to create a hybrid language and on the fundamental role of human intervention before and after the process.
Beth: In some of your plays the characters speak with constraints, and develop their own language when transforming into animals. Can you explain the language of hybridity?
K.M.: The cat language was a happy accident. I had written a play called The One And Only Amanda Palmer, in which a group of girls, all named Amanda Palmer, can turn into cats.
In conversation with Laura Aina, a computational linguist at Pompeu Fabra University in Barcelona, we had got to talking about the Oulipo movement of the 1960’s, where writers would use absurd language constraints in their work — the most famous is the 300 page novel by Georges Perec, La Disparition, written entirely without using the letter ‘E.’ Machine Learning is so sophisticated now, this can be easily done with the latest computational techniques — but to what end? Unlike Oulipo’s conceptual feats, meant for the page, which were all about technical virtuosity — I am interested in living language, in spoken language that conveys something fundamental about the person speaking it. I started to wonder if maybe we could find a particular constraint that could convey the quality of “catness” of the Amanda Palmer in my play.
We used a pre-trained language generation model available on the web, augmenting it to produce sentences that have only words that don’t contain the letter “E”. Once that was working, it was trivial to omit any letter, or even groups of letters. You find out quite quickly that omitting a vowel produces interesting results, but omitting several letters at once just produces strings of gobbledygook. The art is in choosing the right kind of constraint.
For me this was a lightbulb moment: that different constraints produce distinctive sound textures, the way timbre works in music. As an exercise, I ‘translated’ one of the speeches in The One And Only Amanda Palmer into “no E” text — and it worked so well, that it is now the opening speech in the play. Amanda Palmer is talking about transforming into a cat, and her language is otherworldly, just weird enough to convey the strangeness of the moment, as she is in the middle of an act of transformation. The language has an operatic quality that for me captures “catness.” The audience doesn’t necessarily have to know what the constraint is to appreciate the uncanniness of the effect — in fact, I have wondered whether it is even necessary to tell them that it’s happening. And then for me it’s interesting that a constraint would lead to the expressiveness of the characters, allowing them to describe these dangerous, subterranean, cat-like impulses.
A lot of my plays work off of metaphors of hybridity and transformation. Ovid’s Metamorphoses has been an ongoing source of fascination for years. For example, I had written another play, Lizardly, in which a couple is turning to lizards in the middle of a fight. So it felt only natural to ask: is there a language for “lizard-ness”? I also wanted to push further technically with the model: instead of excluding sounds, I wondered what it might be like to favor certain ones. When you favor “F” and “Z,” the effect is comical and a little vulgar — it felt just right for the lizardly characters in the play, so I am developing a new version of the script that works off this particular texture.
More broadly, I am investigating how different kinds of computational techniques might evoke different types of hybridity: for example, “fly-ness” or “tree-ness.” This process is, in my playwriting work, a tool for giving voice to the kinds of improbable characters that seem to populate my story worlds. But even when I write about impossible creatures, and do unsettling things to language, my aim is still to find ways to talk about what it means to be human.
Beth: What is the role played by Artificial Intelligence in your writing process?
K.M.: To be clear, it is definitely me composing, and not the AI. What the model helps to do is move faster through the initial research phase. By producing a lot of language quickly, I can start to feel out if it has the texture I want, before I go through the laborious process of actually composing. And I do end up using snippets of text the model produces verbatim, whenever it is interesting. But anyone who has worked with human speech understands that you can’t simulate the psychological coherence of human communication — the way human speech has subtext and velocity and intent. Machines don’t have goals when they produce text, and it shows in what’s produced — so without human intervention and rewriting, that kind of word salad, no matter how beautiful, just falls flat as dramatic speech.
Beth: How does a synthetic language impact the performance? How did the actors deal with a synthetic language?
K.M.: Project Y Theatre is producing an online version of The One And Only Amanda Palmer, while in residence at The Orchard Project. It’s a learning experience for all of us: theater is tricky online, because what comes across in person is easily lost on a screen. (Although we are no longer in full lockdown in NYC, we are far from being able to gather safely in a theater space). What I’ve been thinking about for this performance in particular is how the synthetic language is an opportunity to express the uncanniness of the story, as well as the uncanniness of this approximation of a lived experience online.
We perceive language so differently when it is spoken than when it is written. You can do all kinds of language pyrotechnics on paper, but if the text doesn’t also contain psychological complexity as thought — if it doesn’t carry meaning, and is just a pretty string of words — then actors don’t know what to do with it. In good dramatic writing, each sentence has to do at least three things at once: it has to sound good, it has to sound probable as something that particular person is likely to say in that particular moment, and it has to carry a sense of urgency and emotion — otherwise, why bother saying it? Actors and directors are the first stress-test for a play: they can usually tell right away if the writing is weak. My job as a playwright is to give them language that does all of these things: so, whether they are synthetic or not, first and foremost the lines have to be dramatic.
Beth: What do you think of recent systems that generate language (e.g., GPT-3)?
K.M.: You know those composite faces created by averaging out thousands of human faces into one? Text generators are the language equivalent of a composite face: somewhat beautiful, but way too bland.
Beth: In Instagram you are staging a digital performance called Voidopolis. Can you tell us more about this project?
K.M.: Voidopolis is a loose retelling of Dante’s Inferno, informed by the grim experience of wandering through NYC during a pandemic. Instead of Virgil, my guide is a caustic hobo named Nikita. The story unfolds on my Instagram feed over 40-ish posts.
I consider this project a digital performance, meant to be temporary and then to disappear: a peculiar work in a peculiar moment in time — a time that is so haunting, that most of us will want to one day forget it entirely, like a bad dream. I started it on July 1, 2020, and it will last until approximately October 2020. Once the story is complete, I will be deleting it all from my feed.
Someone called this work “Oulipo-adjacent,” and I would agree. It is certainly more about the written than the spoken word, though I’m still looking for the text to have a verve and style, to be not only a technical feat but to have some punch and velocity. To be exciting to read. But I am using synthetic language here in a very different way than in my hybridity plays. The missing letter is more conceptual, tied to the missing people all over the city who became sick and died all these past months. The images are also about loss: I create them by ‘wiping’ humans from stock photography. I was inspired to do this by the artist damjanski’s ‘post-human’ app, Bye Bye Camera.
Why did I choose to omit ‘E’ again, out of all letters that could be omitted? In English, regular verbs in the past tense end in “-ED,” and by eliminating the past, time warps for the story’s narrator, who is somewhat disoriented to begin with. This mirrors my real-life experience under lockdown: I feel like I’m trapped in an endlessly unfolding present, disjoint from whatever everyday life was before, as if there is no past at all. Days feel like months, with only a vague sense of what the future is.
And this is why working with synthetic language is so powerful and eerie. This summer I was composing so much without “E,” that I started having dreams about trying to speak without “E” and feeling tongue-tied. Something similar has happened to me after being immersed in a foreign language. This was even more peculiar of a sensation, as if my brain was adjusting to being rewired not only for what’s possible to say, but also for what has to remain unsaid. There is so much talk of ‘training’ AI models, but in a sense, the model ended up training me.∎
To follow Kat Mustatea:
Twitter: @kmustatea
Instagram: @kmustatea
Website: http://www.mustatea.com
Announcement
The upcoming performance for The One And Only Amanda Palmer (play) is scheduled for August 15th: link here.
References and Resources
http://www.mustatea.com
https://www.forbes.com/sites/christinawallace/2018/01/15/podcast-kat-mustatea-makes-art-in-the-age-of-machine-intelligence/#79886bf61fd4
https://medium.com/edgecut/puppetry-or-how-i-learned-to-stop-worrying-and-love-art-made-by-ai-3201c770a176
https://www.ted.com/tal/kat_mustatea_what_is_the_value_of_art_in_an_age_of_thinking_machines
About the author: Beth Jochim is the Creative AI Lead at Libre AI, and Director and Co-Founder at Cueva Gallery. She works at the intersection of technology and arts. She is actively involved in different activities that aim to democratize the field of Artificial Intelligence and Machine Learning, bringing the benefits of AI/ML to a larger audience. Connect whit Beth in LinkedIn or Twitter.
