Experiments in experiential AI — Insights on the uncanny interplay of humans and machines
Drew Hemment, Matjaz Vidmar, Daga Panas report on AI & Arts research by The New Real, a joint initiative of The Alan Turing Institute and University of Edinburgh.
L-R (bottom): images from Sarah Cistion, generated using Craiyon
Recent developments in diffusion models and large language models have powered a new generation of AI tools, opening unprecedented opportunities for artistic creation, with profound implications for the arts and society.
In The New Real, we explore the ways artists can push creative boundaries with AI, and how AI can be enriched or challenged by the Arts. We study and develop what we call ‘experiential AI’, in which AI is made tangible and explicit, to fuel cultural experiences, and to make AI systems more accessible to human understanding.
In our current project we investigate how to address limitations in the current crop of generative applications. Often, the only creative input is through a text prompt, and they are trained on massive datasets scraped from the Internet without permission or fair pay for the original creators. Through an extension of our platform, and an Art Commission, we are testing methods to give artists increased access and control over an AI model and to creatively explore a machine learning dataset. We want to see how more granular control can contribute to transformative experiences for audiences, and open new thinking on key challenges such as authorship, consent, harmful bias or energy use.
We developed an experiential AI platform with and for artists combining AI algorithms and a number of easy-to-use tools. Using our platform, artists can iteratively curate data by training an AI model and creatively exploring the results. You define the dimensions you want the algorithm to explore, and use simple tools to probe the latent space. The first release of the platform in 2022 worked with images and Generative Adversarial Networks, and integrated localised climate predictions. New for 2023, we added an ability to work with natural language processing, words and texts. We integrated Word2vec, which is not as advanced as more recent algorithms, but is a good choice to test various techniques to give artists more creative agency and control. We have developed bespoke tools such as a “Slider” and a visualisation tool as an accessible interface to explore the latent space, without the need to run your own code.
Through an open call we awarded five development awards for artists to conduct R&D with our platform and data from the British Library — Kasia Molga, Johann Diedrick & Amina Abbas-Nazari, Alice Bucknell, Sarah Ciston and Linnea Langfjord Kristensen & Kevin Walker.
Intelligent avatars of people we have lost
Kasia Molga went on a very personal journey to sense and ‘recreate’ fragments of her dad’s personality in word associations generated from the diaries he left behind after forty years on the high seas — to create an illusion of his presence. She experimented with small data, and worked between English and Polish languages, with unexpectedly positive results. She next plans to deepen her own understanding of the ocean by combining his writing on marine habitats with ship logs, a British Library dataset of maps from the Mediterranean Sea, and climate prediction data from The New Real’s platform.
Voicing the unspoken
Johann Diedrick and Amina Abbas-Nazari tackle the lack of diversity in AI data by giving voice to unheard voices. Using an archive of newspapers from the UK and Caribbean from The British Library, they question how it is possible to use textual AI models to “fill in the gaps” of voices missing from historical data. The artists examine the origin of synthetic voices that are speaking to us today, and plan to produce speculative speech that never existed, and make those voices audible for contemporary ears.
Planetary intelligence — probabilistic, symbolic and speculative
Alice Bucknell trained a model on contemporary and historical reporting of extreme weather events and combined it with The New Real platform’s climate prediction data to 2097. The output is a fictional film narrated by an AI that imagines ways of visualising, describing, and understanding the weird weather patterns of the future. Her research brings together probabilistic information from climate models and generative AI with speculative fiction to expose how these different modes of knowledge intersect.
Ethical and conscientious approaches to generative AI
Sarah Ciston explores the artistic and ethical implications of large-scale collaboration between machines and humans. Their work involves careful curation of datasets around specific problems as an alternative to all-purpose models generated by hoovering up all data without consent. The artist presents a vision for community-centred approaches to machine learning and more conscientious dataset stewardship.
New metaphors for the entanglements of humans and machines
Linnea Langfjord Kristensen and Kevin Walker use the metaphor of the “fold” to explain our entanglements with AI. Original creative content is absorbed into the vast datasets on which AI is trained. The AI generated content is used to train new AI models, and the models are used to generate something original and new. In their creative experiments, the artists compare Generative AI to automatism and automatic writing in Surrealism using a model trained on The Surrealist Manifesto and the Living with Machines ‘atypical animacy’ dataset. Through live performance, they aim to show how we are caught in a perpetual loop in a folded world of multiple intelligences.
Findings and next steps…
The results of the development awards have been made into this film produced by The New Real:
It has been a pleasure to go on a journey with these inspirational artists, and to develop insights together on co-creation and agency in creative AI. The artists fed back to us how they used the tools to train and manipulate a model as a part of their creative process. They reported the platform gave them more granular control and creative agency than they have with other AI tools. They shared insights on how the platform enabled them to work with AI and data in inclusive and responsible ways. Novel features were reported, such as the comment on the results from combining the visualisation tool with Word2vec:
“a really unique output that falls between language modelling and text to image modelling … offering an exciting view into the process and word associations that occur in the high dimensional latent space of machine learning.” – Alice Bucknall
The recipient of the full New Real 2023 Art Commission will be announced in May 2023.
The New Real 2023 Art Commission is a partnership with Scottish AI Alliance – and is supported by The Alan Turing Institute as part of Accelerating AI in Arts and Humanities.
The New Real is a joint initiative of The Alan Turing Institute and University of Edinburgh based within Edinburgh Futures Institute and Edinburgh College of Art.
www.newreal.cc
@newrealcc
