The Model is the Art: Pondering the New Paradigm of Generative Art
A new form of art is emerging in the popular consciousness. Not painted on a canvas, not even drawn stroke by stroke on a digital screen, but art that is generated purely by code. This generative art can come in many flavors, such as the modified flow field algorithm used in the famous Fidenza collection. As a machine learning artist I am mostly interested in the category of generative neural networks known as GAN (Generative Adversarial Network). Most recently I used a StyleGAN2-ADA neural network to train a model I call “T3RRA”.
What makes GANs unique are their two sub neural networks, a generator and a discriminator. One creates example images and the other determines if those examples are accurate compared to the dataset provided and gives a score or weight. The generator starts with what is essentially random pixel values and then uses these weights to learn and create better examples. After hours, days, or even weeks of training the example images will closely match the dataset used. The final trained model is like a landscape of possibilities that will fool the discriminator, resulting in endless variations of images that closely resemble yet do not exactly match images from the given dataset.
Much of what is currently popular in this category comes from algorithms such as VQGAN+CLIP. This is a pre-trained machine learning model that generates images based on text you feed it. This is made possible by the combination with CLIP, a neural network that has been trained on over 10 million images paired with text descriptions. For a more in-depth technical description you can read about CLIP here and VQGAN+CLIP here.
Another example of a GAN is Artbreeder, an online tool for generating images from a pre-trained model. With this combination of BigGAN and StyleGAN models you can produce random images, fine tune, and combine them to form oddly realistic combinations of animals, art, architecture, and tons of other categories. You can learn more about Artbreeder here.
All of these publicly available tools use models that are pre-trained. This means the creator or creators of the tool curated a dataset and spent numerous days, if not weeks, training and molding the neural network. In addition they craft the user interface that interacts with the model. Here lies some controversy and a lack of acceptance by some in the world of traditional artists. To be honest, I understand some aspects of the controversy. If anyone can use an existing tool and simply type in some words or combine images to generate highly realistic and visually appealing images, is that really art? This is an open ended question and one that the artistic community is still figuring out. I do feel the annoyance of a painter who spends hours if not days painting one piece of art, only to see a “GAN artist” pump out numerous quality images in the same time frame.
To give some credence to those using these tools, it can take hours to craft the exact set of words for the perfect text to image generation. You can spend hours using the interface in Artbreeder to get exactly the right image. It can also serve as a friendly introduction to the field. On the other hand if 100 different people used the same model, they would all get the same general style or essence in the images they generate. Images from the same model begin to blend together and I believe art collectors in particular will start to notice. This leads me to my main point on machine learning art and generative art in general: the model is the art.
The model is the art.
If someone else used the flow field algorithm from the Fidenza collection to generate an image, it would still be recognized as part of that collection. Despite someone other than Tyler Hobbs modifying the parameters. I think the same can be said for GAN art, although admittedly GAN models are more versatile. One caveat to this assertion is manipulation or composing on top of images generated from these tools. Much like collage or glitch art, if done well, I would consider this a new piece of art separate from the original model. In fact this is how I started my path into machine learning art.
Another way to think about this is as a photography set. Consider one photographer spending days setting up a particular set for a photoshoot where other photographers take shots from different angles and of different portions of the set. The quality of the final photos may depend on other photographers’ ability to find the right angles or the best lighting, but all the photos taken will contain the same essence and general aesthetic. The heart of the artistic vision comes from the photographer who designed the set. For machine learning art the bulk of the work comes from preparing datasets, setting proper training conditions and training the model. The generated images that come from the model all have the same essence and are really just different representations of the model. The model is the art.
If you are in control of the training and resulting model, you can also utilize model manipulations like network bending or blending, and flesh digressions. These techniques have only recently started to be experimented with in artistic applications. This is where things get really interesting and also where my current machine learning art collection and its associated model, T3RRA, comes in.
In part two of this article I will go through the process of how I trained and molded this model as well as the unique techniques and tools I utilized and developed.
Follow me on twitter and join my discord for the latest details on part two as well as the collection auctions.
