AI Art Profiles: Cody Claus — Retro Diffusion
by Reed Berkowitz
AI Artist Series
Perceptions about AI art are divided today. There are those who consider it a technological break-through and a boon to creativity. Others see it as theft on a grand scale.
Which ever way you look at it, AI art is here to stay and is likely to become inextricably tangled with the future of art as we know it. AI tools are already becoming part of the workflow of artists around the world and this is just the beginning. It’s a Pandora’s box that can never be shut again.
As prevalent as AI has become, the generative AI art industry is very young and there is a growing gap between the reality of how AI art is being generated and how it is perceived.
Is it theft? Is it art? Are the tools complicated enough that the artist can express themselves in a deep an meaningful way? Or is AI just an artistic slot-machine mish-mashing random stolen images into an uninspired collage, with too many fingers?
This series doesn’t attempt to answer any questions, but simply shines a spotlight on what people are doing in their work and practice, and how they are doing it.
I’ll diverge a little and try to look at the larger industry trends and see where some of this might be going.
There are no easy answers. All we can do is learn about what is happening, be informed, be curious, and be adaptable.
(If you are an AI artist and would like to be considered for this series or know someone who you think should be, please let us know here or at https://www.threads.net/@curiouserstream )
Cody Claus
I’m happy to kick off this series with the art of Cody Claus.
Cody Claus is the 22 year old creative entrepreneur behind Astropulse LLC and the generative pixel art AI model Retro Diffusion. In a few short years Retro Diffusion has become the single most popular AI tool for creating pixel-perfect pixel art with over 6K users including many game studios. Cody has over 23 AI models under his belt and a cloud-based service that lets people use his models through a pay as you go web service.
Cody and I first met when I was the Creative Director and Producer of a new generative pixel art game at Webaverse. I chose him as the kick off for this series because working together gave me a unique opportunity to see his process up close and illustrated to me how working with his tools changed the dynamic of the artist/producer relationship.
I feel as if he is a new kind of artist. An artist building AI tools for other artists to make art with. His struggles and successes may be ones that are quite common in the coming years.
Origin Story
Cody didn’t start off attempting to make a hot new AI startup. In fact, he never thought of his AI work as anything other than a way to facilitate his and his friends’ deep interest in pixel art. It started as most of the best AI projects start, as an exploration into the unknown.
After a brief stint in a traditional art school, he was looking for the next thing and turned his attention to what was occurring in the generative AI art scene.
It was the site Artbreeder that first caught his interest. He realized he could use the site as a way of creating reference material for his hand drawn pixel art. Instead of searching Google for an inspiring image, he could just create it directly in Artbreeder, saving him a lot of time and providing him with unlimited creative material. At this point, he was still creating all of his images by hand.
The more he began to explore the world of AI art it became evident to him that pixel art was not well represented in the generative AI space. There were some pixel art models, but he found the results to be disappointing. The artistic quality he aspired to achieve surpassed the capabilities of the existing tools.
In fact, unlike most artists, Cody’s fear wasn’t that the AI would steal his art or copy his style, but that it couldn’t!
He loved pixel art, and wanted to see if the AI could be trained to do it as well as he could.
So he decided to train his own model.
A Model Student
But what does it actually mean to “train your own model”?
Perhaps the biggest misperception is that AI models somehow store every image they are trained on and “mash them together” when a prompt is entered. Models don’t have any images in them at all. Just like a human brain, they are taught to generate images. When they create an image they create it from scratch based on what they have been previously trained on.
Cody started his process by using a “base model”, in this case Stable Diffusion, and a tool called DreamBooth that is used to “teach” the base model how to create art in a more specific style.
The base model is a large AI model trained on billions of images. Stable Diffusion v 1.5 is trained on roughly 2.3 billion images. This creates a model that is large, creative, and general. After looking at billions of images and attempting to create images similar to the ones that are described to it, it learns. It’s similar to showing a child how to draw. The more things it sees, and the more times it attempts to draw those things, and the more times it is corrected, the better it gets at drawing them. There are no images stored in the model. Just the results of the lessons.
The pixel art style is one of the hardest to recreate because no matter how many pixel art images are in the training, they are a drop in the ocean compared to the number of non-pixel art images.
Cody has discovered that his original images were already used to train the base model itself, but this is not nearly enough to get the AI to copy him or his style. Very few artists can evoke the base model into producing something in their style. So not only were Cody’s images lost in a sea of other images, a needle in a haystack, the entire concept of pixel art is drowned out by the sheer volume of pictures that are NOT pixel art.
On top of that, pixel art is extremely rigid, and our eye can see the differences down to the pixel. AI pixel art is often a “style” that the AI feels is “close enough”. The pixels can change shapes and resolutions from one image to the next or even within a single image. And a trained eye can see it immediately.
The base model has to be fine-tuned to be able to do something as strange as pixel art. Fine-tuning gives artists the ability to personalize the output of this general model to do things it couldn’t do before.
If the base model is like a brain, the fine-tune, through DreamBooth is like a lesson. The model might be good at general images, but if you want to create images similar to a specific artist, concept, or style, it is usually lost in the vast amounts of information in the original training data. The fine-tune focuses the model in on a specific item. It functions a little like how you can teach an art student to emulate the style of a new artist. The student has the general intelligence to draw in this style, but it doesn’t have the experience until it is shown many examples of the artist and creates many failed attempts, corrected by a teacher.
In this case, Cody trained the base model by showing it a large number of pixel art images.
Responsibly Trained
Cody did what most amateur AI artists do which is to scrape together a bunch of pixel art and use DreamBooth to create a simple fine-tune. At this point, he had no commercial aspirations and did use random pixel art images he found on the web. He just wanted to see if it was possible.
To his surprise, he discovered that his fine-tune was working quite well.
He realized that with his background in AI and pixel art, he might actually be able to train a model that could compete with, or even surpass, the output of any of the other pixel art models.
In order to do that, he had to figure out how to increase the quality of the model, and he wanted to make sure that the model he trained was not trained on anyone else’s art without their permission.
You know, because he’s a good person.
He ditched the model trained without permission and started on a new model.
For his next series of experiments he decided to train on his own artwork exclusively. Making sure that no art was included in his training data that he did not have express permission to use.
To his surprise, the model’s quality was much higher. Not only because his own artwork raised the bar, but he could describe the images much more precisely since he was the one that created them and this is a huge part of the fine-tuning process.
He also felt that training the model with his own artwork made it feel more personal and grounded. In some ways, training an AI model can feel “dreamlike” or bizarre. It’s not a straight forward process and it’s very hard to see the cause and effects in the training; but when the images are all yours and you are so familiar with them, you can be much more conscious of how and why the models are outputting certain images. You can clearly see why they are pushing into what to other people would be strange directions.
Nothing about AI is easy however. The base model is not good at pixel art, and in order for the fine-tuned model to become good at pixel art, some of its creativity is lost. It has to be “over trained” so that nothing but pixel art comes out. And that means that the creativity of the model itself gets degraded. Soon, the model can’t create things the base model used to be able to do. So Cody had to water-down the pixel art-ness in order to let the model be creative.
Also, the more variety of pixel art that is added into the model, the more creative the model can be and still create pixel art. Realizing he needed more varied quality pixel art, he reached out to his friends and colleagues and asked if they would donate their work to to help him train his model. And they did!
I put the question to him, “Why?! Why would your friends help you train an AI model that could actually take their jobs. That could be trained to do their job better and faster than they could? Shouldn’t they be blocking you, not helping you?”
His friends all had a similar point of view. Especially his friends working as paid artists. This tool was going to help them in their work and ultimately, it was going to make them more money. It already has. Cody and his friends consider it a tool to help them with their artwork and their business.
Outcome
To date retro-diffusion is supported by assets from over 30 consenting pixel artists including Cody. And yes, many of them are paid for their contributions.
It’s a community of people creating, improving, and maintaining the tool they want to use.
And how is it doing so far? Here are some stats from Astropulse LLC
- 6000+ users across all products
- 4.9 star rating
- 23 models for creating pixel art
- Over 25,000 images generated through the website
- Used by big game studios like Halfbrick, and smaller indie studios
- First latent diffusion model to crack pixel perfect generations
- Most popular AI tool for creating pixel-perfect pixel art
- Partnered with the creator of the most popular open source pixel art model (Pixel Art XL)
- Developed huge open source contributions to creating pixel art with AI (https://github.com/Astropulse/pixeldetector, https://github.com/Astropulse/sd-palettize)
- Supported by assets from over 30 consenting pixel artists
- 28 major updates since January 2022
- Zero outside funding or compute, 100% independent and self sufficient
Cody and the team are always updating and communicating with the Discord community.
So far his dedication to his craft and the customers have allowed Cody to make this his full-time job.
Recently he started Retro Diffusion AI, a web based version of his models with a pay as you go no subscription payment plan! Pay for what you actually use. https://www.retrodiffusion.ai/home
Backlash
While Cody’s community has fully supported his efforts, not everyone has felt the same way.
Originally, Retro Diffusion was created to be used as an extension for Aseprite, a popular pixel art tool. Eventually, the developer of Aseprite came out harshly in opposition to Retro Diffusion.
They asked him to remove all association with Aseprite in their marketing, which Astropulse LLC did at a heavy cost to them.
It was a hard blow for Cody emotionally.
“That was pretty devastating. That hurt a lot.
Absolutely crushing blow having one of your most respected developers basically accusing you of bastardizing their product.
Their product is incredible. It’s genuinely the best tool for making pixel art. It just puts me in a little bit of a weird place because they hate my guts and I’m like “you guys are awesome!”
My own opinion is that it’s pretty easy to hold the (falsely, in my opinion) moral high ground when it doesn’t cost anything. It’ll be interesting to see how Aseprite feels about AI art when it’s an industry standard and a majority of studios are using it. Will they still be so outspoken against it? I don’t think so when their ability to work with AI tools will be a requirement for use in a modern game development workflow.
I guess time will tell.
Artists Are The Best AI Handlers
Could models like Retro Diffusion take away jobs from artists? I think it would be naive to say ‘no’. They will. I also think that no matter how many jobs it takes away, AI will continue to grow because people want it.
I started out my career as a pixel artist and game developer so long ago that pixel art was all there was! So this hurts. But there is a big upside.
It will take away jobs, but it will also create jobs.
First, most of the users of models like Retro Diffusion are not professionals. They are hobbyists and indie game developers. Their use is not going to effect job creation. If it does, it’s just going to make games that could not have been made before and make them better and faster.
For instance, making asset packs for Unity is probably going to go away or be done with AI assist. But is it a career or even a job? Not for most people. But making a whole game? Even with tools like Retro Diffusion, we are a long long way away from not having to hire artists. In fact, who is going to run those models?
I’ll even go one step further and say that it will be artists that create, run, and “art direct” internal and external AI tools.
I’ve had the pleasure to work with several great artists who also fine-tune their own models. For instance Minta from Scenario.gg was one of the most brilliant fine-tuners I have had the pleasure to meet.
I also chose to use Kali Yuga’s models which are stellar, I believe because of her artistry and her sensibilities.
I have found over the years that the very best people to do fine-tunes are artists. They have the training to see why a model is failing or succeeding. They have the background to adjust the training set to get the model to learn and can describe what they are seeing accurately enough for the model to become guidable.
After you get the results you need, artists are also the best to run the models, and because they are artists themselves, they can create hand drawn art to either fill-in hard to prompt areas, draw animations, backgrounds, different specific gear, and plug them back into the model making them even better.
Working with Cody even helped our traditional procedural generation because he had the eye to communicate to our programmers how everything was supposed to fit together.
This kind of artist-AI whisperer is a new field. I can’t imagine using a base model right out of the box for any kind of serious game dev work. In fact, any game project of any weight will probably have quite a few custom models all run by artists.
New Uses
Not only will there be new jobs for artists, but in the case of Retro Diffusion, we had a unique use-case, where the game we were creating could not have been done by a traditional artist.
The game we were developing that we first hired Cody for was an ever-evolving AI world in the tradition of Gather.Town and top-down jrpgs. Traditional tile-sets would be used with procedural generation to create different biomes like “a forest”. Then, if the user typed in “happy rainbow forest” those twenty small tiles would be sent to a pixel art model, altered based on the prompt, and sent back as happy rainbow forest trees, grass, and houses. Random new items created for that biome would also be created (rainbow statues, sparkle clouds, etc. based on an LLMs decision) Players could edit tiles by hand, import tiles, or create their own on the fly with different models. In minutes, they could create a place in the metaverse where friends could meet, play, and create their world simply by typing.
This is something a traditional pixel artist can’t do. It doesn’t take away jobs from creators, but lets everyone become a creator. This is a new kind of game-play based on a new kind of technology.
Conclusion
Cody and AI artists like him are on the cusp of a revolution. Will the net effects be positive or negative? Will it take jobs away, or be a net gain that will grow the industry in general?
I don’t know, but stay tuned and learn more as we spotlight more people creating the future of art and technology.
