Fine-Tuning a Generative Model on My Art: an Experiment with DreamBooth
Introduction
As an AI Developer and Artist, I’m always thrilled to find new ways of combining and mixing these two realms. Lately, with the introduction of text-to-image models and the exponential growth of the generative AI sector, the possibilities to experiment are almost endless.
Integrating AI in the creative process
Personally, I often use generative AI during my creative process and not as the end result of it. I think text-to-image models are great tools to make the whole artistic process smoother and faster. In fact, they’re really useful for testing different color combinations, for finding inspiration, for creating reference images and prototypes that play with patterns and concepts in unexpected ways.
I wanted to take this one step further and an idea struck me: what if, instead of utilising general models in my process, I fine-tuned a generative neural network on my own art and then used it instead?
I was also curious about which features about my creations the network would pick up and reproduce in its outputs. This would mean identifying some key characteristics that remain constant across my whole body of work and that are distinctive of my own personal style. The ultimate goal was not only gaining a tool for generating references but also better understanding my artistic persona.
Fine-Tuning a generative model on my art with DreamBooth
I decided to use DreamBooth [1]-[2], i.e. a method to personalize text-to-image models like Stable Diffusion by giving them just a few example images (even just 3–5 are enough!). It allows us to create our own model variant with some extra knowledge of a specific face, object or style (more generally, a chosen “subject”). [3]
Given as input just a few images of this subject, a pretrained text-to-image model is fine-tuned so that it learns to bind a unique identifier with that specific subject. Once the subject is embedded in the output domain of the model, the unique identifier can be used to synthesize novel images of the same subject, but in different scenes. It allows the model to generate contextualized images of the subject in different scenes, poses, and views, all while preserving the subject’s key features.
HuggingFace has a complete, ready to use, colab notebook guide that take you through the steps necessary for this process (you can find it here) and the website of Keras also offers a tutorial about it.
In my case, the subject would be my artistic style, so I fine-tuned the pretrained stable diffusion v2 model by feeding my whole body of creative works to it, so that it would learn its key features. I chose the “sks sarasixti style” (sarasixti is my nickname) as the unique identifier of the subject: I’ll use this phrase in the prompts too. These are some images I used for fine-tuning (you can see all my works on my website):
And, finally, these are the images that the personalized DreamBooth model generated and their relative prompts:
Conclusion
This experiment was really inspiring. Based on these results, the key features about my style according to DreamBooth are the following:
- Saturated and strong colors
- Expressionistic (and not at all realistic) style, sometimes almost similar to Surrealism and Abstractism
- The objects being painted are the protagonists of the scenes: the backgrounds usually just contain colors or patterns
It’s really cool that the images generated by the fine-tuned model show a consistent style. I also agree that these characteristics define my style, and I’m honestly surprised that the neural network recognized them this clearly and made them evident in its outputs.
[1] “DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation” https://arxiv.org/abs/2208.12242
[2] DreamBooth Github repository https://dreambooth.github.io/
[3] HuggingFace page on DreamBooth https://huggingface.co/docs/diffusers/training/dreambooth
