Photo by Hans-Peter Gauster on Unsplash

Diffusion Model for Image Generation

I had a theoretical experience with image classification models (like convolutional NN). Today, as I was introducing my wife to MidJourney, Dalle-E and Adobe Firefly, the engineer inside me got curious about how generative AI for image generation is working.

While navigating many articles, I got hooked on Diffusion Model. Initially, when I heard that they are inspired by non-equilibrium thermodynamics, I couldn’t get make intuitive sense. However, after much thought, this chat of mine [I] with ChatGPT [C] helped me to build some level of intuitive comparision. Supplemented the chat with an introduction to non-equilibrium thermodynamics (for the sake of different readers).

I: Explain non-equilibrium thermodynamics in simple words.

C: Imagine you have a cup of hot tea and you leave it on the table. Over time, the tea will cool down until it reaches the same temperature as the room. This is an example of a system moving towards equilibrium, where everything becomes balanced and stable.

Now, non-equilibrium thermodynamics deals with situations where things aren’t balanced or stable. It’s like when you’re blowing up a balloon. The air inside the balloon is at a higher pressure than the air outside, and the rubber of the balloon stretches. This is a system that’s not in balance — there’s a difference in pressure and tension between the inside and outside of the balloon.

Non-equilibrium thermodynamics helps us understand how systems like the balloon change and behave when they’re not in balance. It’s like looking at how the balloon stretches, how the air flows in and out, and how the pressure changes over time. This branch of science helps us understand processes that are ongoing, dynamic, and not just sitting in a stable state.

In essence, non-equilibrium thermodynamics is about understanding how things change and move when they’re not in a comfortable, balanced state like our tea cooling down. It’s like studying the exciting and often unpredictable ways systems react and transform when they’re not at rest.

<In short, making the observation when a system goes in a non-equilibrium state>

I: A kid sees a puzzle. Let’s say in finite step we diffuse it, i.e. add some noise to. Then we ask the kid to now bring back to the original or a target image. After a lot of practice on this exercise, they will learn how to create an image from the noisy image without seeing the initial target image. Is that a better example of a diffusion model for generative images?

C: Absolutely, that’s a great way to illustrate the concept of diffusion models for generative images!

Imagine a kid who is given a puzzle, but instead of the puzzle pieces being in their correct places, they are all mixed up and noisy. The kid’s task is to rearrange the pieces step by step to recreate the original or target image. Over time, through practice and learning, the kid becomes skilled at identifying the relationships between puzzle pieces, figuring out the correct order, and removing the noise. Eventually, they can take a noisy arrangement of pieces and reconstruct the original image without needing to see the target image.

This analogy closely mirrors the way diffusion models for generative images work:

1. Noise and Mixing: The initial noisy image is like a state of high “concentration” of noise. Just like particles diffusing in a system, the noise gradually spreads out as the kid tries to piece the puzzle together.
2. Learning and Reconstruction: As the kid practices, they learn how the puzzle pieces should fit together. Similarly, in diffusion models, the model learns the underlying patterns and features in the data by observing how the noisy image evolves over time.
3. Dynamic Process: The process of rearranging puzzle pieces and refining the noisy image is a dynamic one. Similarly, in diffusion models, the generation of high-quality images is achieved through a step-by-step process that transforms a starting point (noisy image) into a more accurate representation of the target distribution (original image).
4. Non-Equilibrium Behavior: The puzzle-solving process is inherently non-equilibrium. The kid is constantly working to reduce the difference between the noisy arrangement and the target image. Similarly, diffusion models operate in a non-equilibrium manner as they iteratively refine the generated images to match the desired data distribution.

In this analogy, the kid’s ability to reconstruct the original image from a noisy arrangement demonstrates the generative power of diffusion models. These models can learn the intricate relationships and patterns within a dataset, enabling them to generate realistic and high-quality images that resemble the original data distribution, even starting from noisy or incomplete inputs.

Overall, your example beautifully captures the essence of diffusion models for generative images and how they utilize non-equilibrium processes to create coherent and accurate representations.

I: Can you provide another similar example?

C: Certainly! Let’s consider the example of an artist trying to recreate a painting that has been splattered with random paint droplets. The artist’s goal is to remove the random paint splatters and bring the painting back to its original state.

1. Original Painting: Imagine a beautiful painting with vibrant colors and intricate details. This is the “target” image we want to recreate.
2. Random Paint Splatters: Now, imagine that someone has splattered random paint droplets all over the painting, obscuring some parts of it. This represents the “noisy” or altered version of the image.
3. Artist’s Efforts: The artist’s task is to carefully analyze the painting, study its patterns and colors, and then use their artistic skills to recreate the obscured portions. With practice, the artist learns to recognize the original details and accurately repaint the areas that were covered by the paint splatters.
4. Learning and Reconstruction: Over time, the artist becomes adept at transforming the altered painting back into its original form. They no longer need to see the pristine original painting — they’ve learned the patterns and details well enough to recreate them from the noisy version.

In this analogy:

• The original painting represents the “target” image that we want to generate.
• The random paint splatters correspond to the “noise” added to the image.
• The artist’s efforts mirror the iterative process of diffusion models, where the model learns to generate accurate images by gradually reducing the noise and matching the underlying data distribution.

Just like the artist learns to recreate the original painting from the altered version, diffusion models learn to generate high-quality images by iteratively refining noisy inputs. This example helps illustrate how the model doesn’t need direct knowledge of the target image but can still produce images that closely resemble it.

Overall, this analogy highlights the role of learning, practice, and iterative refinement in both the artist’s creative process and the operation of diffusion models in generating realistic images.