Understanding LoRA Training, Part 2: Offset Noise, Epochs and Repeats

Let’s talk about Offset Noise, which is supposed to be quite a crucial part of the training process.

Ever tried training a Stable Diffusion model against a single solid black image? Chances are you won’t get anything much better than this:

This image comes from the key article about Offset Noise.

It seems like not only does Stable Diffusion not have the ability to produce overly dark or light images out of the box, but it also can’t even learn to do it.

It’s a fun read!

If you prefer a visual explanation, here is koiboi’s version, amazing as always:

Remember how I said in Part 1 that a weak handshake between LoRA and the source model results in vivid colors? Well, I wasn’t entirely wrong there, but the reason for this color shift always happening early in training is not a ‘weak handshake’. It’s the issue that authors of Offset Noise algorithm are trying to solve.

Over the course of the training the brightness range of a LoRA is getting closer and closer to the average brightness of the dataset. That’s the idea of training and a good thing, of course, unless you want to generate low key or high key images.

Let’s try using black squares as class images and see what happens over the course of 18 epochs. I will be using Cosine with 18 restarts as LR scheduler.

Before we start, this is what I had with native class images (see Part 1 for details and settings):

Now let’s change class images to black squares.

Sample prompt is the same:
dfw woman, headshot, sunflower field

The brightness range has clearly extended. I got brighter highlights and darker shadows, but the quality of character training predictably went down. This was done with 10 repeats of 14 training images and 140 repeats of 1 black square class image to get the same number of epochs as in the first test.

If I had 14 images with no repeats per epoch (which is more logical) and 14 repeats of the black square to balance the training dataset, this what I’d get over 18 epochs (only 252 steps):

• More on epochs and repeats in a separate section below!

To retain the training quality and brightness range, I guess I can add ‘low key’ and ‘high key’ to the class images prompts. It’s such an obvious trick that there is no need to test the results.

For now, let’s try a different sample prompt:
dfw woman, headshot, sunflower field at night

This doesn’t really look like night, so yeah, it’s a clear example of the average brightness issue.

Let’s add Offset Noise 0.12:

The night is here! Although it looks like the training set was affected in more ways than the average brightness change.

Let’s try Multi-Resolution Noise at 8 iterations and 0.45 discount (a bit of an overkill) instead of the original Offset Noise:

Okay, I got it. I hope you did too!

There is probably a solid middle ground for noise settings, but I can’t say that I’m a fan of the approach.

It’s good to be aware of the average brightness issue, especially if you are training a low (or high) key style, but I’m not sure adding the noise is worth sacrificing the textures quality during the training.

I’d rather be vigilant of my dataset brightness for both training and class images. And I can always fix the brightness on the generation stage too.

Epochs and repeats

It seems that there is a lot of confusion around kohya-ss folder structure, repeats and epochs. As kohya-ss explained here, there are two reasons for repeats:

1. Balancing the number of training images with class (reg) images. Usually there are more class images than training images, so it is required to repeat training images to use all regularization images in the epoch.

• For instance, if you have 10 training images and 100 class images, you repeat them 10 times (10x10 = 100), to compare them to all class images within each epoch.

2. Control ‘weight’ over folders. If you have a mix of high quality images and low quality images, you can set higher number of repeats for high quality images and lower for low quality.

• It doesn’t make sense to repeat training images more times than needed for balance or weight within one epoch, otherwise you lose the flexibility of using epochs in your training.

Kohya-ss scripts default settings (like 40 repeats for the training dataset or Network Alpha at 1) are not ideal for everyone. Most of these settings are at the very low values to avoid issue reports from users with weaker systems. Sure enough, it makes sense to fine-tune them to your hardware and training needs.

Make sure to check if you really need AdamW8bit set as optimizer, take a look at Precision settings, the number of CPU threads and everything else. You might save yourself days of training in the long run by speeding things up.