How Does Normalization Layer Affect Image Styles? What is AdaIN?
In Style Transfer and some generative networks, special normalization layers such as CIN, AdaIN are used to control the style of the output image. But how and why does a normalization layer affect image style?
It is crucial to understand the rationale behind it in order to know why they are integrated into many modern neural network models.
Gram Matrix
The story starts with the first Neural Style Transfer paper — A Neural Algorithm of Artistic Style by Leon Gatys et al. published in 2015.
In Neural Style Transfer, the style loss is calculated as the distance between the Gram matrices of some CNN layer outputs of the target style image and your generated stylized image.
It is found that by using the Gram matrix, which computes the correlations between different filter responses from a CNN, we can extract the style representation of an image.
The following diagram shows how the Gram matrix is computed for one of the CNN layer outputs: conv1_1.
