The introduction of Neural Style Transfer

Author: Katsura, intern researcher of Corpy&Co., Inc.

Do you like artwork? And do you have your favorite artist, e.g., van Gogh, Christian Riese Lassen, and Banksy?

You might already be aware but I introduce Neural Style Transfer(NST). This field is gathering attention both from academia and industry.

1. What is NST?

NST refers to Style Transfer with Neural Network. Style Transfer is creating artistic imagery by separating and recombining image content and style.

For example, look it.

A is contents image. down-left image is style image. Reference from Gatys et al.(2016)

Gatys et al. succeeded Style Transfer with CNN. Since this was published, the field become prosperous.So I would like to take a closer look at this research of Gatys et al. this time.

2.Algorithm

2.1 Model

Gatys et al. use pre-trained VGG19. And unlike many studies using CNN, the model freezes and optimizes on the image itself.

model of Gatys et al.(2016)

2.2 Loss function

they use two loss function. These are Contents Loss and Style Loss.

Contents Loss

where F is Feature map of generated image, P is feature map of content image.

Style Loss

where F is feature map ,l on F is layer of VGG, and i and j is position of feature map. this is Gram matrix.

where A is Gram-matrix of Style image.

2.3 Optimization

Gatys et al. apply L-BFGS as optimizer to input image. L-BFGS is based on Newton’s method. This is not only the first order gradient such as Adam or SGD, but also optimization using second order gradient.

3.Implementation

I used the Implementation in pytorch tutorial.

And I use the photos of our office as content image and style image.

Content image

Style image

4. Result

I set the number of epoch to 300 and I get the output.

You can see that style color is extracted. And the following is the transition.

5. Conclusion

This time, I use Gatys et al.(2016) algorithm. However, this is some issue.

e.g.)

• speed is slow
• computational cost is high because speed is slow
• in contents loss, because they use only high-level feature map, so lack of detail.
• Gram-matrix is not good at realistic synthesis

Now, to address the issue, there are many proposal. in 2016, Gatys et al. propose Image-Optimisation-Based Online Neural Methods that optimizes image. However, this is slow. So, Many methods for optimizing models are also coming out. Especially GAN is one trend. To address lack of detail, there is also a method to add Laplacian loss. People who are interested are encouraged to see this.

Reference

• Neural Style Transfer: A Review, Jing et al.
• Image Style Transfer Using Convolutional Neural Networks, Gatys et al.