Week 3 — Image Transformation according to Art Style
Hi everyone, we have done specific research and some implementation stages. In this article, we will talk about the basic features of the gan model, differences between gan and cycle-gan, and the basics of TensorFlow.
What is the GAN?
If you give the GAN model enough samples of the data you want to produce, you can have new samples produced with the same data type. If you give thousands of horse photos to the system, the system will learn what a horse should look like and will generate new horse photos or translate another photo to horse photos. There are 2 different artificial neural networks in GANs that conflict with each other. These are called Generator and Discriminator networks. While the generative network generates new data (images, models, etc.) that look like reality, the Distinctive network tries to distinguish between fake and real data. As these two neural networks clash with each other, the Distinctive network gradually begins to better distinguish between real and fake images. The producer network produces more realistic fake images. We can compare this scheme to the counterfeit and the police duo. As time goes on, the police are better at catching counterfeit coins to find counterfeiters. The counterfeiter starts producing more realistic counterfeit coins to beat the police.
Cycle-GAN
The system first converts the picture to the desired content using 2 nested GANs and then converts it back to its original state. A loss is calculated to show how well the result fits into the new content and another loss that shows how similar it is to the original. Later, these losses are reduced. Thanks to this cycle, paintings that are very similar to the original form and that have achieved the desired transformation are produced. For example, by looping a horse -> zebra -> horse, one can find out what a horse would look like if it were a zebra. In this work, we use a pix2pix based cycle-gan model.
Implementing the Cycle-GAN
Many frameworks can be used in the implementation of CycleGAN like Keras, Pytorch and Tensorflow. In our implementation part, we have used the Tensorflow framework because of easy to use.
For our first style transformation, we have imported Van Gogh datasets and real image datasets. The real images mostly include landscape images. Since we fed the discriminator model with these landscape images, the discriminator made a dedication with real images and Van Gogh images. So, the discriminators give information to generate Van Gogh style on the learned real images.
We used random jittering and mirroring as image augmentation. The importance of image augmentation is, generate more train set and decrease the overfitting.
As a generator model, we have used a U-net generator. U-net is a special type of architecture for image segmentation purposes. That means the arrangement of deep learning tools resulted in image segmentation. By using UNet we can do image localisation by predicting the image pixel by pixel.
We initialized 2 generators and 2 discriminators. The first pair is responsible for transforming a real photo into Van Gogh style paints, and the second pair is responsible for transforming Van Gogh images into real photos.
After each generation and discrimination, we have used cycle consistency loss. The cycle consistency is calculated differences between real photo as input and transformation of generated Van Gogh images by using the input. After, we have used these loss values in the calculation of the gradient.
We have used Adam optimizer as stochastic gradient descent. The Adam optimizer calculates the weights for the discriminator and generator with gradient values.
On our first trying, we didn’t use human images as real photo. For his reason, while the model is transforming trees, buildings and skies very well, it isn’t successful to transform humans as much. In our purpose is to feed more real photo to helps the discriminator in the discrimination of real image and arts. See you next week!
References:
- https://developers.google.com/machine-learning/gan/gan_structurehttps://www.tensorflow.org/tutorials/generative/cyclegan
- https://towardsdatascience.com/cyclegan-learning-to-translate-images-without-paired-training-data-5b4e93862c8d
- https://www.tensorflow.org/tutorials/generative/cyclegan
Contributors:
