CycleGan for Season Transfer

A GAN — short for Generative Adversarial Networks — have been used in a wide array of applications ranging from fashion, to face aging, object visualizations and art transfer to name a few. CycleGan is one such network which can be used for unpaired image translations.

Image translation typically involves paired images. One in Domain A and one in Domain B for each image. Now what happens if we do not have such paired images? What if we have a set of photographs of Yosemite during the summer and another random bunch of photographs of Yosemite during the winter? Is it possible to translate the summer pics to winter and vice versa? Turns out that Cycle GAN allows us to do this.

The architecture of the cycle gan is based up on the encoder — decoder structure. The idea of a cycle gan works on the basis that if we translate an image from domain A to domain B, and then apply a reverse translation on the generated image back to domain A, the cycle should generate something as close to the original as possible. This loss (between the original image and the reconstructed original image) is termed as the cycle consistency loss.

The GAN loss function in Cycle GAN is modeled on the basis of the original work by Ian Goodfellow and co in the original paper as an adverserial loss. The cycle GAN paper also mentions about adding an identity mapping to preserve the original image color in the generated output. The GAN loss comprises of the discriminator loss and the generator loss functions. As mentioned in the Cycle Gan paper:

L_GAN(G;D(Y) ;X; Y ) = E[logD(y)]+ E[log(1 - D(G(x))]

L_GAN(F;D(X) ;Y; X ) = E[logD(x)]+ E[log(1 — D(F(x))]

L_CYC(G,F) = |F(G(x))-x| + |G(F(y))-y|

L_EYE(G,F) = |F(x)-x| + |G(y)-y|

The entire loss function of the model is modeled as:

L = L_GAN(G) + L_GAN(F) + lambda * L_CYC + 0.5*lambda*L_eye

Here are some of the best results on my effort in implementing the model:

Summer to winter: Training results —

Winter to Summer: Training results —

Summer to Winter: Testing results —

Summer To Winter — Failures:

Winter To Summer — Failures:

The link to the original author’s work and results are here. The model was trained for 100 epochs with random sampling of the training images.

The interesting aspect of the results were that the testing results on the winter to summer translation were not complete in quiet a few of the testing images. Some of the examples are below:

May be more training would improve the results of the second generator in the model.

The next step in this direction would be to train the model for the next 100 epochs with a training decay as mentioned in the paper. I would also love to explore a system where the learning rate on the discriminator is slightly lower in comparison to that of the generator to see if this facilitates better conversion from the generators. Further avenues of exploration would be to reduce the weight on the identity mapping (since seasonal transfer would mean a complete change in color of the environment and the identity mapping was predominantly added in the original paper for painting to photo mapping).

P.S: The data set used to train the above model was obtained from here. The code used to train the model is on my git page here. I learnt a lot about implementing the Cycle GAN through the machine learning mastery page here.