[ Archived Post ] SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis

Please note that this post is for my own educational purpose.

Create a realistic image from sketches → this is a GAN approach → as well as can be used as a data augmentation method.

Drawing a sketch → can be easier from time to time → do not need a camera → but not really realistic → it would be a good thing if we can create realistic images from sketches. (there are already some methods that do this → mostly on image query options). Deep CNN → and GAN → can have a lot of potentials. (sketch based image generation). (image synthesis approach). (previous work → does not deal with sketches). (also new block for a neural network is proposed). (the author's methods → very realistic → and new data augmentation technique).

The difference between the edge map and image sketches → we can see that the sketch map does not contain any background images. (most of the related works → involve some kind of image retrieval → and there are some limitation about this methods). (and other works involve creating a dataset of sketch images). (image to image translation → this is a huge work).

And we can see that for each step → the image more looks like a sketch image. (very good way of data augmentation). (learning the mapping between edge map to sketches need a lot of data).

Generated image of a bee → all of the five images are plausible looking. (so this one example of how to create more data → but quite a simple data is what we need to thank for).

The network architecture → and how each MRU is composed of. (this is how the sketchy GAN looks like and we can see some generated images below. (MRU → have two inputs → feature map and image → output a feature map → similar to GRU but not exactly the same).

The problem is → there are some artifacts that might not be too realistic. (the author's method → gives the cleanest result).

The author's methods → give higher inception score. Finally, the objective function → a lot of work has put into creating the objective function → very interesting result.

MRU → gives more emphasis on the object of the sketch.

Adam optimizer was used → and the inception score was used as a metric → additionally perceptual study was conducted.

So not the best overall → but still very good results. (Pix2Pix model itself seems to be a good model). (sketch to image → seems to be a more difficult task → but pix2pix coupled with augmented data → gives much better results).

And all of the loss is important for the model to create the most realistic images. (every component of the loss is needed → class information helps a lot as well). (even human prefer the author's mode’s results).

This work → sketch to image synthesis → also performed data augmentation for creating more sketch-like images. (and MRU have been created for generating more realistic images).

https://jaedukseo.me I love to make my own notes my guy, let's get LIT with KNOWLEDGE in my GARAGE