Using Deep Learning DC-GAN to add featured effect on anything.
Recently I started the Creative Applications of Deep Learning with Google’s Tensorflow of Parag K. Mital at Kadenze, Inc.
After my final project submission and earning a Certificate of Accomplishment in the course I just want to share with you what I did. May be this could help someone well understand and use DCGAN.
For this project I chose to create an application to wear eyeglasses or hats to people without glasses or hats, using DCGAN (Deep Convolutional Generative Adversarial Networks) and hat or/and eyeglass vectors through the VGG model network we used during the course.
DC-GAN uses AutoEncoder (AE) and GAN (Generative Adversarial Networks) to generate a featured output according to the input you fit in it. The original research paper we use during the course is here for more detail.
Below a representation of the process.
So to solve my problem I have to well understand these crucial steps :
1- Select the suitable input dataset (x) to encode, as it will be useful for good feature creation. I give more detail in the notebook on my github.
2- The encoded dataset (z) will allow us to create featured vectors (glass vector and hat vector) which will be applied to the decoder/generator to produce generated vectors (generated glass vector and generated hat vector : the x tilde).
3- Then the generated featured vectors (x tilde) will be combined to the input (x) through the discriminator to produce the desired effect (the output).
4- Once the generated vectors (generated glass vector and generated hat vector) are created we could apply any other input data (the second x) on and compare the effect of generated vectors on using arithmetics.
The part of the network in charge of the featured vectors creation is the variational encoder.
So the GAN part of the network starts from the encoded data (z) and ends with the final result which should be the desired effect (generated output).
The last thing I have to say is that you have to fine tune parameters [linspace (Linearly spaced vector), number of iteration, the input data used to produce vectors, etc.] to perform best results. And this could take a long time, tests and errors according to what you want as result.
This below is what I got when applied to a set of 100 face images of the Celeb dataset.
And below with only one face of the Celeb dataset. You can distinguish the original face image, the generated ones through the linspace and the final glass vector effect on.
You can have all details in the notebook tutorial on my github where I tried to be as clear as possible in the implementation to help newbies understand all HERE.
So if you have time, please test it on anything [music, video (videos are also a sequence of images), etc.] and share it with others via the github.
With my method, all the secret is in the featured vector creation using the variational encoder network part and after comes the parameters tuning. So, for example if you pass time fine tune a dataset of hats or any other feature you could create something very interesting. You could for instance improve the hat vector using a dataset of more big hats, with another color, etc. You could also work on the parameters to produce generated images more close to the original image, etc.
So please, try and share with others.
You could find another post where Melissa Runfeldt uses DC-GAN to remove glasses from faces HERE.
And also the post of Chanchana Sornsoontorn on “How do GANs intuitively work?”.
And the post of Julien Despois on “Autoencoders — Deep Learning bits #1”.
Thanks for reading this. If you liked this, click the 💚 below so other people will see this here on Medium.
