DeepFakes AI — Improved Lip Sync Animations With Wav2Lip
Overview of the paper ”A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild” by K R Prajwal et al.
A few months back, I shared a very exciting paper for automated generation of lip animations using an AI based technique called LipGAN. My experiments on certain games with the pre-trained model of LipGAN showed some really impressive results which closely matched with the game’s inbuilt animations, but only took a fraction of the efforts in doing so.
Fast forward a few months, and we already have an improved version of this AI coming from the same group of researchers. The paper is titled “A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild”. Check out some of the results from this paper in the video embedded below.
It seems the major improvement in Wav2Lip over LipGAN is that of a better and more robust lip-sync Discriminator. If you recall, this is the Discriminator that has the task of identifying if the generated lip movements match the audio input for a particular frame. In case of Wav2Lip, this is now extended to multiple frames so that there is a temporal context in the mix when deciding how accurate the generated lip movement is.
Moreover, there is now a pre-trained lip-sync expert Discriminator whose weights are frozen during training of this model so that it does not get affected by the visual artifacts of the Generator and can focus solely on the correctness of the generated lip movements. There is an additional loss in the mix here. A visual quality loss is added to ensure that the overall face in the output frame looks real, thereby minimizing artifacts that were noticed previously in LipGAN.
There’s an interactive demo available for now on the authors’ project page if you want to go and play with this AI now. It is really impressive work so I do recommend you check out this demo!
Thank you for reading. If you liked this article, you may follow more of my work on Medium, GitHub, or subscribe to my YouTube channel.
