Revolutionizing Video Editing: Harnessing the Power of Voice Cloning
Heygen and rask.ai have gained popularity again due to the viral videos of Taylor Swift speaking Chinese. The combination of speech translation, lip sync generation, and subtitles has become the new secret to gaining traffic for content creators. However, both rask.ai and HeyGen can only translate videos and cannot make the characters in the video speak freely. How can these characters be manipulated to speak with their own voice in the edited videos?
Recently, in order to clone the voice of a character for video editing, I have tried several solutions, including:
speaking.ai/ eleven labs/ resemble.ai/ Microsoft Azure AI/ XTTS
1. Azure AI: Although Microsoft’s blog has been updated, I have not seen the feature in Azure yet. It claims to “quickly replicate a user’s voice by providing a one-minute voice sample.” Now it only offers text-to-speech, and the operation of Azure’s page is quite complex. Although it is comprehensive, the tutorials and customer service are difficult to use, which seems to be a common issue with Infrastructure as a Service (IAAS).
2. speaking.ai’s voice cloning: Compared to elevenlabs and resemble.ai, it offers a free cloning service. I cloned the voice of Walter White from “Breaking Bad” by uploading a one-minute audio of his voice and then entering the words I wanted him to say. The result was quite good, although it took more than ten minutes to generate.
3. Personally, I believe that the best current solution for voice cloning is XTTS:
https://huggingface.co/spaces/coqui/xtts
The final audio result is here
When generating, remember to select “Clean up Reference Voice” to remove noise. However, it has a character limit of 200, so longer content must be divided into segments.
In my research, I found connections in learning, such as the tools I had used before, like AWS and Hugging Face, as well as principles of speech-to-text, voice cloning, and training.
Passion for AI doesn’t always need help from others, as GPT has made it easier for non-coders to become independent developers.
Buckle up for more AI adventures in upcoming articles!
