So I was trying to watch a Japanese TV show the other day…
And the subbed version of the show came out a whole 2 weeks after the show came out, with barely understandable subtitles. While it sounds like a trivial first-world problem, it is nonetheless an important and valid one.
Subtitles & Voice-overs Take Wayyyy Too Long
TV Shows made in other countries like anime take weeks to get limited subtitles, and months to get voice-overs. Voiceovers cost hundred for short ads to thousands for medium-length (20 min ~) episodes
Content on YouTube rarely gets made in other languages, and auto-generated subtitles are understandable at best.
However, a staggering 1.445 billion YouTube users are from low or no English proficiency speaking countries, which is over 5 times the amount of English-speaking users!
This means billions of dollars lost in AD revenue, for both creators and YouTube as a company.
In fact, statistics show that youtube videos have a 10% — 50% increase in views when converted into different languages. Now, this depends on a variety of different variables, such as how marketable the video is, how professional the voice-overs are, and vital the voice is to the video.
Take a look at Kurzgesagt for example, a channel focused on creating animated educational content. It’s spanish counterpart came out 2 years after the original, and received 6.7 million views, roughly 25% of the original video
Another example is Mr Beast, who had professional voice actors revoice the links in another language and post it on his channel, with a 15% subscriber count increase, 23 million.
Speaking of professional voiceovers, a thing that AI voices currently lack is lip-syncing, while TikTok shorts can get away with a robotic AI voice, shows and real-life YouTubers can not. Not only does the current AI-generated voice sound automated, it also has no connection with the lip movement of the speaker, making the speaker in question look unnatural and reducing video quality.
Solution
To solve this problem, our company is working on new and innovative AI that can automatically change your media into different languages, in both voice and subtitles.
Our solution can be split into 3 parts:
- Create accurate subtitles — B2C Product — Targeted towards youtube & TV Shows
- Create accurate voice-overs /w emotion + personalized based on video — B2C & B2B — Targeted towards creators & companies
- Create lip-syncing to match /w voice-overOur company is currently at stage 2 of this plan, and you can check out our automatic subtitle generator here:(https://buzzai.vercel.app/). We also have further plans such as:
- Chat GPT Analysis
- Emotional Detection
- Video Enchantment
While we’re not going to go into the nitty-gritty of our product, here’s a brief overview of our product and plans for the future:
- Accurate subtitles
Here we implanted the Whisper API to get the automatic transcript of videos after scraping them off YouTube, along with a couple offeatures, such as analysis, summarization, questions, and downloading.
2. Accurate Voice-overs
To do this, we’re going to use an artificial voice generator. Unlike the previous product which was free, this will be a paid product for creators and studios, and the beta version will be launching soon (check out our website!).
Unlike many generic AI voices nowadays, our voiceovers will be personalized for creators, generating human-like voices filled with emotion, whether that be soothing yoga tutorials, educational animations, action-packed documentaries, romantic animes or even video game dialog.
3. Lip Sync
Finally, we’re going to add lip-syncing on top of the voice-over, using PyTorch GAN models that first detect that face outline, then generates lips to match the audio.
Our Vision
Our goal is that any media is accessible to anyone, in any country, removing any barriers. Where someone from Taiwan can laugh with their friends about that new Mr. Beast episode, at the same time as children from all over the world. Where TV shows come up as soon as they come out.
Where creators can paste their YouTube links into our website…
And Boom!
They now have a video in a different type of language.
