Breaking Language Barriers: Towards Real-Time HDMI Translation for Japanese RPGs

Tesseract OCR + Meta’s MLLB for Online Translation of Screen Text

Unleash the power of multi-lingual translation for your Japanese RPGs. Image created by DALL-E 2.0

Imagine playing a captivating Japanese RPG but facing a language barrier that hinders your immersion. What if you could have a device that translates the text on your screen in real-time? Introducing the HDMI Babelfish — a translation device that allows seamless translation from HDMI input to HDMI output. This blog post will walk you through the initial steps of this process using Optical Character Recognition (OCR) with Tesseract to detect Japanese text on the screen. This text is then translated using Meta’ NLLB and rendered on the screen.

A Japanese Role Playing Game (RPG) successully captured from PS3. Image released under CC BY 4.0

Introduction to the Concept

The idea is to create a device that intercepts the HDMI signal from your gaming console, processes the video frames to detect and translate the text, and outputs the translated text back to your screen. This is achieved using computer vision, OCR, and language translation technologies.

Access to HDMI with video capture will lead to several problems. The PlayStation 3 (PS3)…