Study on English to Igbo Translations: A Comparative Analysis of Human and Various Machine Translation Approaches
In a previous blog post, we introduced a machine translation algorithm designed for low-resource languages: retrieval augmented translation (RAT). This algorithm operates by employing OpenAI’s chat API, which is driven by GPT-4, to convert a sentence from a source language, English, to a target language, Igbo. The chat model was configured, following system instructions, to function as a translator that converts an English query into Igbo.
In this exercise, we construct a translator using a comparable architecture, as depicted in the image below.
First, we generate vector embeddings of the source English sentence intended for translation. Subsequently, utilizing the embedding vector, we conduct a similarity search within a database containing English sentences, their vector embeddings, and their corresponding Igbo translations. The aim is to identify K English sentences that bear the highest semantic similarity to the sentence awaiting translation. Following this, we construct a contextual framework using the K selected English sentences along with their Igbo translations. Ultimately, we feed both the target English sentence and the contextual framework — comprising K semantically associated English sentences and their Igbo translations — into the OpenAI chat model.
The context database comprised 300 pairs of English-Igbo sentences extracted from an English-Igbo translation database on Hugging Face. The test examples comprised 20 target English sentences sourced from the same English-Igbo translation database on Hugging Face. These 20 test examples undergo the aforementioned steps, and subsequently, the results are analyzed.
For the results analysis, the Igbo translations obtained through the aforementioned methodology are labeled as OpenAI Igbo Translation (With Context). To assess the fluency of the translation architecture, we juxtapose the Igbo translations of the 20 English sentences against translations obtained through the following methods:
- Igbo translations of the 20 English sentences found in the Hugging Face dataset, denoted as Hugging Face Igbo Translation in the Results section.
- Igbo translations of the 20 English sentences obtained via Google Translate, referenced as Google Translate in the Results section.
- Igbo translations of the 20 English sentences derived from OpenAI’s chat model without context, labeled as OpenAI Igbo Translation (Without Context) in the Results section.
Results
An Igbo linguist ranked the various translations from the best to the worst, according to their fidelity to the source English sentences. Our proposed retrieval augmented translation (RAT), OpenAI Igbo Translation (With Context), ranked the highest in fluent translations in 11 of 20 test English sentences. In 5 additional examples, it tied for first place with translations that leveraged the OpenAI chat model without context, OpenAI Igbo Translation (Without Context). In 1 example, it tied for first place with the translations gotten from the Hugging Face dataset, Hugging Face Igbo Translation. Overall, our methodology proved to be superior in 17 out of the 20 translation exercises and second-best in additional 2 translation exercises.
Discussion
One hypothesis is that the impressive performance can be attributed to the high quality of the Cohere’s latest multilingual embedding model, embed-multilingual-v3.0. Intuitively, the higher the quality of your context, the better the quality of your translation is expected to be. Generation or translation systems that rely on extracting context from a vector database do so based on similarity metrics. These systems are predicated on the assumption that the embedding algorithm clusters sentences with similar semantics close together in the embedding space. A good embedding model, consequently, allows you to extract context information of better relevance to your query. The easiest way to enhance the the context building process is by increasing the size of the vector database, so providing a larger pool of information to scour for context. Another lever to pull is to increase the number of relevant information used to build the context — increasing K in the K-nearest neighbor search.
Conclusion
In this article, we explored the use of K-nearest neighbor search in building a retrieval augmented translation architecture and demonstrated its superiority compared to human and various machine translation models. A worthy area of future exploration for is finetuning a pretrained chat model using the translation database.
About Us
Chris Ibe and Okezie Okoye, from Hypa AI, spearhead this journey, dedicated to exploring the depths of machine learning’s potential. Their work at the intersection of technology and humanity reflects a commitment to ethical development and the collective well-being of society.
Hypa AI is committed to pioneering advancements in artificial intelligence, with a vision that emphasizes ethical development and the collective well-being of society. Our commitment extends to democratizing access to intelligence, focusing on creating solutions that are multicultural, multilingual, and multimodal, ensuring that AI technologies benefit all of humanity.
