Multilingual Mastery: A Comparative Study of AI Embedding Models

Introduction: The Linguistic Odyssey

Embarking on a quest to uncover the multilingual capabilities of AI, we compare three leading embedding models: Vertex AI’s Gecko, Hugging Face’s BERT, and OpenAI’s embeddings. Through the lens of English-Igbo and English-French translations, this study aims to illuminate the prowess of these models in navigating languages with varying resources and complexities.

Igbo and French: Contrasting Linguistic Realities

Igbo, with its rich cultural tapestry, contrasts sharply with the ubiquity of French. The former’s status as a low-resource language offers a unique challenge to AI, while the latter’s pervasive presence online provides a fertile ground for linguistic modeling. This contrast sets the stage for an insightful analysis of AI’s multilingual effectiveness.

The Data: A Look at the Sample Sentences

To give context to the embeddings and subsequent PCA plots, here’s a glimpse at the sentences used in this comparative study:

Table 1: English-Igbo-French Sample Sentences

Each sentence pairs would be analyzed using both the PCA plot in 2 dimensions and the dot product/ cosine similarity. These values represent the geometric similarity between the corresponding English-Igbo and English-French sentence embeddings, with a higher dot product indicating greater similarity.

Vertex AI’s Gecko: The Linguistic Explorer

English-Igbo Analysis: Despite its robust design, Gecko struggled with English-Igbo translations. The PCA visualizations laid bare the semantic distances, underscoring the challenges of deeply understanding Igbo’s nuances.

Figure 1: English-Igbo Embedding with Vertex AI (Gecko)

The dot products for the English-Igbo sentence pairs are as follows:

Table 2: Dot Product Scores for English-Igbo pairs Using Vertex AI

English-French Analysis: Turning to the more familiar terrain of English-French, Gecko showed improved performance. While not perfect, the embeddings were more closely aligned, reflecting French’s high-resource status. However, it still trailed behind OpenAI’s performance, suggesting room for growth in handling well-documented languages.

Figure 2: English-French Embedding with Vertex AI (Gecko)

Below are the dot product values for the English-French sentence pairs, which quantify the geometric similarity between each pair’s embeddings:

Table 3: Dot Product Scores for English-French pairs Using Vertex AI

The table suggests that the English-French sentence pairs have relatively high dot product values, indicating a stronger semantic correlation as captured by the embeddings. This performance is reflective of French’s status as a high-resource language and suggests that Vertex AI’s model is more adept at handling translations between English and French compared to English-Igbo.

Hugging Face’s BERT: The Bilingual Specialist

English-Igbo Analysis: BERT grappled with the complexities of Igbo, its bilingual focus not quite bridging the vast semantic gaps. The PCA plots revealed a dispersed landscape, indicating a need for more extensive Igbo resources and training.

Figure 3: English-Igbo Embedding with Multilingual BERT

For the Hugging Face multilingual BERT model, cosine similarity was used to measure the closeness of the English-Igbo sentence pair embeddings. This metric is particularly useful for this model as it does not output normalized embeddings. Here are the calculated cosine similarity values:

Table 4: Dot Product Scores for English-Igbo pairs Using Multilingual BERT

These values indicate how closely related the sentence pairs are in the vector space, with a value of 1 indicating identical vectors and a value of 0 indicating orthogonal vectors. The cosine similarities suggest that while the model captures some degree of semantic similarity between the English and Igbo sentences, there is considerable room for improvement.

English-French Analysis: In the realm of English-French, BERT fared better, benefiting from the abundant resources available for these languages. However, it was the least performer among the trio, its smaller model size reflecting in its limited capacity to capture the full breadth of semantic similarities.

Figure 4: English-French Embedding with Multilingual BERT

The cosine similarity values for the English-French sentence pairs.

Table 5: Dot Product Scores for English-French pairs Using Multilingual BERT

The results demonstrate a relatively strong semantic correlation for the English-French translations, which is expected given French’s high-resource nature and its strong representation in the training data of many multilingual models. These values confirm the model’s robust performance for well-represented languages.

OpenAI’s Embeddings: The Semantic Powerhouse

English-Igbo Analysis: OpenAI’s embeddings led the pack in English-Igbo translations. The model, likely benefiting from a vast parameter space reminiscent of GPT-3’s or possibly even GPT-4’s scale, offered the closest semantic alignments, showcasing its potential in handling diverse linguistic challenges.

Figure 5: English-Igbo Embedding with OpenAI ADA Model

The OpenAI Ada model provides the following dot product values for the English-Igbo sentence pairs, which indicate better geometric similarity between the embeddings of each pair compared to the other models used:

Table 6: Dot Product Scores for English-Igbo pairs Using Open AI ADA

These dot product values are relatively high, suggesting that the Ada model embeddings captured a substantial degree of similarity between the English and Igbo sentences, which is impressive given Igbo’s status as a low-resource language. This performance reflects the comparatively advanced capabilities of the Ada model in processing and understanding multilingual content.

English-French Analysis: OpenAI continued its strong performance with English-French pairs, emerging as the top performer. Its probable connection to the massive architectures of GPT-3 or GPT-4 provided it with a significant advantage, enabling a nuanced understanding and alignment of translations that surpassed its counterparts.

Figure 6: English-French Embedding with OpenAI ADA Model

Here are the dot product values for the English-French sentence pairs as provided by the OpenAI Ada model:

Table 7: Dot Product Scores for English-French pairs Using Open AI ADA

These dot product values, which are quite high across all sentence pairs, indicate that the Ada model embeddings are capturing a strong semantic relationship between the English and French translations. This underscores the model’s robust performance in translating between English and one of the most well-represented languages on the internet, French, affirming the hypothesis that Ada is well-equipped to handle high-resource languages effectively.

PCA Visualizations: Plotting the Path to Multilingualism

Through PCA, we visualized the challenging terrain of English-Igbo and the relatively smoother landscape of English-French translations. The plots not only demonstrated each model’s performance but also highlighted the inherent disparity in handling low- versus high-resource languages.

Conclusion: Toward a More Inclusive Linguistic Future

This comparative study paints a vivid picture of where AI stands in its quest to understand and translate languages. While notable progress has been made, particularly in well-resourced languages like French, significant gaps remain, especially for languages like Igbo. The insights gleaned from OpenAI’s superior performance point towards the potential of large-scale, advanced models in bridging these gaps. As we look to the future, the development of more inclusive, powerful, and sensitive AI models remains a paramount goal, promising a world where every language finds its voice in the digital chorus.

Future Work

Future work should explore further integration of low-resource languages into AI models, enhancing the representation and accuracy of multilingual translation tasks. This can be achieved through the development of more sophisticated algorithms, larger and more diverse training datasets, and increased computational power. The goal is to build AI systems that understand the full spectrum of human languages and dialects, ensuring that no language community is left behind in the digital revolution.

About the Authors:

Chris Ibe is the Head of AI/ML Research at Hypa AI, a breakthrough startup with a mission to accelerate the advent of AGI (Artificial General Intelligence). Chris is dedicated to exploring the depths of machine learning and its potential to transform our understanding of intelligence.

Okezie Okoye serves as the Chief Research Scientist at Hypa AI. With a keen focus on the harmonious integration of AGI into society, Okezie’s work is at the forefront of ensuring AI’s inclusive and sustainable coexistence with humanity.

Together, they steer Hypa AI towards innovative horizons, aiming to shape a future where technology and human life synergize for the greater good.