Evaluating 10M Context Length of Gemini 1.5 : How good is it?
Gemini 1.5 now has the bragging rights of LLM model with maximum context length(10M tokens) possible across all the Large language Models available, surpassing even Anthropic’s Claude 2.1’s 200K Context Window.
But, higher context length does not always means better performance and that too over multiple modalities, considering Gemini 1.5 is a multimodal model. So in order to better understand if that high of a context length, without impacting core multi-modal capabilities, it makes sense to do a deep dive understanding of how the model was evaluated over long context length.
Going over Gemini 1.5’s technical report [1], long-context evaluations can mainly be grouped into two high level categories which are:
Qualitative evaluations
Testing model capabilities on tasks for which no quantitative benchmarks exist, via some of the following interactions
Quantitative evaluations
Evaluations on synthetic and real-world tasks with well defined metrics
a) Perplexity over long sequences:
Models ability to make use of long context length to improve next token prediction was the objective function used.
To evaluate that objective function, negative log-likelihood (NLL) was recorded for tokens at different positions in the input sequences from held-out text (i.e., not used in training). Lower value of NLL implies improved prediction, with higher NLL expected at beginning when there is no context.
As, it can be seen from graph above, Gemini 1.5 improves predictions up to 1M tokens for long-documents and 10M tokens for code.
b) Needle-in-a-haystack evaluation:
Long-context recall tested on recently introduced needle-in-a-haystack evaluation (Kamradt, 2023 [2]) which tests a model’s ability to retrieve a text (i.e., “needle”) inserted at various positions into a sequence (i.e., “haystack”).
These tests are performed for text, audio and video where pieces of text, audio and video were inserted at linearly spaced intervals from beginning to end of context, and model was tasked to identify it.
As it can be inferred from the graph and figures above, Gemini 1.5 Pro compared to GPT-4 Turbo, achieved higher recall at shorter context lengths, and a very small decrease in recall towards 1M.
c) In-context language learning:
In-context learning abilities over long context was evaluated on Machine Translation from One Book (MTOB) benchmark (Tanzer et al., 2023 [3]). MTON measures ability to learn to translate between English and Kalamang which is language with fewer than 200 speakers ,no web presence and fewer learning resources(~ around 250k tokens).
As per human evaluation results above, Gemini 1.5 outperforms GPT-4 Turbo and Claude 2.1 by a big margin.
d) Long-Document Question Answering
For this evaluation, questions were created using the book “Les Misérables” (by Victor Hugo), in order to test the model’s ability to answer them correctly when the entire 1,462 page book (ie., 710K tokens) is provided as input.
In order to evaluate model response, a set of 100 questions were created and a human evaluation was performed following the Attributable to Identified Sources (AIS) protocol (Rashkin et al., 2021 [4])
AutoAIS is a LLM metric, which is used to assess the factual accuracy of generated responses by checking their alignment with the source material. As per both the automated and human evaluation, Gemini 1.5 outperformed Claude 2.1, in retrieving relevant answer from the longer context text provided as input.
e) Long- Context Audio
Without the added complexity of extra input segmentation and pre-processing, Gemini 1.5 Pro can transcribe 15-minute videos more accurately than other models, achieving a WER of 5.6%.
f) Long-context Video QA
Gemini 1.5 Pro achieves SotA accuracy over GPT-4V for this task.
Conclusion
With the evaluations as mentioned across various modalities, it can be concluded that Gemini 1.5 do perform significantly well over longer context as compared to other comparable longer context models across different tasks.
However , given the limitations of existing benchmarks across both length and complexity, and human labeling being costly and time-intensive, there is a limitation of how those evaluations closely reflect how good Gemini 1.5 is for longer context tasks. So, that creates a need to have more sophisticated evaluation benchmarks, datasets and methods for proving out how good the 10M context length of Gemini 1.5 is and how that translates to provide better value and performance for LLM applications across various use cases.
References:
[1] Gemini 1.5 Pro technical report (Gemini1.5:Unlocking multimodal understanding across millions of tokens of context)
[2] https://github.com/gkamradt/LLMTest_NeedleInAHaystack/blob/main/README.md.
[3] A Benchmark for Learning to Translate a New Language from One Grammar Book by Tanzer et al. arXiv:2309.16575
[4] Measuring attribution in natural language generation models by Raskin et al..CoRR,abs/2112.12870