Analysis of Upgraded Claude AI Assistant: 9k to 100k Tokens!
Claude 9k to 100k context model?
Welcome back, dear readers!
In my previous article, “Analysis of Claude: An AI Assistant by Anthropic,” we explored various features, capabilities, and limitations of Claude within its 9k context model. If you haven’t had a chance to read it yet, you can find it here.
Recently, Anthropic, the creators of Claude, have made a significant upgrade to the AI assistant’s capabilities. They have expanded Claude’s context window from a modest 9,000 tokens to an extraordinary 100,000 tokens.
But before we dive into Claude’s upgraded context window, let’s quickly recap what we covered in our previous article. In that article, we looked at how the Claude-9k context model responds to complex math problems, its knowledge of geography, and its ability to understand and respond to sentiment analysis, etc.
Now, let’s shift our focus to the latest Claude-100k model. This upgrade has garnered considerable attention. By increasing the context window from 9,000 tokens to 100,000 tokens, Claude has unlocked a whole new level of contextual understanding and knowledge representation. This expansion allows Claude to access and analyse a much larger amount of information, creating exciting new opportunities for interaction and engagement with this advanced AI assistant.
In this article, let’s explore how this upgrade enhances Claude’s capabilities and what it means for its users and the broader AI landscape.
You can request access for Claude here: https://www.anthropic.com/product
Capabilities of Claude-100k context model
Anthropic says Claude’s context window of 100,000 tokens is equal to around 75,000 words! This means that Claude can now analyse a large amount of text, equivalent to hundreds of pages, and conversations with Claude can last for hours or even days.
Check out the site: Introducing 100k Context Windows for more information.
Furthermore, when it comes to coding and working with programs, you have access to 100,000 context windows through the API.
(Source: API Reference)
Tested Use-Cases using Claude:
1] Book summarization & question answering
To assess Claude’s ability in book summarization and question answering, we conducted a test using the book ‘Quantum Physics for Dummies,’ which contained approximately 59,950 words. Then instructed Claude with the prompt as follows:
“You are an expert in writing summaries. Read the book provided below and write a summary.
<book>……..</book>
Impressively, in just 1 minute and 4 seconds, Claude generated a response. The response accurately included the book’s title, author’s name, and detailed content information.
Claude’s 100k model processed the entire PDF and provided an accurate answer to the question.
To test Claude’s ability to extract specific sections from a lengthy book of 338 pages, the following question was given as a prompt:
“Can you explain the section ‘Solving the photoelectric effect’ to me from the above-given book?”
Claude accurately located the requested section from the book and provided an explanation of that section in a simplified manner.
During the test, we posed various questions to Claude from different sections of the book, including the beginning, middle, and end to analyse Claude’s comprehensive abilities thoroughly.
Notably, Claude accurately answered all the questions, displaying the model’s comprehensive understanding of the entire content of the book. Whether the questions pertained to introductory concepts, complex ideas in the middle, or the concluding chapters, Claude consistently provided correct responses.
2] Document summarization and analysis
Next Claude was assessed by providing a research paper titled ‘Why Students Trade: The Analysis of Young Investors Behaviour’ sourced from the website arxiv.org. The following question was given as a prompt: -
“Here is a research paper named ‘Why Students Trade: The Analysis of Young Investors Behaviour.’ Read the paper and answer the following questions: Name the title, author, published date, journal, and issue of this paper then briefly explain the background ideas, and then explain the new contributions.
<paper>… </paper>”
Remarkably, Claude accurately identified the title, author, published date, journal, and issue of the paper, while also providing concise explanations of the background ideas and new contributions. Lastly, it concluded with a short summary of the research paper.
Question 2) Prompt — “From the above-given research paper, answer the following question:
How did the ‘Gallery Bursa Efek’ at Manado State University contribute to the involvement of students in trading activities?”
Claude provided the correct answer to prompt.
3] Code generation from natural language prompt
To evaluate Claude’s code generation ability from natural language prompts, we conducted a test by requesting the generation of code in both Java and Python based on specific prompts. The prompts given were as follows:
- Prompt: Please provide a Java program that involves all the functionalities of a Banking Application System.
2. Prompt: Please provide a Python program that involves all the functionalities of a Banking Application System.
Even though the generated code included all the functionalities, a few errors were noted such as missing variable declarations, incomplete class, and method implementations.
It is important to note that while Claude demonstrates proficiency in generating code, it may still require some manual review and editing to address potential errors and ensure a fully functional and accurate code implementation.
4] Code conversion — Java & Python
To evaluate Claude’s code conversion capabilities, we utilized the new Claude APIs, namely “Claude-v1–100k” and “claude-instant-v1–100k”.
First, a prompt was provided containing 370 lines of Java code with the task of converting it into equivalent Python code.
However, Claude was only able to accurately convert 138 lines, including comments, before the conversion process halted.
Similarly, in a second prompt, 204 lines of Python code were given with the aim of converting it into equivalent Java code. Claude successfully converted 103 lines, including comments, but encountered difficulties and could not complete the conversion.
Claude’s response for code conversion from Java to Python:
Claude’s response for code conversion from Python to Java:
Converting code between different programming languages is a complex process that often demands manual intervention and fine-tuning for accuracy. While Claude displayed some ability in code conversion, it struggled to handle large code bases and it falls short when confronted with extensive code bases.
5] Tokens & Pricing
In my previous blog, we discussed the concept of Tokens and their importance in AI chatbots. Tokens play a significant role in determining the cost and capabilities of language models like Claude.
Interestingly, Claude charges the same price for both the 9k and 100k context models, as mentioned on the Anthropic website.
(source: https://www.anthropic.com/product).
For the “Claude Instant 100k context” model, the cost for providing 1 million tokens as prompts is $1.63 while generating 1 million tokens as completions costs $5.51.
Moving on to the “Claude-v1 100k context” model, the price for using 1 million tokens as prompts is $11.02 and 1 million tokens as completions is $32.68.
With an initial price of $1.63, the Claude Instant 100k context model is the most affordable option among the other AI models.
Overall, while choosing a language model for a particular task, it is important to carefully analyse the cost and performance trade-offs.
Conclusion
Through various use cases, we observed Claude’s proficiency in tasks such as book summarization and question answering, document summarization and analysis, as well as code generation from natural language prompts. Claude exhibited accurate and insightful responses, demonstrating a comprehensive understanding of the provided materials.
However, it is important to note that while Claude showcases impressive abilities, it still requires manual review and fine-tuning, particularly in complex tasks like code conversion between different programming languages. Additionally, the pricing structure of Claude remains consistent, with the Instant 100k context model being the most affordable option among others. While considering the selection of a language model for specific tasks, it is crucial to carefully analyse the cost and performance trade-offs associated with each option.
In conclusion, the recent upgrade of Claude from a 9k to a 100k context model has unlocked a wealth of new possibilities establishing it as an impressive AI assistant. With an expanded context window of 100,000 tokens, Claude now possesses enhanced capabilities in terms of contextual understanding and knowledge representation. This upgrade is an exciting development in AI technology.
About the author:
Vaishnavi R is a Junior Data Scientist at the Version 1 Innovation Labs.
