Claude 3 Sonnet 🆚 Gemini 1.5 Pro
Recently Google launched Claude 3 Sonnet model on Google Cloud. To read about how to get started, follow the below blog.
In this blog, we will focus on how “Claude 3 Sonnet” and “Gemini 1.5 Pro” perform in a head-on battle where I will provide both with exact same prompt and check which one performs better.
Disclaimer: While both Claude 3 and Gemini 1.5 Pro achieve similar overall performance, this comparison aims to highlight specific areas where one model might be preferable over the other.
For easy of writing, I will call “Claude 3 Sonnet” as “Claude” and “Gemini 1.5 Pro” as “Gemini” for the context of this blog.
Text Prompts Example 1:
The First thing I tried was giving both models a very simple prompt, and both were able to give answer quite nicely.
Prompt: How to make a Banana Protein Shake in less than 100 words
While the response time on Claude was ~7 sec, the response from Gemini was ~3 secs. Also, the presentaion of response from Gemini is far better than Claude. Giving a nice title to the response, giving the response in a listed format, adds a nice touch to the overall experience.
Text Prompts Example 2:
In this Scenario, I tried to give incorrect spelling to understand if the model is able to pick that up and correct the prompt to give me the desired output. Notice the intentional spelling mistake of “Moana” to “Maona”.
Prompt: Tell me about the movie name Maona and who all starred in the movie
Despite very close response times, the outputs from the two models differed dramatically. While Gemini was able to correct the spelling mistake and get the desired output, Claude could not identify even a small spelling mistake and give response which is near to the query.
Image Prompt Example 1:
In this, I tried to give a single image and a prompt with the image to ask some questions.
Prompt: From the image try to guess the city and weather
In this test, there was a huge difference in time, where Claude is taking ~10 Sec, Gemini is able to get very precise response in just ~2 Secs.
Image Prompt Example 2:
Prompt: what is the name of character and give more details about him/her
In the above Scenario, Despite very close response times, the outputs from the two models differed dramatically. Claude 3 did not give any response, saying that it would be against the privacy, which i do not think is the right. The above shown character was part of a famous movie, giving details about the character is not at all a privacy infringement.
Also this is not specific to Movie characters, Claude is able to predict Famous cartoon characters like Tom and Jerry, Loony Toons, etc. on giving the same prompt as mentioned about, But when given a Anime character, it says that “It would go against respecting individual privacy.”
Image Prompt Example 3:
Image used from YouTube. Please refer to the video to understand the Integration steps.
Prompt: What is this equation for? Solve the problem.
In the above Scenario, Gemini was faster compared to Claude and also the response given by Gemini was quite readable. But Both were not able to solve the problem correctly. Claude made mistake right from the first step, but Gemini was able to perform all the integration part and it failed on the very last step while doing (390625/24+15625/6)
Chat Prompts:
I tried to ask one question and then asked question related to the response from first question.
While Gemini has Capability to maintain chat history and give response accordingly, But Claude does not have this capability.
The response time for each question is between 1 and 2 Secs.
Conclusions:
- Response Time: Gemini is Faster as compared to Claude 3 in all cases.
- Accuracy: Gemini has overall higher accuracy while giving responses.
- Response Quality: Gemini is able to give Precise answers when asked for it. (as shown in Image Prompt Example 1)
- Mathematical Capability: While both were not able to get to the final answer of the double integration problem, Gemini could reach till the last step, before it failed.
- Chat Capability: Capability to have a continues chat with Gemini is a big plus point over Claude.
- Spell Correction with Context: Gemini is able to correct the spelling based by understanding the context from prompt. (as shown in Text Prompt Example 2)
- Markdown Answers: Gemini provides responses with markdown, which gives a good Presentation of the response (as shown in Text Prompt Example 1)
