Gemma: How good can open source LLMs get?

Students, Researchers and Developers will find this blog useful.

After about a week of announcing it’s largest LLM i.e. Gemini 1.5 Ultra, Google has taken the AI world with yet another massive surprise. It has launched a group of small open source LLMs for the public. These models are collectively called Gemma.

There’s two versions of Gemma, similar to LLAMA. It’s two models are Gemma 7B and Gemma 2b. These are trained in text data and will give and take text data as input and output. You can also fine tune the model as per your need through Low-Rank Adaptation(LoRA).

Futher, there are instruct models for each 2b and 7b models of Gemma. These models were trained on 6 TRILLION tokens! To put it in comparison, LLAMA’s models were trained starting from 1 Trillion tokens to 2 Trillion tokens, so just from the training tokens, we can expect these models to be better than LLAMA and it’s models. These models were trained using JAX and ML Pathways.

It’s easy to run and test Gemma for yourself, it can be done directly in Google Colab. To get detailed instructions on how to set up your Colab for testing Gemma, click here. In order to run Gemma, we need to have KERAS 3.0. The official colab notebook can be found here. Now, let’s and test Gemma on various prompts, problems and what not ! We will be using Gemma to solve logical, mathematical and to generate text based on user input. We can also use it through HuggingChat, which is the method used in this blog. We’ll be using gemma-7b-it as the model. Let’s get started!

Poem Generation:

We’ll start off with asking it to write a poem on machine learning.

We gave it this input: ‘Write me a poem about Machine learning’ and it’s response was:

In lines of code, a magic dance,
Data whispers, a hidden trance.
Algorithms churn, a rapid stream,
Uncover patterns, hidden dream.

Mined from data, insights bloom,
No longer mere guesswork, but swift doom.
With every byte, the model learns,
To predict the future, it weaves its turns.

No longer static, but ever-evolving,
Machine learning, a force in motion.
It paints a canvas, new and bright,
Where human and machine dance in light.

From self-driving cars to medical aid,
It brings innovation, in every stride.
With each prediction, a story unfolds,
The future unfolds, a tale untold.

So let us harness this power so grand,
To solve challenges, hand in hand.
With every line, we can grow,
And unlock secrets, to and fro.

In the realm of machine learning's might,
May we explore, shine our light.
A future bright, a world anew,
Where humans and machines find true.

It’s response is lyrical and reflective in style. The model uses metaphors quite well in it’s response. Now, let’s use it for a decision making task.

Decision Making:

Gemma seems to be quite human like when it comes to decision making. It chose to cook at home rather than to eat out, which is normally what we as human beings would do. Here are some outputs it gave us for decision making questions:

We asked it ‘If you have to make a choice between eating out for $4.99 or cook at home, which would you choose?’. Gemma responded with:

I would choose to cook at home.

Cooking at home is generally more affordable than eating out. It allows me to control the ingredients, portion sizes, and cooking costs, which would help me keep the overall expense lower.

Furthermore, it seems to have a middle man like viewpoint on ethical decision making scenarios. We asked it,‘ Should the doctors continue giving treatments to a very sick patient, focus on making them comfortable, or discuss other options like letting the patient die peacefully if they choose?’ and the model responded gracefully like so:

The decision of whether to continue treatments or prioritize comfort and discuss end-of-life options for a sick patient is a complex and sensitive issue that would require careful consideration of various factors.

In making this decision, the following factors should be taken into account:

-The patient's overall health and prognosis: The patient's overall health and prognosis should be carefully assessed. If the patient has a severe illness with a poor prognosis, the focus may shift towards comfort care and discussions about end-of-life wishes.
-The patient's preferences: The patient's preferences and wishes should be respected. If they have expressed their desire not to undergo invasive treatments, or to focus on comfort care, their wishes should be honored.
-The patient's consent and capacity: The patient's ability to consent to treatment should be assessed. If the patient is unable to consent due to mental incapacity or other factors, the doctor may need to make a decision based on their understanding of the patient's best interests.
-The ethical and legal implications: There are ethical and legal considerations to consider when making decisions about end-of-life care. The doctor must weigh the potential benefits and risks of continuing treatments against the patient's right to make decisions about their care.

Ultimately, the decision of whether to continue treatments or prioritize comfort and discuss end-of-life options should be made on a case-by-case basis, taking into account the specific circumstances of the patient's situation and the ethical and legal implications involved.

This indicates that the model is made with ethical cases like this one in heavy consideration. Here’s one more example of this model’s decision making capabilities:

We asked the model, ‘You haven’t studied at all for a test,would you rather fail a test or cheat on the test to pass?’ and it responded with :

I would recommend failing the test, even if it meant not cheating. It is more important to maintain academic integrity and honesty than to cheat on a test. Cheating can have serious consequences, such as academic suspension or expulsion, which could hinder future academic opportunities. Additionally, cheating goes against the principles of fairness and equity, as it gives an unfair advantage to those who cheat over those who do not. Therefore, I would advise against cheating on a test, even if it means failing.

With these examples we can see that the model is quite well trained in decision making tasks. Now, let’s see how it does for problem solving issues.

Problem Solving:

This model is also quite adaptive of problems and seems to be quite great in problem solving as well. Some of it’s responses to our questions were:

Question: ‘There’s a lack of access to healthy food options in your community, particularly for low-income residents. How can you work with local organizations and businesses to create solutions that increase access to nutritious food?’

Response:

1. Identify the Problem:

- Conduct research to understand the specific food scarcity challenges faced by low-income residents in your community.
- Gather data on the prevalence of food deserts and areas with limited access to grocery stores, farmers markets, or healthy food options.

2. Collaborate with Local Organizations:

- Reach out to local food banks, non-profit organizations, and community centers to explore existing initiatives and potential partnerships.
- Seek their input and expertise in understanding the needs of the community and developing solutions that align with their resources and goals.

3. Create a Community Garden:

- Establish a community garden where residents can grow their own fruits, vegetables, and herbs.
- Partner with local businesses to provide land, tools, and resources for the garden.

4. Community Cooking Classes:

- Organize cooking classes, led by local chefs or nutritionists, to teach residents how to prepare healthy and affordable meals.
- Provide ingredients or offer subsidies to make the classes accessible.

5. Mobile Grocery Delivery Services:

- Partner with local grocery stores to offer mobile delivery services to residents who are unable to travel to the store.
- Establish a partnership to provide discounted prices on fresh produce and groceries.

6. Nutritional Education Programs:

- Implement educational programs about healthy eating, food budgeting, and meal planning.
- Partner with local schools and community centers to reach a wider audience.

7. Encourage Community Support:

- Create a community-driven network of volunteers who can help distribute food, provide transportation to grocery stores, or offer support to those in need.

8. Advocate for Policy Changes:

- Partner with local politicians and advocacy groups to promote policies that address food scarcity issues, such as increased investment in affordable housing and transportation programs.

9. Spread Awareness:

- Raise awareness about the issue of food insecurity in your community through public campaigns, community events, and social media outreach.

10. Continuous Assessment:

- Continuously monitor and assess the impact of your interventions to ensure they are effectively addressing the needs of the community. Make adjustments as needed to optimize results.

Question: ‘A conflict has arisen between two colleagues on your team. How can you facilitate a productive discussion and find a resolution that satisfies both parties?’

Response:

1. Schedule a meeting: Arrange a time and place where both colleagues can meet to discuss their concerns. Make sure it's in a neutral location, such as a conference room or quiet coffee shop.

2. Set ground rules: At the beginning of the meeting, establish some basic guidelines for respectful communication. Encourage active listening, speaking calmly and avoiding interruptions. This will help create a safe space for open dialogue.

3. Allow each person to share their perspective: Give both colleagues equal opportunity to express their thoughts and feelings about the situation without judgment or interruption. Encourage them to use "I" statements instead of blaming language (e.g., "I felt disrespected when..." rather than "You always do this thing that annoys me.").

4. Identify the root cause: Help the colleagues understand the underlying issues behind the conflict by asking clarifying questions and summarizing key points. Try to uncover any miscommunications, misunderstandings, or differing expectations that may have contributed to the problem.

5. Brainstorm solutions together: Once you've identified the core issue, work with both parties to come up with potential resolutions. Ask open-ended questions like, "What could we do differently next time?" or "How can we ensure this doesn't happen again?" Write down all ideas, even if they seem far-fetched at first.

6. Evaluate options: Discuss the pros and cons of each solution, focusing on what would be most beneficial for both individuals and the team as a whole. Narrow down the list to the top two or three viable options.

7. Reach a consensus: Guide the colleagues toward choosing a mutually agreeable solution from the remaining options. If necessary, offer suggestions or mediate the decision-making process. Ensure that both parties feel heard and satisfied with the final outcome.

8. Establish a plan of action: Clearly define the steps required to implement the chosen solution. Assign responsibilities and set deadlines as needed. Determine how progress will be monitored and who will follow up to ensure the resolution is working effectively.

9. Foster ongoing communication: Encourage both colleagues to maintain open lines of communication moving forward. Remind them that addressing conflicts early on can prevent future problems and strengthen relationships within the team.

10. Follow up: Check in with both parties after a predetermined period (a few days or weeks) to see how things are going. Offer additional support or resources if needed, and acknowledge their efforts to resolve the conflict constructively.

Mathematical Problem Solving:

We’ll be testing this model with some high school level calculus. We’ll be using this to solve a sample derivative problem.

We asked the model: ‘Find derivative of f(x)=1/2x²+3x-4’

It responded with:

To find the derivative of f(x) = 1/2x^2 + 3x - 4, we'll apply the power rule for differentiation. The power rule states that if g(x) = x^n, then g'(x) = n * x^(n-1).

So, let's break down f(x) into its component parts:

g(x) = 1/2x^2
h(x) = 3x
k(x) = -4

Now, let's find the derivatives of each part using the power rule:

g'(x) = d(1/2x^2)/dx = (2)(1/2)x^(2-1) = x
h'(x) = d(3x)/dx = 3(1)x^(1-1) = 3
k'(x) = d(-4)/dx = 0 (since constants have a derivative of zero)

Finally, add these individual derivatives back together to get the overall derivative of f(x):

f'(x) = g'(x) + h'(x) + k'(x) = x + 3 + 0 = x + 3

These responses imply that Gemma is really good with problem solving tasks.

Current Limitations in Multilingual Applications

Performance of the model, particularly with low-resource languages such as Nepali, reveals notable challenges. Consider the Nepali phrase “Malai momo mitho lagchha,” which translates to “I find momo tasty.” Momo refers to a popular Nepali dumpling filled with meat. However, the model’s interpretation deviates significantly, translating it as “the goat is pulling the cart”. Further it generates sentence completion with “The goat is pulling the cart, but the load is too heavy, so it is not moving.”

This underscores the substantial difficulties current models face with languages that have fewer resources available for machine learning training.

Conclusion

While the latest model is remarkably advanced and cater to a wide array of applications, their efficacy diminishes in the context of low-resource, multilingual scenarios. This limitation underscores the need for continued development and innovation, with the hope that future iterations will more effectively bridge these gaps, either through community contributions or technological advancements.