Google Chrome’s Built-In AI Is A Game Changer.

On-device Gemini Nano is now available on Chrome Canary

We all love using ChatGPT, Claude, Gemini, and other AI tools to ask everything from silly questions to serious ones that might affect hundreds of people. Despite the abundance of AI tools available, the experience may not feel seamless due to frequent server issues or horrible latency when your internet provider decides to take a break. This is where on-device AI becomes a real boon.

Since Google launched the Gemini family of models, Gemini Pro has been in the spotlight for its capabilities, such as improved coherent responses, better understanding, and a large context window. However, with great power comes great resp — resource consumption. Gemini Pro is quite resource-hungry, which is why you cannot run it locally.

To address this issue, Google is now experimenting with Gemini Nano on Chrome Canary, which would run the language model locally on your machine.

So what is Chrome Canary?

For those who are unaware, Chrome Canary is an experimental version of Google Chrome designed for developers and tech enthusiasts who want to test the latest features and APIs before they are widely available.

To test out on-device AI, we will need to use Chrome Canary, as this feature is still in beta and potentially unstable.

Why do we need on-device AI?

1. Enhanced security and privacy
2. Improved performance
3. Faster response times
4. Cost efficiency

Personally, I believe on-device AI is also more sustainable since these models are lightweight and use local resources, indirectly leading to a smaller carbon footprint. While this may not create a significant impact immediately, we should acknowledge this potential for better future prospects.

How to get started with Gemini Nano on Chrome Canary?

Chrome Canary Flags

1. Install Chrome Canary (obviously).
2. In the address bar, enter chrome://flags
3. Change the ‘Enables optimization guide on device’ to Enabled BypassPerfRequirement
4. Set the ‘Prompt API for Gemini Nano’ to Enabled
5. Restart the browser and wait for it to download Gemini Nano to your local machine (22gb of free space is required although the model itself isn’t more than 3gb)
6. To check the status, navigate to chrome://components and ensure that the Optimization Guide On Device Model shows version 2024.6.5.2205 or higher. If not, click 'Check for updates'.
7. Done!

Using Gemini Nano to generate responses

To verify if Gemini Nano is installed successfully and ready to generate responses, use the following command in the console by going to the developer tools of Chrome: await window.ai.canCreateTextSession() and it should respond with ‘readily.’

Let’s test it out on some basic queries:

Demonstration

To keep the overall application simple, we will be building a summarizer for the selected text within the console itself.

We can retrieve the text selection using this function:

function getSelectedText() {
    var selectedText = window.getSelection().toString();
    return selectedText;
}

document.addEventListener('mouseup', function() {
    var selectedText = getSelectedText();
    if (selectedText) {
        console.log(selectedText);
        // Perform further actions with the selected text
    }
});

The next step is to inject this data into Gemini Nano. This can be performed by slightly refactoring and modifying the above function.

function getSelectedText() {
    return window.getSelection().toString();
}

async function handleSelection() {
    const selectedText = getSelectedText();
    if (selectedText) {
        const canCreate = await window.ai.canCreateTextSession();
        console.log("Summarizing Data...");
        if (canCreate !== "no") {
            try {
                const session = await window.ai.createTextSession();
                const result = await session.prompt("Summarize the following text in bullet points: " + selectedText);
                
                console.log(result);
            } catch (error) {
                console.error("Error creating session or prompting:", error);
            }
        }
    }
}

document.addEventListener('mouseup', handleSelection);

Based on your machine’s hardware the responses should be typically generated in a couple of seconds.

What does this mean for us?

Well, this means building GenAI apps and wrappers (let’s not be too critical of them, shall we?) has become much simpler. Once this feature is available publicly on Google Chrome, the number of extensions and websites being developed will likely be staggering.

The best part for developers is that it requires zero capital investment and eliminates worries about overshooting API quotas and declaring bankruptcy.

I hope you enjoyed reading this fun article and found it informative. Follow me for more such articles to stay updated on exciting developments in the tech landscape. People have already started experimenting with Gemini Nano and building impressive projects, such as Gmail plugins, as demonstrated in this tweet: