Gemini CLI Tutorial Series : Part 13 : Gemini CLI Observability

Thank you for reading! If you found this series helpful, please consider taking this 2-minute survey. Your feedback directly helps me create more tutorials for the developer community.

Gemini CLI Tutorial Series:

Part 1 : Installation and Getting Started
Part 2 : Gemini CLI Command line options
Part 3 : Configuration settings via settings.json and .env files
Part 4 : Built-in Tools
Part 5: Using Github MCP Server
Part 6: More MCP Servers : Firebase, Google Workspace, Google Gen AI Media Services and MCP Toolbox for Databases
Part 7: Custom slash commands
Part 8: Building your own MCP Server
Part 9: Understanding Context, Memory and Conversational Branching
Part 10: Gemini CLI and VS Code Integration
Part 11: Gemini CLI Extensions
Part 12: Gemini CLI GitHub Actions
Part 13 : Gemini CLI Observability (this post)
Part 14: Gemini CLI extensions for Google Data Cloud

➡️ Codelab : Hands-on Gemini CLI

Welcome back to the Gemini CLI Tutorial Series! In this part, we are going to look at what it means to create our own private analytics pipeline for Gemini CLI.

For e.g. Have you ever wondered which custom commands you use the most? Or how many tokens you’re consuming on a specific project? What if you wanted to track metrics like this to understand your own behaviour? It could be very useful and is usually the norm in organizations that need to track software usage and gain insights into usage patterns.

Taking a cue from the official documentation around telemetry in the Gemini CLI documentation, here are the key benefits (produced from https://github.com/google-gemini/gemini-cli/blob/main/docs/cli/telemetry.md#key-benefits):

• 🔍 Usage Analytics: Understand interaction patterns and feature adoption across your team
• ⚡ Performance Monitoring: Track response times, token consumption, and resource utilization
• 🐛 Real-time Debugging: Identify bottlenecks, failures, and error patterns as they occur
• 📊 Workflow Optimization: Make informed decisions to improve configurations and processes
• 🏢 Enterprise Governance: Monitor usage across teams, track costs, ensure compliance, and integrate with existing monitoring infrastructure

Gemini CLI uses OpenTelemetry, which is a popular standard now across vendors for an observability framework. It supports metrics, logs and traces and can export to any OpenTelemetry backend but we will focus in this part on Google Cloud as the backend itself.

We shall be looking at doing the following next steps:

1. Setting up the necessary services and permissions in your Google Cloud project.
2. Configuring Gemini CLI’s settings.json to ensure that the telemetry configuration is done correctly. As mentioned earlier, we shall be using Google Cloud as the telemetry backend and will funnel the metrics, logs and traces into the Google Cloud Project.
3. Generate some sample data using the CLI.
4. Finding and visualizing the private telemetry data in Google Cloud Logging and Monitoring.

Prerequisites

Before we begin, please ensure you have the following set up. We’ve covered most of these in earlier parts of the series, but it’s always good to double-check.

• A Google Cloud Account: You’ll need a Google Cloud account with an active billing account enabled.
• Google Cloud CLI: Make sure you have the gcloud command-line interface (CLI) installed and authenticated to your Google account. You can follow the installation instructions here.
• Gemini CLI: You should have Gemini CLI installed and authenticated with your Google account. We covered this in Part 1.

A gentle reminder that Gemini CLI, with its open-source nature, releases versions frequently. If you already have it set up, please remember to upgrade it to the latest version to ensure you have the telemetry features we’ll be using today. The command to upgrade Gemini CLI to the latest stable version is given below:

npm install -g @google/gemini-cli@latest

Once that’s done, you can verify your version with gemini -v.

At the time of writing this post, my Gemini CLI version is 0.8.1

Setup your GCP Project, Account and Project APIs

The first step is to ensure that you are working with a specific Google Cloud Platform project that you want to send the observability data to. Ensure that you use the following gcloud CLI command and replace YOUR_PROJECT_ID with your actual Google Cloud project ID.

gcloud config set project YOUR_PROJECT_ID

While Gemini CLI supports sending the observability data to a different or the same project that you are using as inference, I am going to use a different project typically for the inference. This might help in a situation where you can possibly combine multiple users sending their data into a single project, where you can then analyze the metrics, logs, etc.

So provide the following command to set the telemetry in a separate project from inference:

export OTLP_GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"

In case, you want to use telemetry in the same project as the inference, use the following environment variable:

export GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"

The next step is to determine whether you want to use your own user account which you have authenticated with gcloud CLI or you want to use a Service Account in GCP to be the identity for Gemini CLI, when it interacts with your Google Cloud Project.

But first, we need to identify the roles that are needed for either your user account or the service account. They are:

Google Cloud provides the following specific, predefined roles:

• Cloud Trace Agent: The required role is roles/cloudtrace.agent. This allows the principal (user or service account) to write trace data to the Cloud Trace service.
• Monitoring Metric Writer: The required role is roles/monitoring.metricWriter. This allows the principal to write metric data to the Cloud Monitoring service.
• Logs Writer: The required role is roles/logging.logWriter. This allows the principal to write log entries to the Cloud Logging service.

The principal here is the user or service account and and Gemini CLI will use that identity when it interacts with Google Cloud.

Now that we have identified the roles required, we need to assign them to either the user account or the service account.

The base command to assign a role to a principal for a specific project is gcloud projects add-iam-policy-binding. You will need to replace the placeholder [PROJECT_ID] with your actual Google Cloud project ID.

Grant Cloud Trace Agent role:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
     --member="user:[USER_EMAIL]" \
     --role="roles/cloudtrace.agent"

Grant Monitoring Metric Writer role:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
     --member="user:[USER_EMAIL]" \     
     --role="roles/monitoring.metricWriter"

Grant Logs Writer role:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
     --member="user:[USER_EMAIL]" \     
     --role="roles/logging.logWriter"

If you plan to use a Service Account, then note that a service account is also identified by its email address. The member type is serviceAccount.

Replace [SERVICE_ACCOUNT_EMAIL] with the service account's full email address (e.g., my-sa@your-project-id.iam.gserviceaccount.com).

Grant Cloud Trace Agent role:

gcloud projects add-iam-policy-binding [PROJECT_ID] \     
   --member="serviceAccount:[SERVICE_ACCOUNT_EMAIL]" \     
   --role="roles/cloudtrace.agent"

Grant Monitoring Metric Writer role:

gcloud projects add-iam-policy-binding [PROJECT_ID] \     
   --member="serviceAccount:[SERVICE_ACCOUNT_EMAIL]" \     
   --role="roles/monitoring.metricWriter"

Grant Logs Writer role:

gcloud projects add-iam-policy-binding [PROJECT_ID] \     
   --member="serviceAccount:[SERVICE_ACCOUNT_EMAIL]" \     
   --role="roles/logging.logWriter"

The final step is to ensure that certain Google Cloud APIs are enabled in your project. This is required to ensure that when Gemini CLI sends it data, these APIs are enabled in the project for them to receive and do their magic. For e.g. we need to enable the Cloud Logging API to receive event logs and the Cloud Monitoring API to receive the metrics.

Run the following command in your terminal to enable them:

gcloud services enable \
  cloudtrace.googleapis.com \
  monitoring.googleapis.com \
  logging.googleapis.com

You can even enable the services individually if you want by giving one command after the other.

gcloud services enable cloudtrace.googleapis.com

gcloud services enable monitoring.googleapis.com

gcloud services enable logging.googleapis.com

Note that each command may take several seconds to complete. You’ll see a success message like Operation "operations/..." finished successfully.

Configuring Gemini CLI Telemetry

Now we’ll edit the Gemini CLI configuration to enable the Google Cloud telemetry destination. As we learned in Part 3, Gemini CLI’s settings are managed in a settings.json file. This can be a global file in your home directory (~/.gemini/settings.json) or a project-specific file in your current working directory (.gemini/settings.json). For this tutorial, let's modify the global file so the setting applies everywhere.

Open your ~/.gemini/settings.json file in your favorite text editor. We need to add a telemetry block with a googleCloud object inside it.

Here is what the configuration should look like. Copy this into your settings.json file, making sure to replace the placeholder values.

{
  ... //other settings

  "telemetry": {
    "enabled": "true",
    "target": "gcp"
  }

   ... //other settings
}

Save the file, and you’re all set! The next time you start Gemini CLI, it will read this configuration and begin sending telemetry to your project.

Generating and Verifying Telemetry Data

Now for the fun part! We’ve set up the destination and configured the source. Let’s generate some data and see it arrive in our Google Cloud project. It’s important to know that the data won’t appear in a pre-made “Gemini CLI Dashboard.” We are sending logs and metrics, so we’ll need to use the Logs Explorer and Metrics Explorer to find them by the names defined in the documentation.

You now have a stream of data flowing into your project, but what does it all mean? The telemetry is divided into two main types: Events (which appear in Cloud Logging) and Metrics (which appear in Cloud Monitoring).

Let’s Generate Some Data!

First, if you have Gemini CLI running, quit it with /quit. Then, restart it from your terminal by simply typing gemini. This ensures it picks up the new settings.

Now, just use it as you normally would. This will generate events and metrics that will be sent to your project. Try a few things:

• Ask a simple question: What are the wonders of the world?
• List the available tools: /tools
• Run a shell command: !ls -l

Each of these actions generates telemetry data that is now flowing into your private pipeline.

Viewing Events in Cloud Logging

After a minute or two, the log events should start appearing. Let’s go find them.

1. Navigate to the Google Cloud Console.
2. Open the main navigation menu (the “hamburger” icon ☰) and go to Logging -> Logs Explorer. Alternately, if you are logged into the Google Cloud Console, use https://console.cloud.google.com/logs/.
3. In the Query builder, we’re going to search for the logName , which by default is “projects/YOUR_PROJECT_ID/logs/gemini-cli”. Type the following query, replacing YOUR_PROJECT_ID with your project ID, and click Run query:

logName="projects/YOUR_PROJECT_ID/logs/gemini-cli"

or you can even get it from the dropdown list of the log names as shown below:

You should see a list of log entries appear! These are the events from your Gemini CLI session. Click on one of the entries to expand it and view its JSON payload. You’ll see structured data corresponding to events like gemini_cli.user_prompt or gemini_cli.tool_call, complete with attributes like prompt_length and function_name.

Let’s investigate into our log entries for the queries that we did.

The first one was a user prompt given for What are the wonders of the world?, and here is the log entry. The key thing to note is the event.name because you might eventually want to add that to the log filters.

The above prompt resulted in Gemini CLI using the google_web_search tool to query Google Search for the prompt asked. The log entry for that is also visible:

Similarly, we gave the /tools command and here is the log entry:

Let’s move on to the next part i.e. metrics and visualizing them using Google Cloud Monitoring.

Visualizing Metrics in Cloud Monitoring

Now let’s see some numbers! Metrics are numerical time-series data, perfect for creating charts and dashboards.

• In the Google Cloud Console, open the navigation menu and go to Monitoring -> Metrics Explorer. Or you can directly navigate to Metrics Explorer via the following URL: https://console.cloud.google.com/monitoring/metrics-explorer
• In the Select a metric field, a dropdown will appear. In the filter box, start typing gemini_cli. You should see a new category of metrics appear, prefixed with custom.googleapis.com/gemini_cli/. This prefix indicates that these are custom metrics being sent from our application.

• Let’s visualize the number of tool calls. Select the metric tool.call.count.
• The Metrics Explorer will automatically generate a chart. You can further refine it. For example, in the Group by field, you can select the function_name label to see a separate line on the chart for each tool that was called.

Neat, right? You’ve just created your first custom monitoring chart for your personal Gemini CLI usage!

You can look at several other metrics like token.usage, etc. A sample chart for token.usagefrom my project is shown below:

You can look up the documentation for Metrics to learn about the metrics and what kind of data you can expect.

Summary

You’ve successfully configured a private, secure telemetry pipeline for your Gemini CLI usage. You’ve gone from being a user of the tool to an owner of its data. You now have the power to analyze your own workflows, track token consumption, and gain deeper insights into how you use Gemini CLI.

You can take this further by creating custom dashboards in Cloud Monitoring to track the metrics that matter most to you, or set up alerts to notify you of specific events, like an increase in API errors. Let me know in the comments, if you have created some interesting usage dashboards, alerts or even combined it with some other external data.

Please consider leaving feedback

Thank you for reading! If you found this series helpful, please consider taking this 2-minute survey. Your feedback directly helps me create more tutorials for the developer community.