Writing Logs in GCP Logs Explorer for Compute Engine on Golang

Photo by Dorelys Smits

The Cloud Logging API lets you programmatically accomplish logging-related tasks, including reading and writing log entries and creating log-based metrics.

If our project is deployed in Google App Engine(GAE), program logs are automatically logged by GAE in Logs Explorer. But in the case of deploying project in Google Compute Engine(GCE) it is a hassle to see program logs. So we have to manually write program logs in order to display in Logs explorer.

We can write Request logs or Specific logs according to our desire using cloud logging API

You should have project deployed in GCP compute engine or App engine

We will follow the below steps to write logs:

• Create a GCP project
• Create a VM instance in Compute engine
• Deploy your app in Compute engine
• Understanding Logging API
• Create service Account
• Generate Service Account Key to authenticate API request
• Using Cloud Logging Client Library
• Writing Request Logs
• List Logs Entries
• Delete Logs Entries

I am assuming that you have completed first three steps already. So, we are skipping these steps and follow other steps.

Understanding Logging API

The Logging API must be enabled before it can be used. You can check the status of the API and, if necessary, enable the API if necessary by using either the Google Cloud console or Google Cloud CLI.

Enable Logging API

1. In the Cloud console, select the Cloud project for which you want to enable the API, and then go to the APIs & Services page:
2. Go to APIs & Services
3. Click the Enable APIs and Services button.
4. Search for “Logging”.
5. In the search results, click Cloud Logging API.
6. If “API enabled” is displayed, then the API is already enabled. If not, then click Enable.

Authorize use of the API

Before you can use the Cloud Logging API, you must authorize users to access it. Here we are going to authorize user using Service Account Key.

Access to the API is controlled by Identity and Access Management (IAM) roles and permissions. There is a permission associated with each method, and permissions for related methods are clustered into roles, which can then be granted to users.

In GCP console, Select a project navigate to IAM and admin section and then navigate to Service Account.

Here you can see list of available or already created service accounts. You can either use default service account key or create a new service account key.

Create Service Account

Enter service account details

Grant the service account access to project

In this step, We assign role to grant service account access to project. Here we are giving role Logging admin , this role can read, write, delete logs and other logger bucket related tasks.

We can also specify other roles such as Logs writer which has only access to write logs.

Grant users access to this service account (optional)

In this step we specify the users access to this service account. We can specify user emails which will have access to this service account. For now, let’s keep it blank.

Generate Service Account Key to authenticate API request

After successfully creating service account. Next step is to generate the Service Account Key.

Now from the service account list page we select the recently created service account and navigate to keys tab and add a new key as shown in screenshot below and you will be promoted to download json file, download the file and save into your machine.

Till now, we have created Service Account and generated the key to authenticate our Logging API.

To use the service account key in the project, create a file named cloudApiKey.json in project’s root directory and Copy the content of previously downloaded json file and paste into the cloudApiKey.json file

Using Cloud Logging Client Library

Install the client library

go get cloud.google.com/go/logging

Writing structured logs

package main
import (
        "context"
        "fmt"
        "log"
        "cloud.google.com/go/logging"
)
func main() {
        ctx := context.Background()
        // Sets your Google Cloud Platform project ID
        projectID := "YOUR_PROJECT_ID"
        // load the file from root path to authenticate
        cloudApiKey, err := filepath.Abs("./cloudApiKey.json")
		if err != nil {
		logger.Zap.Panic("Unable to load             cloudApiKey.json file")
		}
		option := option.WithCredentialsFile(cloudApiKey)
		// Creates a client with credential file
		client, err := logging.NewClient(ctx, projectID, option)
        if err != nil {
                log.Fatalf("Failed to create client: %v", err)
        }
        // Sets or create the name of the log to write to 
        logName := "my-log"
        // Selects the log to write to.
        logger := client.Logger(logName)
        defer logger.Flush() // Ensure the entry is written.
        logger.Log(logging.Entry{
        // Log anything that can be marshaled to JSON.
        Payload: struct{ Anything string }{
                Anything: "The payload can be any type!",
        },
        Severity: logging.Debug,
        })
}

Writing the request Logs

import (
        "context"
        "log"
        "net/http"
        "os"
        "cloud.google.com/go/logging")
func main() {
        ctx := context.Background()
        // Creates a client.
        projectID := os.Getenv("GOOGLE_CLOUD_PROJECT")
        client, err := logging.NewClient(ctx, projectID)
        if err != nil {
                log.Fatalf("Failed to create client: %v", err)
        }
        defer client.Close()
        // Sets the name of the log to write to.
        logger := client.Logger("my-log")
        // Logs a basic entry.
        logger.Log(logging.Entry{Payload: "hello world"})
        // TODO(developer): replace with your request value.
        r, err := http.NewRequest("GET", "http://example.com", nil)
        // Logs an HTTPRequest type entry.
        httpEntry := logging.Entry{
                Payload: "optional message",
                HTTPRequest: &logging.HTTPRequest{
                        Request: r,
                },
        }
        logger.Log(httpEntry)}

List Log Entries

var entries []*logging.Entryconst name = "log-example"
lastHour := time.Now().Add(-1 * time.Hour).Format(time.RFC3339)
iter := adminClient.Entries(ctx,
        // Only get entries from the "log-example" log within the last hour.
        logadmin.Filter(fmt.Sprintf(`logName = "projects/%s/logs/%s" AND timestamp > "%s"`, projID, name, lastHour)),
        // Get most recent entries first.
        logadmin.NewestFirst(),)
        // Fetch the most recent 20 entries.for len(entries) < 20 {
        entry, err := iter.Next()
        if err == iterator.Done {
                return entries, nil
        }
        if err != nil {
                return nil, err
        }
        entries = append(entries, entry)}return entries, nil

Delete Log Entries

const name = "log-example"if err := adminClient.DeleteLog(ctx, name); err != nil {
        return err
}

So, this is how we can write any kinds of logs to GCP log explorer for the programs deployed in Google Compute Engine (GCE).

References: https://cloud.google.com/logging/docs/apis

Happy coding 🎉