Open in app

Sign in

Write

Sign in

Retrieving Log Entries from Google Cloud Logging in Go

Nikhil Shrestha
readytowork, Inc.
Published in
10 min readJan 13, 2023

Logs are very important for monitoring, troubleshooting, and improving application performance. At present, the Google Cloud Platform has established itself as a major platform with several cloud-based services. Among them, Google Cloud Logging provides storage for logs, and also a user interface, namely the Logs Explorer. While the Logs Explorer provides many functionalities to operate on logs on various levels, working with logs on the programming level can be unavoidable when it may serve as an integral feature of a system.

In this article, we will focus on retrieving the stored logs in the Google Cloud. We will be exploring two different methods to retrieve the logs; Client Libraries and Google Cloud API.

Project Structure

  1. main.go //file containing all the functions that we will be using
  2. .env //file containing environment variables
  3. defaultAPiServiceAccountKey.json //service account key generated from the Google Cloud project

Key Points

  1. Service key, generated from the google cloud project, is integral to accessing the logs in the cloud.
  2. Initialize the project id in the .env file to use it in the program.
  3. Retrieving logs also depends on the retention period. Log entries are stored in Cloud Logging for a limited time.

The method I: Using Cloud Logging Client Library

Let us import all libraries required for performing our operation

import (
   "cloud.google.com/go/logging"
   "cloud.google.com/go/logging/logadmin"
   "context"
   "fmt"
   "github.com/joho/godotenv"
   "google.golang.org/api/iterator"
   "google.golang.org/api/option"
   "log"
   "os"
   "path/filepath"
   "time"
)

Next, let’s write a function called GetLogEntries() which is responsible to get the log entries from the project based on the cloud.

func GetLogEntries(adminClient *logadmin.Client, projectID string) ([]*logging.Entry, error) {
   ctx := context.Background()
   const name = "stderr"
   searchKeyword := "CUSTOM_LOG"
   lastMonth := time.Now().AddDate(0, -1, 0).Format(time.RFC3339)

   var entries []*logging.Entry
   iter := adminClient.Entries(ctx, logadmin.Filter(fmt.Sprintf(`logName = "projects/%s/logs/%s" AND textPayload:"%s" AND timestamp > "%s"`, projectID, name, searchKeyword, lastMonth)), logadmin.NewestFirst())

   for len(entries) < 50 {
      entry, err := iter.Next()
      if err == iterator.Done {
         return entries, nil
      }
      if err != nil {
         return nil, err
      }
      entries = append(entries, entry)
   }
   return entries, nil
}

GetLogEntries()

This function is taking two arguments adminClient and projectID of *logadmin.Client and string type respectively. To read the log entries from the cloud, Go provides a specific package, logadmin, containing a Cloud Logging client which we have used to read logs.

There may be several logs stored in the cloud storage, which may or may not be of your importance. However, we can use filters to query the logs that meet our requirements and the same has been done above. Furthermore, we have used a for loop to iterate through each entry that meets the filtered query and append it to a slice which is finally returned at the end.

Now, moving to the main function

func main() {
   if err := godotenv.Load(); err != nil {
      log.Fatal("error loading .env file")
   }
   ctx := context.Background()
   projectID := os.Getenv("GCLOUD_PROJECT_ID") 
   defaultServiceKey, err := filepath.Abs("./defaultApiServiceAccountKey.json")
   if err != nil {
      log.Fatalf("unable to load defaultApiServiceAccountKey.json file while initializing cloud logging: %v", err)
   }

   optionWithCredential := option.WithCredentialsFile(defaultServiceKey)
   adminClient, err := logadmin.NewClient(ctx, projectID, optionWithCredential)
   if err != nil {
      log.Fatalf("Failed to create client: %v", err)
   }
   defer adminClient.Close()

   log.Print("Fetching and printing log entries")
   entries, err := GetLogEntries(adminClient, projectID)
   if err != nil {
      log.Fatalf("error getting entries: %v", err)
   }
   for _, entry := range entries {
      fmt.Println(entry)
   }
}

main()

In the main function, we define all the required environments for the operation, such as the project id, and default service key of the concerned project. The main function creates an admin client which, along with the project id, is passed to the GetLogEntries(). Finally, the log entries returned from the GetLogEntries() are each printed using for loop. Each log entry can be used to retrieve the timestamp, payload, and the name of the log it belongs to, at the very least.

Finally, the complete code:

//main.go
package main

import (
   "cloud.google.com/go/logging"
   "cloud.google.com/go/logging/logadmin"
   "context"
   "fmt"
   "github.com/joho/godotenv"
   "google.golang.org/api/iterator"
   "google.golang.org/api/option"
   "log"
   "os"
   "path/filepath"
   "time"
)

func GetLogEntries(adminClient *logadmin.Client, projectID string) ([]*logging.Entry, error) {
   ctx := context.Background()
   const name = "stderr"
   searchKeyword := "CUSTOM_LOG"
   lastMonth := time.Now().AddDate(0, -1, 0).Format(time.RFC3339)

   var entries []*logging.Entry
   iter := adminClient.Entries(ctx, logadmin.Filter(fmt.Sprintf(`logName = "projects/%s/logs/%s" AND textPayload:"%s" AND timestamp > "%s"`, projectID, name, searchKeyword, lastMonth)), logadmin.NewestFirst())

   for len(entries) < 50 {
      entry, err := iter.Next()
      if err == iterator.Done {
         return entries, nil
      }
      if err != nil {
         return nil, err
      }
      entries = append(entries, entry)
   }
   return entries, nil
}


func main() {
   if err := godotenv.Load(); err != nil {
      log.Fatal("error loading .env file")
   }
   ctx := context.Background()
   projectID := os.Getenv("GCLOUD_PROJECT_ID")
   defaultServiceKey, err := filepath.Abs("./defaultApiServiceAccountKey.json")
   if err != nil {
      log.Fatalf("unable to load defaultApiServiceAccountKey.json file while initializing cloud logging: %v", err)
   }

   optionWithCredential := option.WithCredentialsFile(defaultServiceKey)
   adminClient, err := logadmin.NewClient(ctx, projectID, optionWithCredential)
   if err != nil {
      log.Fatalf("Failed to create client: %v", err)
   }
   defer adminClient.Close()

   log.Print("Fetching and printing log entries")
   entries, err := GetLogEntries(adminClient, projectID)
   if err != nil {
      log.Fatalf("error getting entries: %v", err)
   }
   for _, entry := range entries {
      fmt.Println(entry)
   }
}

Method II: Using Cloud Logging API

Code Implementation

Let us import all libraries required for performing our operation

import (
 "bytes"
 "encoding/json"
 "fmt"
 "github.com/joho/godotenv"
 "golang.org/x/oauth2"
 oauth2google "golang.org/x/oauth2/google"
 "io/ioutil"
 "log"
 "net/http"
 "os"
 "path/filepath"
 "time"
)

Now, we will declare a struct, Entry, which we will later use in our function.

type Entry struct {
   TextPayload string
   Timestamp   string
}

Let’s write a function called ListLogEntries() which is responsible to get the log entries from the project based on the cloud using Cloud Logging API.

func ListLogEntries(projectID string, token *oauth2.Token) ([]Entry, error) {
   name := "stderr"
   keyword := "CUSTOM_LOG"
   lastMonth := time.Now().AddDate(0, -1, 0).Format(time.RFC3339)

   filter := fmt.Sprintf(`logName = "projects/%s/logs/%s" AND textPayload:"%s" AND timestamp > "%s"`, projectID, name, keyword, lastMonth)

   var entries []Entry
   nextpageToken := ""
   for {
      postBody, err := json.Marshal(map[string]interface{}{
         "resourceNames": []string{"projects/" + projectID},
         "filter":        filter,
         "orderBy":       "timestamp desc",
         "pageSize":      1000,
         "pageToken":     nextpageToken,
      })
      if err != nil {
         err = fmt.Errorf("unable to marshal json: %v", err)
         return nil, err
      }

      requestBody := bytes.NewBuffer(postBody)
      request, err := http.NewRequest("POST", "https://logging.googleapis.com/v2/entries:list", requestBody)
      if err != nil {
         err = fmt.Errorf("unable to create request: %v", err)
         return nil, err
      }

      request.Header.Add("Content-Type", "application/json")
      request.Header.Add("Authorization", "Bearer "+token.AccessToken)

      client := &http.Client{}
      response, err := client.Do(request)
      if err != nil || response.StatusCode != http.StatusOK {
         err = fmt.Errorf("unable to fetch logs: %v", err)
         return nil, err
      }

      body, err := ioutil.ReadAll(response.Body)
      if err != nil {
         err = fmt.Errorf("unable to unmarshal response body: %v", err)
         return nil, err
      }

      result := struct {
         Entries       []Entry
         NextPageToken string
      }{}
      if err := json.Unmarshal(body, &result); err != nil {
         err = fmt.Errorf("unable to unmarshal response body: %v", err)
         return nil, err
      }
      entries = append(entries, result.Entries...)
      if len(result.Entries) == 0 || result.NextPageToken == "" {
         response.Body.Close()
         break
      } else {
         nextpageToken = result.NextPageToken
         response.Body.Close()
      }
   }
   return entries, nil
}

ListLogEntries()

In the function above, we are sending a POST request to get the log entries from Google Cloud. The process has been iterated with a for loop so that all the pages returned can be read through the function. The function takes two argument projectID and token of string and *oauth2.Token type respectively to perform the defined operations and return the retrieved entries.

Since we have our function, let’s break down the major happenings:

name := "stderr"
keyword := "CUSTOM_LOG"
lastMonth := time.Now().AddDate(0, -1, 0).Format(time.RFC3339)

filter := fmt.Sprintf(`logName = "projects/%s/logs/%s" AND textPayload:"%s" AND timestamp > "%s"`, projectID, name, keyword, lastMonth)

These are the filters set up to find the logs that meet the requirement. It is important to note that the date in the filter must be in time.RFC3339 format. There are also other filters available that can be used to make your query more efficient.

postBody, err := json.Marshal(map[string]interface{}{
   "resourceNames": []string{"projects/" + projectID},
   "filter":        filter,
   "orderBy":       "timestamp desc",
   "pageSize":      1000,
   "pageToken":     nextpageToken,
})

Here, we are preparing the request body to send to the API. We have initialized resourceNames, filter, orderBy, pageSize and pageToken. Apart from the resourceNames, other fields are optional. The pageSize determines the maximum number of results to return from the request and must not exceed 1000 or the request will be rejected. The pageToken retrieves the next batch of results from the preceding call to this method. For the first request, the nextpageToken is set to empty.

requestBody := bytes.NewBuffer(postBody)
request, err := http.NewRequest("POST", "https://logging.googleapis.com/v2/entries:list", requestBody)
if err != nil {
   err = fmt.Errorf("unable to create request: %v", err)
   return nil, err
}

request.Header.Add("Content-Type", "application/json")
request.Header.Add("Authorization", "Bearer "+token.AccessToken)

client := &http.Client{}
response, err := client.Do(request)
if err != nil || response.StatusCode != http.StatusOK {
   err = fmt.Errorf("unable to fetch logs: %v", err)
   return nil, err
}

This part of the code is used to make a request to the API.

result := struct {
   Entries       []Entry
   NextPageToken string
}{}
if err := json.Unmarshal(body, &result); err != nil {
   err = fmt.Errorf("unable to unmarshal response body: %v", err)
   return nil, err
}

In this section, we have parsed JSON encoded data from the response body and stored it in the struct initialized as result.

entries = append(entries, result.Entries...)
if len(result.Entries) == 0 || result.NextPageToken == "" {
   response.Body.Close()
   break
} else {
   nextpageToken = result.NextPageToken
   response.Body.Close()
}

Finally, we appended the result.Entries to entries of []Entry type just outside the a loop. Then, check whether the length result.Entries is 0 or the result.NextPageToken is empty, if the result matches the condition the loop will break and the function will return the stored entries. On the other hand, if the result.NextPageToken is not empty then the loop inside the function will iterate the same process again.

Now, moving to the main function

func main() {
   if err := godotenv.Load(); err != nil {
      log.Fatal("error loading .env file")
   }
   projectID := os.Getenv("GCLOUD_PROJECT_ID")

   defaultServiceKey, err := filepath.Abs("./defaultApiServiceAccountKey.json")
   if err != nil {
      log.Fatalf("unable to load defaultApiServiceAccountKey.json file while initializing cloud logging: %v", err)
   }

   serviceKeyData, err := ioutil.ReadFile(defaultServiceKey)
   if err != nil {
      log.Fatalf("unable to read defaultApiServiceAccountKey.json file for getting token: %v", err)
   }

   //Reading the service key credential to authorize and authenticate request to API
   tokenSource, err := oauth2google.JWTAccessTokenSourceWithScope(serviceKeyData, "https://www.googleapis.com/auth/logging.read")
   if err != nil {
      log.Fatalf("unable to create token source from service key data: %v", err)
   }

   token, err := tokenSource.Token()
   if err != nil {
      log.Fatalf("unable to get token from token source: %v", err)
   }

   entries, err := ListLogEntries(projectID, token)
   if err != nil {
      log.Fatalf("unable to list log entries: %v", err)
   }

   for _, entry := range entries {
      fmt.Println(entry)
   }
}

main()

In the main function, we define all the required environments for the project and also retrieve the service key credential to authorize and authenticate request to the API. While making a request to the Cloud Logging API, we need to retrieve token from the defaultServiceAccountKey.json file. Then, we call the ListLogEntries() function by passing projectID and token and iterate through the log entries returned while printing each of them in the loop.

Let’s see how the complete code looks:

package main

import (
   "bytes"
   "encoding/json"
   "fmt"
   "github.com/joho/godotenv"
   "golang.org/x/oauth2"
   oauth2google "golang.org/x/oauth2/google"
   "io/ioutil"
   "log"
   "net/http"
   "os"
   "path/filepath"
   "time"
)

type Entry struct {
   TextPayload string
   Timestamp   string
}

func ListLogEntries(projectID string, token *oauth2.Token) ([]Entry, error) {
   name := "stderr"
   keyword := "CUSTOM_LOG"
   lastMonth := time.Now().AddDate(0, -1, 0).Format(time.RFC3339)

   filter := fmt.Sprintf(`logName = "projects/%s/logs/%s" AND textPayload:"%s" AND timestamp > "%s"`, projectID, name, keyword, lastMonth)

   var entries []Entry
   nextpageToken := ""
   for {
      postBody, err := json.Marshal(map[string]interface{}{
         "resourceNames": []string{"projects/" + projectID},
         "filter":        filter,
         "orderBy":       "timestamp desc",
         "pageSize":      1000,
         "pageToken":     nextpageToken,
      })
      if err != nil {
         err = fmt.Errorf("unable to marshal json: %v", err)
         return nil, err
      }

      requestBody := bytes.NewBuffer(postBody)
      request, err := http.NewRequest("POST", "https://logging.googleapis.com/v2/entries:list", requestBody)
      if err != nil {
         err = fmt.Errorf("unable to create request: %v", err)
         return nil, err
      }

      request.Header.Add("Content-Type", "application/json")
      request.Header.Add("Authorization", "Bearer "+token.AccessToken)

      client := &http.Client{}
      response, err := client.Do(request)
      if err != nil || response.StatusCode != http.StatusOK {
         err = fmt.Errorf("unable to fetch logs: %v", err)
         return nil, err
      }

      body, err := ioutil.ReadAll(response.Body)
      if err != nil {
         err = fmt.Errorf("unable to unmarshal response body: %v", err)
         return nil, err
      }

      result := struct {
         Entries       []Entry
         NextPageToken string
      }{}
      if err := json.Unmarshal(body, &result); err != nil {
         err = fmt.Errorf("unable to unmarshal response body: %v", err)
         return nil, err
      }
      entries = append(entries, result.Entries...)
      if len(result.Entries) == 0 || result.NextPageToken == "" {
         response.Body.Close()
         break
      } else {
         nextpageToken = result.NextPageToken
         response.Body.Close()
      }
   }
   return entries, nil
}

func main() {
   if err := godotenv.Load(); err != nil {
      log.Fatal("error loading .env file")
   }
   projectID := os.Getenv("GCLOUD_PROJECT_ID")

   defaultServiceKey, err := filepath.Abs("./defaultApiServiceAccountKey.json")
   if err != nil {
      log.Fatalf("unable to load defaultApiServiceAccountKey.json file while initializing cloud logging: %v", err)
   }

   serviceKeyData, err := ioutil.ReadFile(defaultServiceKey)
   if err != nil {
      log.Fatalf("unable to read defaultApiServiceAccountKey.json file for getting token: %v", err)
   }

   //Reading the service key credential to authorize and authenticate request to API
   tokenSource, err := oauth2google.JWTAccessTokenSourceWithScope(serviceKeyData, "https://www.googleapis.com/auth/logging.read")
   if err != nil {
      log.Fatalf("unable to create token source from service key data: %v", err)
   }

   token, err := tokenSource.Token()
   if err != nil {
      log.Fatalf("unable to get token from token source: %v", err)
   }

   entries, err := ListLogEntries(projectID, token)
   if err != nil {
      log.Fatalf("unable to list log entries: %v", err)
   }

   for _, entry := range entries {
      fmt.Println(entry)
   }
}

Result

Run the program with the following command:

$ go run main.go

You can check the terminal for the log entries which might look something like this:

The fields and structure of the log entries will depend on the format in which they have been stored.

Conclusion

Both method I and method II can be used for the effective retrieval of log entries. If you are looking for the answer to which one is the best among them, the answer might depend upon what you are planning to achieve. For example, if you are trying to retrieve a large number of log entries then, using method I, i.e., Cloud Client Library might not be the best option for you as it iterates through each log to retrieve them which can make the whole process time-consuming and in such scenarios, using Cloud API can be more efficient.

That’s all for this article. Thank you for reading, and I hope this article has been useful.

Go
Golang
Google Cloud Platform
Logging

--

--

Nikhil Shrestha
readytowork, Inc.

Written by Nikhil Shrestha

18 Followers
Writer for

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams