Importing Logs from Google Cloud Platform (GCP)

Cloud logging

Logs are essentially records of events, activities, and transactions that take place within a computer system or application. They might include everything from error messages, status updates, user actions, and security alerts to resource consumption metrics. These humble lines of text, often written in cryptic codes and numbers, serve a critical purpose in the world of IT, offering insights and clues into the inner workings of software, servers, networks, and more.

We need logs for various reasons as:
i> Debugging and Troubleshooting

ii> Performance Monitoring

iii> Security and Compliance

iv> Historical Records

In this article, we will dive into the process of importing logs from Google Cloud Platform (GCP). At first, let us assume you have setup for Golang and server running in GCP.

Let us first create a basic unique log in the server in our API as

cc.logger.Zap.Info(fmt.Sprintf("[API_CUSTOM_LOG] ["Success"] ["Fetch_user"] ["GET"]));

This will create a server log in GCP logging section.

We should put such logger in each api as we want.Now we need to fetch such logs from the GCP to our server. And we can manipulate as we want, as saving to database, or just monitor it.

Before fetching the log , we need to give the access to the “firebase-adminsdk” user we are using in GCP for “Logging Admin”

Now let us create a function to retrieve the logs with “API_CUSTOM_LOG” in GCP logs

func (cc UserLogController) LogsFromGCP() {
// get accessToken
 defaultServiceKey, err := filepath.Abs("./serviceAccountKey.json")

 if err != nil {
  cc.logger.Zap.Panic("Unable to load serviceAccountKey.json file for getting short lived token")
 }
 serviceKeyData, err := ioutil.ReadFile(defaultServiceKey)
 if err != nil {
  cc.logger.Zap.Panic("Unable to read serviceAccountKey.json file for getting short lived token")
 }

 var serviceAccountKey struct {
  ProjectID string `json:"project_id"`
 }

 // Unmarshal the JSON data into the struct
 if err := json.Unmarshal(serviceKeyData, &serviceAccountKey); err != nil {
  panic("Unable to parse serviceAccountKey.json")
 }

 // Access the project_id field
 projID := serviceAccountKey.ProjectID

 search_keyword := "[API_CUSTOM_LOG]"
 filterQuery := fmt.Sprintf(`textPayload:"%s" AND timestamp >= "%s"`, search_keyword, from_timestamp.Format(time.RFC3339))
//we can manipulate as our requirement for filterQuery

 tokenSource, err := oauth2.JWTAccessTokenSourceWithScope(serviceKeyData, "https://www.googleapis.com/auth/logging.read")
 if err != nil {
  cc.logger.Zap.Panic("Unable to create token source from service account file")
 }
 token, err := tokenSource.Token()

 if err != nil {
  cc.logger.Zap.Panic("Unable to get token from token source")
 }
 cc.logger.Zap.Info("Token: ", filterQuery)
 var entries []logtypes.CustomEntry
 nextPageToken := ""

 for {
  post_body, err := json.Marshal(map[string]interface{}{
   "resourceNames": []string{"projects/" + projID},
   "filter":        filterQuery,
   "orderBy":       "timestamp desc",
   "pageSize":      1000,
   "pageToken":     nextPageToken,
  })

  if err != nil {
   cc.logger.Zap.Error("Error marshelling json: ", err.Error())
   return
  }
  req_body := bytes.NewBuffer(post_body)
  req, _ := http.NewRequest("POST", "https://logging.googleapis.com/v2/entries:list", req_body)
  req.Header.Add("Content-Type", "application/json")
  req.Header.Add("Authorization", "Bearer "+token.AccessToken)

  client := &http.Client{}
  response, err := client.Do(req)

  if err != nil || response.StatusCode != http.StatusOK {
   errorMsg := fmt.Sprintf("Error making HTTP request: %v, Status Code: %v", err, response.StatusCode)
   fmt.Println(errorMsg)
   cc.logger.Zap.Error(errorMsg)
   return
  }
  
  body, err := ioutil.ReadAll(response.Body)
  if err != nil {
      cc.logger.Zap.Error("Error reading response body:", err.Error())
      return
  }

  result := struct {
   Entries       []logtypes.CustomEntry
   NextPageToken string
  }{}
  if err := json.Unmarshal(body, &result); err != nil {
   cc.logger.Zap.Error("Error unmarshelling reponse body of user logs")
   return
  }
  entries = append(entries, result.Entries...)

  cc.logger.Zap.Info("Entries count:", len(entries))
  if len(result.Entries) == 0 || result.NextPageToken == "" {
   response.Body.Close()
   break
  } else {
   nextPageToken = result.NextPageToken
   response.Body.Close()
  }

//here all logs are in entries array
 

}

Here, after all the setup and serviceAccountKey setup, we can find all the logs in the entries array. We can monitor it, and save it to the database, as per the requirement.

Things to keep in mind- Logs can be in huge numbers, suggested applying enough filterQuery during fetching the logs from GCP.

Hope it helps.
Happy coding 🙏