Monitoring Log File for Its Idleness

Over the weekend, I have a requirement to monitor an application on Kubernets. The application is kind of a black box, the only clue we have is that when the application is working fine, it spits a lot of log entries. When it hangs there is no log activity.

Thanks to Golang, this unconventional monitoring logic can be easily achieved.

Read from Pipe

Let’s start the reading of log entries from Stdin. The expected usage is to use a shell pipeline to get the log streams.

func reading(logChan chan string) {
 reader := bufio.NewReader(os.Stdin)

 for {
  line, err := reader.ReadString('\n')
  if err != nil {
   log.Fatalf("Failed to read line. err:%v", err)
  }
  logChan <- line
 }
}

This simple function reads through the Bufio reader line by line forever. If there is no log entry or no new line, the reading is blocked. The line read is sent to a string channel for further processing.

Monitor Log

With the Golang channel magic, the log idleness can be easily detected. The main function is listed below.

func main() {
  var conf Config
  if _, err := toml.DecodeFile("config.toml", &conf); err != nil {
    log.Fatalf("Failed to parse config.toml:%v", err)
  }

  logPattern := regexp.MustCompile(conf.LogErrorPattern)
  
  logChan := make(chan string)
  go reading(logChan)

  for {
    select {
    case logLine := <-logChan:
      if logPattern.MatchString(logLine) {
        log.Printf("Log pattern matched: %s", logLine)
        go notify(conf, true, logLine)
      }
    case <-time.After(time.Duration(conf.Threshold) * time.Second):
      log.Printf("timeout detected...")
      go notify(conf, false, "")
    }
  }
}

I set some configurations in the Toml file. The program starts to load those configurations.

Then we initiate a string channel, kick off the reading goroutine with this channel.

Followed that is the common Golang idiom, an infinite loop to select any channel activities. Upon entering Select, the channel operations on the case statement are all evaluated in order. So here our timer starts. If there are any log activities, the program will go down to the case branch for the respective processing, and the timer branch will be ignored in the current Select loop. If there is no log channel activity till the timeout value, the program will run the timeout branch, we detected the idleness. Finally, after the branching, the program goes back to the loop.

If we have any log entry picked up by the reader() and sent to the “ logChan” channel, we receive it and do the traditional log matching stuff, notify if match the pattern.

For idleness, we simply call notify() function for notifications.

Sample Usage

I compiled the program as logMon, then we can monitor the application of idleness as below.

kubectl logs -f --tail=200 mypod | ./logMon