Java Logging on Cloud Run with Stackdriver

In the age of microservices, adequate logging and tracing are even more critical for monitoring and debugging your services. When working with containers, there’s an added layer of complexity due to the flexibility and ephemerality of containers. A user must think about how to get info “out of the container” by mounting a persistent volume. However you’re in luck! Similar to other Google Cloud Platform (GCP) compute products, the Stackdriver logging agent is built into the Cloud Run environment, so your logs automatically get piped and consolidated into Stackdriver.

This blog post will cover Java logging on Cloud Run, though these concepts can be applied to the Google App Engine (GAE). We will discuss how different log types look in Stackdriver and the associated product Error Reporting, along with how to upgrade your logs with trace Ids.

Let’s start at the beginning. Any logs written to standard out and standard error will show up in your Stackdriver logs, but without any additional information or functionality.

System.out.println(“This log was made with System.out.println”);
System.err.println(“This log was made with System.err.println”);

This is the same functionality that you will get when using frameworks such as java.util.logging, log4j2, or slf4j without any additional configuration. With all of these frameworks, we can start with some basic logging tips:

1. Set the severity of your log message. Setting the severity can help you filter your logs, visually find log entries, and set up metrics and alerts.
2. Add a clear and actionable error message. Don’t hate yourself later when you’re trying to debug.
3. Add a stack trace (as a Throwable). Any bit of additional information on where the error is coming from is better than none!

utilLogger.log(
    Level.INFO, “Error report INFO with java.util.logging”, e);
utilLogger.log(
    Level.SEVERE, “Error report ERROR with java.util.logging”, e);
slf4jLogger.info(“Error report INFO with Logback”, e);
slf4jLogger.error(“Error report ERROR with Logback”, e);

Passing a throwable means you get the benefits of Stackdriver Error Reporting! Error Reporting aggregates and displays errors produced in your running cloud services in order to provide real-time exception monitoring and alerting.

For App Engine, Google offers additional library support for logging to Stackdriver with the Logback appender and JUL (java.uti.logging) handler. That allows for the log level to be correctly parsed as well as additional configuration to be added like severity thresholds, log name, and enhancers. The default configuration on GAE would result in these logs in the Stackdriver console:

Unfortunately, Cloud Run has a slightly different runtime environment from App Engine; therefore these integrations don’t work out of the box. You can easily regain this functionality by upgrading to JSON formatted logs using Logback, which means you can continue to use its native interface, Simple Logging Facade for Java (SLF4J).

Your Logback configuration must append to the console. Here we use the appender ch.qos.logback.core.ConsoleAppender. Then using the encoder net.logstash.logback.encoder.LogstashEncode allows for the transformation to JSON format.

<configuration>
  <appender name="jsonConsoleAppender"
      class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <fieldNames>
        <timestamp>[ignore]</timestamp>
        <version>[ignore]</version>
        <logger>[ignore]</logger>
        <thread>[ignore]</thread>
        <levelValue>[ignore]</levelValue>
      </fieldNames>
    </encoder>
  </appender>
  <root level="INFO">
    <appender-ref ref="jsonConsoleAppender"/>
  </root>
</configuration>

This configuration ignores additional default fields that are unneeded. The dependencies needed are:

<dependency>
  <groupId>net.logstash.logback</groupId>
  <artifactId>logstash-logback-encoder</artifactId>
  <version>5.2</version>
</dependency>
<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-classic</artifactId>
  <version>1.2.3</version>
</dependency>

Using JSON as your log format also allows for structured logging, which means you can add more readable/parseable information to your logs for debugging purposes. The traditional way is to use the Mapped Diagnostic Context (MDC), provided by all major logging frameworks. In order for Stackdriver to parse the log level, we must add it as the severity field.

MDC.put(“severity”, “ERROR”);
logger.error(“Logging ERROR with Logback”);
MDC.remove(“severity”);

Using the logstash-logback-encoder library allows for event specific custom fields, which means we can add key-value StructuredArguments to our JSON logs. This makes it easier and cleaner to add multiple log-specific fields.

logger.info(“Logging INFO with Logback”,
            kv(“severity”, “NOTICE”),
            kv(“component”, “arbitrary-property”));

Using StructuredArguments or MDC, we can also append the trace Id to the log using the specific JSON field logging.googleapis.com/trace. Learn more about special fields for structured payloads.

String traceHeader = req.headers("x-cloud-trace-context");
String trace = traceHeader.split("/")[0];
logger.info("Logging with trace Id",
           kv(“severity”, “DEBUG”),
           kv(“logging.googleapis.com/trace”,
           String.format(“projects/%s/traces/%s”, project, trace)));

In the Stackdriver logs viewer, logs correlated by the same trace are viewable in “parent-child” format: when you click on the triangle icon at the left of the request log entry, the container logs related to that request show up nested under the request log.

Logback is also the default logging configuration for Spring Boot applications and using the logging starter with JSON appender makes it easy to integrate with GCP. Using this appender resource, it is unnecessary to specify the severity as a part of the log message.

<configuration>
  <include resource=
 "org/springframework/cloud/gcp/logging/logback-json-appender.xml"/>
   <root level=”INFO”>
       <appender-ref ref="CONSOLE_JSON"/>
   </root>
</configuration>

Don’t leave logging to the last minute! Stackdriver gives you a centralized system to access all your logs across your GCP services. Get the most out of your logs and Stackdriver by using structuring your logs in JSON and adding contextual metadata for easier troubleshooting.

Next Steps

• Check out the official documentation for logging and viewing logs on Cloud Run.
• Learn more about creating charts and alerts in Stackdriver.
• Follow a tutorial on local troubleshooting of a Cloud Run service.