Structured logging in Google Cloud

This post is a part of the How Cloud Logging works series.

A term of the structured logging is in use for quite a long time. With more log analyzing and managing solutions in the market it becomes a widely accepted practice. Many popular solutions that used proprietary formatting migrate to use structured logging.

Structured logging is the practice of implementing a consistent, predetermined message format for application logs that allows them to be treated as data sets rather than text.

There are many different definitions of what structured logging is. At the root of structured logging is the idea of describing each log entry as an object or a record so it can be processed and queried using automated tools. The structured logs are usually ingested using a predetermined text or binary format such as Json, XML or Protocol Buffers (protobuf) and then parsed and stored as objects.

Google Cloud uses structured logging from the start. The logs in Google Cloud are described by the LogEntry structure. As I mentioned it is stored as a protobuf record in the binary format. The logs can be queried and analyzed using Log Explorer or via Logging APIs. It is possible to export log entries to BigQuery or to use Log Analytics for more complex analysis.

Log structure layout

The information stored in structured logs can be roughly divided into three categories:

• Information about the log entry describing when this log entry was created and ingested, its order among other log entries, the “log level” usually referred to as severity, etc.
• Context in which the log was created describing metadata about the environment the log was produced, source code location which generated the log and alike.
• Payload of the log that contains a developer’s defined data and metadata.

In Cloud Logging the fields logName, timestamp, receivedTimestamp, severity, insertId, traceSampled of the LogEntry structure store information about the log entry. The log context is captured in the resource, httpRequest, operation, trace, spanId, sourceLocation fields. The data payload includes a caller’s metadata stored in the labels field in the form of the collection of key-value pairs of strings and the application data which can be stored either as a plain text in the textPayload field or as a Json object in the jsonPayload field. There is a third option to store the application’s log data as a protobuf payload using the protoPayload field. The protobuf payload is usually used by various Google Cloud services. Also it can be ingested only by calling Logging API.

Remember that Structured logs and Structured payload are different. The former can exist without the latter but not vice versa.

How to write logs

When an application sends logs to a remote destination the methods vary depending on the log management solution, the local logging framework and application environment. Cloud Logging exposes Logging v2 API that can be called via REST or gRPC directly or using dedicated Logging framework adapters. Developers can also use Logging client libraries that Google provides (in 7 programming languages). These options ingest logs directly into Cloud Logging backend. In addition, it is possible to leverage logging agents in one of the Google Cloud platforms and to “delegate” the task of the log ingestion to the agents. The platforms that have logging agents are AppEngine, GKE, Cloud Run, Cloud Function, Cloud Build, DataProc and DataFlow. The agent has to be explicitly installed on GCE instances and then it can be configured to support the log capturing too. When running an application on the platform with the logging agent, it is possible to ingest logs into Cloud Logging by simply printing them to stdout or stderr. The agents support capturing a plain text and structured payload. The supported structured payload should be formatted as a single-line Json string. The payload can include special fields that the agent parses and stores into the context information fields(e.g. httpRequest). The following information cannot be provided using the special fields:

• logName field — it is impossible to customize the destination of the log or its name when printing structured payload to stdout
• resource field — the agent populates the resource field by itself. It is not possible to customize this field.

Read the public documentation regarding the supported special fields and other details.

Using the logging agents to ingest logs has certain benefits. I will try to capture the main pros and cons below.

Advantages of ingesting logs using the logging agents

Printing to stdout is simple. It does not require additional libraries or to care about communication latency, possible retrying mechanisms and error handling.

It is fast. Any API call takes time depending on the network latency, backend latency and other factors. Even when the communication part is executed asynchronously, the application should be operational along the time it is required to complete the API call. Printing to stdout is just like calling (almost) any other system call.

It is resource free. API calls require use of resources like CPU, memory, network. When operating on prem it does not matter which component consumes the hardware resources since all applications on the same host get influenced. In cloud environments it is different. Managed environments such as Cloud Run will have dedicated resources allocated for the deployment (e.g. CPU) which are different from resources used by the logging agent.

Concerns when using the logging agents

Formatting requirements for the one-line Json string aren’t easy for implementation as it might seem. Unless a specialized library is used the burden on correct formatting falls on the application’s developers. Though a failure to implement the correct formatting will not lead to the loss of the payload but it probably will result in producing the payload that it is hard to process.

Not all platforms and environments have the logging agent. Neither is there a universal method to identify the logging agent for all platforms. Running such an application on the platform without the logging agent will result in printing logs to stdout with no ingesting of the logs to Cloud Logging backend.

Limitations such as inability to control the log destination might be critical for applications that run in one Google Cloud project but need to ingest logs into a different project.

Developer tools

I mentioned that Google maintains Logging client libraries in 7 languages: Java, Python, NodeJS, Golang, C#, Ruby and PHP. The libraries are maintained on Github under the googleapis organization. The libraries come in two flavors:

• Auto-generated — client libraries that are generated based on the protobuf definition of the API; these libraries provide stubs for implementing low level calls to API using gRrpc transport layer:

• Handwritten — client libraries that are developed and maintained by Google DevRel Engineers which use the auto-generated stubs and provide best practices and improved development experience compared to the auto-generated libraries:

Logging Client Libraries

• Python client library released version 3 with the extended support for auto-populating context metadata, integration with standard Python logging and JSon formatter that can be used with the logging agents. Read more detailed overview of the released features in the what’s new post.
• Java client libraries extended support for auto-populating additional context metadata and featured integration with the logging agent in the JUL and logback adapters. Read more in the Cloud Blog.
• NodeJS client library in releases 9.9+ started supporting a similar set of automatic metadata population for ingested log entries. Recently released post in Cloud Blog explains details.
• Go client library in the release 1.5.0 completed the support for structured logging by adding metadata about source location info, support for W3C context trace and providing support for out-of-process log ingestion. Read more in the dedicated post in Cloud Blog.