Structured Logging — Part 1: What’s the Big Deal

Originally published at blog.current.sh on September 3, 2015 by @evanphx.

Here at current.sh, we’re passionate about logs. Everyone has them but knowing how to manage them — and how to use them to your advantage — takes skill.

We’re aiming to help users make their logs contain more value than just giant walls of text. One of the primary ways we’re doing that is by promoting the concept of structured logging.

More than just plain text

Typical logs are bits of text that have been meagerly and haphazardly formatted by a particular human in the misty past. In fact, the lack of consistent formatting in logs has spawned more than one programming language (e.g. Perl) whose express purpose is helping later humans try to make sense out of the data.

Rather than leave the task of formatting up to some later human, structured logging has logs come out already formatted in a way that can be easily parsed and used. The emergence of JSON in the last decade has given computing a light-weight syntax that is suited wonderfully for this task, but even a simple “key=value” format can bring huge benefits with extremely low buy-in.

Taming a log explosion

The need to better use logs as data is partially driven by the movement toward services as a necessary deployment abstraction. As the logs are emitted by different services in different places, operators need to pull the logs and piece them back together to create a coherent view of what actually took place.

Add in things like queueing and the general push to decouple services and the requirement of quality logs grows even higher. An error might be logged hours after some data was mishandled by another service; the ability to follow the flow of that data is critical.

Structured logging gives operators consistent tools and techniques that can be applied to the logs. In many cases, without structured logging, trying to figure out what happened would prove too difficult to be bothered with at all. Instead teams would just cross their fingers and hope that it didn’t happen again.

Moving beyond operations

Typically, logs have been thought about as a side-effect of running an application that only concerns the ops team. Examples of this would include logs detailing HTTP requests, emails being sent, and packages installed.

While these are all useful domains, logs can and should move deftly up the stack into the business domain. Logging this kind of data can be extremely useful to both the development as well as the product teams. Logging business details such as user=”evan” feature=”new_widget” activated=true can provide not only metrics about a feature but also an audit trail of how a user got to that feature. As teams are figuring out how to improve their applications, these audit trails provide real, actionable data about how users interact with the site.

As when the logs are formatted structurally, it becomes extremely easy to query them and derive information from the results.

Databases?

There are already existing solutions for this: for example, I personally have seen teams use audit database tables for this task. But calling these kinds of tables unwieldy would be an understatement: they’re huge, take up significant resources, and require constant care. Additionally, they’re very brittle: it’s difficult to add new columns (hello migration times) and adding new audits for tables only adds to the burden.

Instead, we believe that utilizing logging infrastructure and techniques is a far better way to manage this kind of data. Structured logging gives form to the data while remaining flexible. The infrastructure is designed to deal with an infinite stream of data and thusly can expand and expire data as necessary.

The Future

As more people move toward structured logging, we’ll see more tools, more services, and more best practices. The movement toward structured logging is almost inevitable at this point as teams see the value in the data locked up in those logs.

Next Time

In the next post, we’ll talk about structured logging formats and what exactly should be logged.