A Love Letter to Lambda Logging
This is the first article in a series on AWS Lambda monitoring from Symphonia, published on The Symphonium. Symphonia is a Serverless and Cloud Technology consultancy based in NYC.
As an author of Java-based AWS Lambda functions, I’ve been generally impressed with the quality of the Java runtime for Lambda, as well as the supporting libraries that AWS provides.
However, the current Java Lambda logging library, aws-lambda-java-log4j, is an outlier, in that it brings an almost five year old version of an unmaintained logging package into your Lambda project as a transitive dependency. This is an unfortunate state of affairs for an otherwise very modern Serverless platform.
In this article, I give some background on logging in Java-based AWS Lambdas, why we might want to use a logging framework, and why that framework should be SLF4J. I go on to introduce a new open source library, lambda-logging, that allows Java Lambda authors to replace aws-lambda-java-log4j while retaining its key benefit.
Lambdas, Java Logging, and the Kafkaesque
Logging in AWS Lambdas is quite straightforward. Any output written to the standard output or error streams (System.out and System.err in Java) will be captured by the runtime and forwarded to Cloudwatch Logs.
Given that simplicity, why would we want to use a logging framework at all? The answer is flexibility. Logging frameworks give us an incredible amount of control over the log output of our programs, and afford far more flexibility than System.out.println statements. In Lambdas, that flexibility helps us write compact functions with a very high business logic to boilerplate ratio.
However, with great flexibility (or even the promise of great flexibility) comes great responsibility. The Java logging ecosystem has been described as a “Kafkaesque mess of complexity that buys you nothing.” The landscape is littered with a profusion of logging frameworks and APIs, which when used properly and exclusively provide great benefit. But, all Java developers with more than a few months of experience can attest to the great pain that comes from trying to build a system using Java libraries that depend on different logging frameworks.
The bottom line here is that most Lambdas will use at least a few 3rd-party libraries, and some of those will use incompatible or undesirable logging frameworks. If we want Lambda logging to be useful, we need to put control of that logging back into the hands of the Lambda developers.
One Framework to Rule Them All
According to the SLF4J website:
The Simple Logging Facade for Java (SLF4J) serves as a simple facade or abstraction for various logging frameworks (e.g. java.util.logging, logback, log4j) allowing the end user to plug in the desired logging framework at deployment time.
Yikes. There’s a lot to unpack in that description — basically, the idea is that you can use SLF4J to route all of the logging in your Lambda function that you *don’t* control (from 3rd-party libraries) into a logging framework that you *do* control and configure. The SLF4J documentation describes how to arrange your Lambda dependencies to achieve that end goal of consolidating and controlling log output.
One Weird Trick
AWS provides the aws-lambda-java-log4j library to enhance Java-based Lambda logging with one specific but very useful feature. The library instructs the Lambda Java runtime to dynamically load the org.apache.log4j.MDC class, and if successful, to insert a value into a mapped diagnostic context (MDC), which allows Log4J to seamlessly add the AWS Request ID to logging statements. All you have to do is add the AWSRequestId conversion specifier to your logging pattern.
This is a very cool bit of functionality, because it allows us to associate Lambda runtime logs with specific incoming events, and makes it easy to find all of the log output for a given request. Props to the AWS team for this thoughtful feature.
Other than that small piece of functionality (a single method call), the library also provides a workaround for a Log4J-only issue in which an exception stack trace cannot be handled by the framework. Lastly, to achieve all of this functionality, the library uses Log4J version 1.2.17, which as of the time of this writing, is nearly five years old.
That dependency issue is especially problematic, because it means that if we use other libraries in our Lambda, or if we prefer a different logging framework, we’re out of luck.
Whatcha gonna do about it?
In addressing this situation, I had several design objectives in mind for a library to replace aws-lambda-java-log4j. In order of importance, those objectives were to:
- Retain the
AWSRequestIdcapability. - Use an existing “logging facade” framework.
- Not complicate the use (from Lambdas) of other 3rd-party libraries that perform logging.
- Provide a reasonable default configuration for logging in Lambdas, without XML. Also, provide a way to override, replace, or ignore that default configuration, because libraries shouldn’t mandate unwanted configuration.
- Maintain or reduce the dependency footprint of logging libraries. Java-based Lambdas are particularly sensitive to large artifact sizes and large numbers of classes.
Announcing lambda-logging
I’m happy to announce that Symphonia has released lambda-logging as an open-source project. You can find the source code and documentation on Github, and Java binaries are available on Maven Central.
lambda-logging is a concise set of Java classes that replaces aws-lambda-java-log4j. It provides a simple SLF4J + Logback-based logging system that’s usable and useful out of the box with no configuration, as well as a base on which to build future Lambda logging capability. It uses SLF4J to satisfy the runtime’s need for an org.apache.log4j.MDC class, as well as to replace any logging APIs used by other libraries. It then uses Logback to actually write log output to System.out.
To use the library with a default logging configuration, simply include the following dependency in your Maven project:
Then, use SLF4J loggers in your Lambda:
This will produce logging output (including AWSRequestId) like this:
lambda-logging uses a service-provider based mechanism for loading a default configuration that makes use of AWSRequestId. That default configuration can be overridden by simply placing a logback.xml file on your Lambda’s classpath.
In addition to being more modern and more flexible, SLF4J and Logback are more performant that Log4J, both in initial configuration parsing as well as during synchronous runtime logging. Lambdas are billed in 100ms increments, so picking a performant logging framework actually results in real cost savings. Logback also supports some great features like Markers and TurboFilters, which allow us to easily filter and route logging statements (and which we will use to great effect in an upcoming project).
The only design objective that we didn’t (yet) meet is the dependency footprint. Logback includes a full implementation of Groovy to handle programmatic configuration, which unfortunately means that SLF4J + Logback are about 828 KB in size. Log4J weighs in at 480 KB, so we’ve almost doubled the size of the logging dependency. Given we’re still only talking about kilobytes, I think that the tradeoff still drastically favors the capability of lambda-logging.
In Conclusion
By encapsulating SLF4J and Logback into a simple library with a sensible default configuration, lambda-logging makes it trivially easy for Lambda authors to get started on the right foot with regard to logging. It also provides a foundation for routing the output of the myriad other logging frameworks that 3rd-party libraries depend on. As an added bonus, it can be used with the standard Clojure tools.logging package without any changes.
In this series’ next article, we’ll see how lambda-logging can be used in concert with another soon-to-be-open-sourced library to help handle metrics logging in Java-based Lambdas.
Stay tuned for the next article in our series on Java-based AWS Lambda monitoring!
