CODEX

AWS Lambda Logging tools and best practices — Python

There are a several articles out there that talk about serverless logging (mostly Lambda) and best practices. In this article, I will try to cover tools and practices for the AWS serverless workload logs (observability) and critical insights they provide to build, plan and scale a reliable architecture from a Well-Architected Framework (AWS-WA framework) point of view.

Before we jump into details, just a quick recap of the fundamental difference between logs, metrics, and events provided by AWS.

Logs can have more than one piece of data in them.

Metrics are specifically focused on just the one data point they measure.

Events are generated whenever something changes in an AWS service

AWS recommends several best practices for observability such as

• structured logging,
• distributed traces, and
• monitoring of metrics.

more details can be found here.

The focus of this article is on the tools themselves, rather than on the theoretical principles.Therefore, I strongly recommend the following tools for monitoring application status:

1. aws-lambda-powertools-python — to log and trace your critical events. Following are some important aspects of powertools logging layer:

• Logging — Structured logging made easier, and decorator to enrich structured logging with key Lambda context details
• Metrics — Custom Metrics created asynchronously via CloudWatch Embedded Metric Format (EMF)
• Tracing — Decorators and utilities to trace Lambda function handlers, and both synchronous and asynchronous functions
• Bring your own middleware — Decorator factory to create your own middleware to run logic before, and after each Lambda invocation

2. aws-lambda-power-tuning — to right-size a lambda function

Perhaps the most important question you must answer is “What do I need to log?” I recommend configuring the following areas for logging and analysis provided by BRIAN CLINE with power tools flavour added to it:

• Use a Logging Library

# use powertools
from aws_lambda_powertools import Logger 
logger = Logger()

As a best practice, AWS Lambda Powertools logging statements are suppressed. use set_package_logger to enable it:

from aws_lambda_powertools.logging.logger import set_package_logger  set_package_logger()

• Include identifying information in log events

from aws_lambda_powertools import Logger
logger = Logger()  
def handler(event, context):  
  order_id = event.get("order_id") # to know which order failed and why
  logger.structure_logs(append=True, order_id=order_id)
  logger.info("Collecting payment")

• Log Errors / Exceptions

from aws_lambda_powertools import Logger 
logger = Logger()  
try:      
   raise ValueError("something went wrong") 
except Exception:      
   logger.exception("Received an exception")

• Be consistent When Logging

cloudwatch sample from powertools

• Take the Time to Really Understand CloudWatch & Logging
• Setup CloudWatch Alarms

A detailed description of each of these configurations can be found here

The final step after understanding Powertools, logging practice is to configure one. Here is a step by step tutorial of configuring powertools with your Lambda functions

Attaching Powertools Lambda Layer

STEP 1:

Just deploy the following app which will automatically create the powertools layer for your lambda functions

aws-lambda-powertools-python-layer

STEP 2:

Use the following arn to attach layer to your lambda function

arn:aws:serverlessrepo:eu-west-1:057560766410:applications/aws-lambda-powertools-python-layer

Honorable mentions

I presume that someone who is reading about logging on AWS would have already aware of the 5 pillars of the AWS-WA framework. So, to give a new perspective of these tools I have attributed each tool to a pillar in the AWS-WA framework. Here it goes.

• Powertools for Overall logs and traces — Operational excellence and Performance Efficiency
• Power tuning — Reliability and cost optimization
• AWS-Xray — Operational excellence

Protecting Log Information

As AWS always recommends, Logging facilities and log information must be protected against tampering and unauthorized access. Administrator and operator logs are often targets for erasing trails of activities.

Common controls for protecting log information include the following:

1. Verifying that audit trails are enabled and active for system components — by using Cloudtrail
2. Implement PoLP to ensure that only individuals who have a job-related need can view audit trail files — by using fine-grain access to IAM users
3. Confirming that current audit trail files are protected from unauthorized modifications via access control mechanisms, physical segregation, and/or network segregation — by using fine-grain access to IAM users
4. Ensuring that current audit trail files are promptly backed up to a centralized log server or media that is difficult to alter — logs storage and analysis using backup s3 bucket and Athena queries to Audit them
5. Verifying that logs for external-facing technologies (for example, wireless, firewalls, DNS, mail) are offloaded or copied onto a secure centralized internal log server or media — logs storage and analysis using backup s3 bucket and Athena queries to Audit them
6. Verifying that regular log reviews are performed for all system components
7. Ensuring that security policies and procedures include audit log retention policies and require audit log retention for a period of time, defined by the business and compliance requirements