Mastering Serverless (Part V): Logging Techniques for AWS Lambda

Logging is a critical component in serverless computing, especially when your product is in production. It’s often underestimated during development, but its value becomes evident in a live environment. Logs are like a compass in the complex sea of production, guiding you through issues and understanding application behavior under various conditions. A consistent, well-structured logging framework is not just an add-on; it’s essential. It simplifies debugging, provides performance insights, and maintains a healthy serverless environment. This part of our Master Serverless series delves into effective logging, highlighting its fundamental role in serverless architecture.

AWS Powertools

The Powertools for AWS Lambda (TypeScript) Logger documentation offers a comprehensive guide on structured JSON logging, capturing Lambda context, and adjusting logging levels and formats. It covers essential features like capturing Lambda context fields, logging Lambda invocation events, log sampling, and structured log enhancements. The documentation also includes guidance on installation, usage, setting service names, logging levels, sample rates, custom log formatters, testing, and Lambda context handling in logs. For a detailed exploration of these concepts, visit the AWS Lambda Powertools TypeScript Logger documentation.

Implementation

Initialization

To streamline my logging process, I developed a centralized initialization file. This file sets the appropriate log level and specifies the service name for my logger. By doing this once, I can easily import this configuration into any file that requires logging. This approach is more efficient and maintains consistency across different parts of the application.

export const MsaLogger = new Logger({
    logLevel: process.env.LOG_LEVEL,
    serviceName: process.env.LOG_SERVICE_NAME
});

Using It

Initially, my logging approach involved adding attributes directly in the message field. However, this method lacked scalability. To enhance consistency and ease of log searching, I transitioned to adding an object as the second parameter in the logger calls. This method allows for more structured and searchable logs, greatly improving the logging process.

logger.info("Analysis", {data: analysisParams});
logger.error("DynamoDB Put error", {data: e})

Logging Examples

In the logging examples, I’ll demonstrate how various attributes are logged and how they appear in CloudWatch, providing insights into the effectiveness of this structured approach.

{
    "cold_start": true,
    "function_arn": "arn:aws:lambda:us-east-1:XXXXXXXXXX:function:prefix-api-prod-get-data",
    "function_memory_size": 1024,
    "function_name": "prefix-api-prod-get-data",
    "function_request_id": "3232db26c-89fa-4e27-b542-8cc410d17545",
    "level": "INFO",
    "message": "DynamoDB Get Item",
    "service": "prefix-prod-logger",
    "timestamp": "2024-01-25T22:14:43.485Z",
    "xray_trace_id": "1-65b2dd52-4703eeb20e4324cc25f393c8",
    "data": {
        "TableName": "prefix-prod-data",
        "Key": {
            "SupporterUuid": "91db5a232-b0bb-417b-a7ce-ec6e4f3a57e7",
            "Username": "admin"
        }
    }
}

CloudWatch

Amazon CloudWatch is a monitoring and observability service designed for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. It provides data and actionable insights to monitor applications, understand and respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing a unified view of AWS resources, applications, and services that run on AWS and on-premises servers.

Query Generator

The “Query Generator” is a transformative feature that streamlines the process of querying logs. This functionality stands out as an innovative tool for those who manage and analyze extensive serverless logs, especially in complex cloud environments.

How It Works: The Query Generator operates on a simple yet powerful principle: Describe what you want to find in your logs, and it crafts the query for you. This user-friendly interface allows you to input a descriptive request, like “Show me all error logs for Service X between 2 PM and 4 PM yesterday.” The Query Generator then translates this description into a precise query syntax compatible with your log group.

Benefits:

1. Ease of Use: Eliminates the need for extensive knowledge of query syntax, making log analysis accessible to a broader range of users.
2. Time Efficiency: Speeds up the process of log analysis, allowing for quicker responses to issues and more time for other tasks.
3. Precision: Reduces errors in query construction, ensuring more accurate results and insights from your logs.
4. Customization: Adapts to various log formats and structures, offering versatility across different serverless architectures.

Applications

The Query Generator is particularly useful in scenarios where rapid log analysis is crucial. For instance, in incident response, it helps quickly pinpoint issues. In performance monitoring, it assists in identifying trends and anomalies without deep diving into complex query languages.

Conclusion

Effective logging has been pivotal in resolving numerous issues, notably in instances where clients reported missing payments. Through meticulous logging and querying, I was able to trace these transactions, identifying the exact point of deletion and the responsible parties. This capability has not only solved operational issues but also had significant legal implications. It underscores that without a robust logging strategy, a product is not truly ready for deployment. While CloudWatch is a powerful tool for this purpose, alternatives like DataDog and other solutions also offer comprehensive logging capabilities, catering to different needs and environments.

Other Articles You May Be Interested In:

New Projects or Consultancy

• Shoot me an email at one@upskyrocket.com

Advanced Serverless Techniques

• Advanced Serverless Techinques I: Do Not Repeat Yourself
• Advanced Serverless Techniques II: Streamlining Data Access with DAL
• Advanced Serverless Techniques III: Simplifying Lambda Functions with Custom DynamoDB Middleware

Mastering Serverless Series

• Mastering Serverless (Part I): Enhancing DynamoDB Interactions with Document Client
• Mastering Serverless (Part II): Mastering AWS DynamoDB Batch Write Failures for a Smoother Experience.
• Mastering Serverless (Part III): Enhancing AWS Lambda and DynamoDB Interactions with Dependency Injection
• Mastering Serverless IV: Unit Testing DynamoDB Dependency Injection With Jest
• Mastering Serverless (Part V): Advanced Logging Techniques for AWS Lambda