How to enable CloudWatch logging with Symfony on AWS

I’ve recently found myself needing to send application logs to AWS CloudWatch via a Symfony API container running in AWS Fargate. After some research and some trial and error I found a solution that is easy to get setup.

Getting Started

Before you can successfully send logs to CloudWatch you will need an AWS Key and AWS Secret token pair for an account that has the following IAM Roles enabled:

"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"logs:CreateLogStream",
"logs:CreateLogGroup",

Ideally a new user would be created specifically with these roles enabled.

Requirements

• An understanding of AWS IAM roles
• An application sitting within AWS infrastructure
• Maxbanton’s brilliant lib
• AWS SDK for PHP

Creating a new Monolog config

By default Symfony sends applications logs to /var/log/dev.log however we can adjust this by providing a new Monolog config file.

Create a new file named monolog.yaml under config/packages/staging/

You could create distinct configurations for dev and prod if you wanted to, by creating a monolog.yaml file in the appropriate directories. For this example I am setting a config file specifically for our staging environment.

Once created add the following example configuration:

monolog:
    handlers:
        main:
            type: fingers_crossed
            action_level: warning
            handler: cloudWatch
        cloudWatch:
            type: service
            id: cloudwatch_handler

Here we are replacing the default Monolog handler with our own named cloudWatch

Note: The cloudwatch_handler does not exist yet, but we are about to create it.

This will stop applications logs being written to files, if you want to still have the ability to write to logs, as well as CloudWatch you can try the following

monolog:
    handlers:
        main:
            type: fingers_crossed
            level: debug
            handler: grouped
        grouped:
            type: group
            members: [cloudWatch, local]
        local:
            type: stream
            path: "%kernel.logs_dir%/%kernel.environment%.log"
        cloudWatch:
            type: service
            id: cloudwatch_handler

Here we set a grouped handler which details multiple handlers, one for CloudWatch and the other for local, file based logging.

Creating the logging service

Now we’ve told Monolog that we want to use a custom handler called cloudwatch_handler we need to create it. To do this I used registered a new logging service

Within service/config/services.yaml add the following configurations:

cloudwatch_client:
    class: Aws\CloudWatchLogs\CloudWatchLogsClient
    arguments:
        -
            credentials: {
                key: "%env(resolve:AWS_CLOUD_WATCH_KEY)%",
                secret: "%env(resolve:AWS_CLOUD_WATCH_SECRET)%" 
            }
            region: "eu-west-1"
            version: "latest"

Here we register an instance of the AWS CloudWatchLogsClient. This is where we need to insert our AWS user key and secret. In this example I am getting the details from the .env file.

Adjust the region key here to match your requirements. If you have no specific regional requirements you can leave it as eu-west-1.

Leave the version key set to latest, unless you have specific version requirements.

In the same file add the following:

cloudwatch_handler:
    class: Maxbanton\Cwh\Handler\CloudWatch
    configurator: ['@json_log_configurator', configure]
    arguments:
        - "@cloudwatch_client"
        - "my-log-group"         # groupName
        - "my-log-stream"        # streamName
        - 30                     # retentionDays
        - 1                      # logsInBatch
        - { mytag: "tag" }       # tags
        - WARNING                # logLevel

Here we register the custom Monolog handler we referenced in the monolog.yaml config earlier. The CloudWatch class listed above extends the Monolog AbstractProcessingHandler.

The configurator option is used to overwrite or augment the formatter used by Monolog. In the example above I have supplied a custom method which I will discuss below.

The first argument provided must be an instance of the AWS CloudWatchLogsClient which we setup above. The arguments here are largely down to your own project requirements, adjust the log group name and the stream name as required.

Note: If using a specific IAM user for CloudWatch it may be tied to a specific log group. If that is the case make sure the log group name and stream name arguments are correct and match the IAM users permissions

In the same file add the following:

json_log_configurator:
    class: App\Utils\CloudWatchJsonConfigurator

By default Monolog will send any errors in plain text to CloudWatch. It can be nice to have these errors sent as JSON for ease of searching and filtering of logs.

Here we pass in a custom configurator which sets the Monolog formatter to default to JSON. I have put this custom class within Utils but feel free to move this if required, just ensure you update the class path.

Creating the custom configurator

It’s possible to augment or override the formatter used by Monolog. in this example we are simply setting the formatter used to be that of JSON, however you could make any other adjustments here that you needed.

<?php

namespace App\Utils;

use Monolog\Formatter\JsonFormatter;
use Monolog\Handler\AbstractProcessingHandler;

/**
 * Configure a given Monolog handler to format messages as JSON
 */
class CloudWatchJsonConfigurator
{
    /**
     * Format logs being sent to CloudWatch to JSON
     *
     * @param AbstractProcessingHandler $handler
     **/
    public function configure(AbstractProcessingHandler $handler)
    {
        $handler->setFormatter(new JsonFormatter());
    }
}

Thats it! That’s all you need to start sending applications logs to CloudWatch. Deploy these changes to your infrastructure and generate some logs.

I’m sure there a few ways of doing this. Please do let me know any other solutions you know of that I could play around with.