Monitor AWS S3 GetObject Events and send notification to Slack using Cloudtrail & Lambda

Have you ever wanted to configure an S3 Event Notification for GetObject and to realise that Amazon S3 Event Notifications does not support it.

In this article, i will be covering how we can create a bespoke solution to track GetObject events using cloudtrail and pipe the events logs to cloudwatch before using cloudwatch logs as a trigger for lambda to execute the notification.

Solution Architecture

In this example, i will be using Slack as a notification channel but you can always customise it to your use case e.g. SNS or Telegram.

Implementation Steps:

• Create a new trail under AWS Cloudtrail Service
• Create a lambda function to read events for Cloudwatch Log Group
• Configure a trigger with metrics filter to trigger the lambda created

Under CloudTrail create a new Trail

Go to CloudTrail → Trails → Create trail

Create a new trail with the default settings

The next step is to enable CloudWatch Logs and take note of the Log group name created.

Enable Cloudwatch Logs

In our use case, we only require cloudtrail to record API activity for Data events and monitor S3 data-event type, filter by readOnly events

Configure Log Events for CloudTrail

Proceed to Step 3 — and create the new Trail.

Create a Lambda Function to Read Events

We will be creating the lambda function (Nodejs 16.x) to invoke Slack Webhook URL. In this example, we will be creating a new IAM Role with the default permission require for lambda.

Create lambda function — Runtime nodejs 16.x

Copy the following code into the lambda function and replace the webhook url.

• The code will read the awslogs from the event.
• Decode the logs and unzip the payload to json format.
• Basic validation to check whether the AWS event object is valid by extracting the principalId and objectKey.
• The code will then do a HTTP Post request to the webhook URL with the notificationData(Do note that the notification data can be customize according to the information you want to display on slack)

const https = require('https');
const zlib = require('zlib');

exports.handler = async (event) => {
    const data = event.awslogs.data
    const compressedPayload = Buffer.from(data, 'base64')
    const jsonPayload = zlib.gunzipSync(compressedPayload).toString('utf8')

    const listDataEvents = JSON.parse(jsonPayload).logEvents

    for (let dataEvent of listDataEvents) {
        const username = JSON.parse(dataEvent.message).userIdentity.principalId
        const objectKey = JSON.parse(dataEvent.message).requestParameters.key

        if (username != '' && objectKey != '') {
            const notificationData = {
                blocks: [
                    {
                        type: 'section',
                        text: {
                            text: `*S3 Object Extracted* Date: *${getDateTime()}`,
                            type: 'mrkdwn',
                        },
                    },
                ],
            };

            const res = await post(process.env.webhook_url, notificationData)
        }
    }

    async function post(url, notificationData) {
        const dataString = JSON.stringify(notificationData)

        const options = {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Content-Length': dataString.length,
            },
            timeout: 1000, // in ms
        }

        return new Promise((resolve, reject) => {
            const req = https.request(url, options, (res) => {
                if (res.statusCode < 200 || res.statusCode > 299) {
                    return reject(new Error(`HTTP status code ${res.statusCode}`))
                }

                const body = []
                res.on('data', (chunk) => body.push(chunk))
                res.on('end', () => {
                    const resString = Buffer.concat(body).toString()
                    resolve(resString)
                })
            })

            req.on('error', (err) => {
                reject(err)
            })

            req.on('timeout', () => {
                req.destroy()
                reject(new Error('Request time out'))
            })

            req.write(dataString)
            req.end()
        })
    }

    function padTo2Digits(num) {
        return num.toString().padStart(2, '0');
    }

    function formatDate(date) {
        return [
            padTo2Digits(date.getDate()),
            padTo2Digits(date.getMonth() + 1),
            date.getFullYear(),
        ].join('/');
    }

    function getDateTime() {
        const today = new Date();
        const date = formatDate(today)
        const time = padTo2Digits(today.getHours()) + ":" + padTo2Digits(today.getMinutes()) + ":" + padTo2Digits(today.getSeconds());
        return date+' '+time;
    }
};

For the code change to take effect, we will have to redeploy the lambda function.

Example of redeploying the lambda function

Configure a cloudwatch trigger with metrics filter to trigger the lambda

With reference to the image above, click on the Add trigger button. Under the Log Group field — Search for the Log Group created earlier in Step 1. Take note of the filter name as we only want to monitor GetObject event.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html

Trigger Configuration

Trigger successfully added

We will be able to see our lambda function being updated with CloudWatch Logs as a trigger.

Conclusion

Let’s verify our implementation to see if the logs are created from cloudtrail and send over to lambda.

• Go to a S3 bucket and open/download any object.
• You should be able to see a log stream created under the cloudwatch logs provision in the CloudTrail step.
• Within a minute, you will be able to see the Slack Alert.

Sample Slack Alert

With these 3 steps, you can now easily monitor your S3 content for any GetObject events. You can refer to the entire terraform code in my github account, if you are familiar with infrastructure as code. Do comment below or reach out to me via LinkedIn if you need any clarification on the implementation.