Serverless SlackOps: Real-time error notifications with deep-linking to Cloudwatch Logs.

When building a first serverless application on AWS Lambda, many engineers quickly realize that they have to rethink their logging and metrics collection strategies. Due to the runtime model of AWS Lambda, you can no longer run your usual logging daemon of choice. Fortunately, AWS Lambda ships some excellent basic metrics and logging out of the box via Cloudwatch which go a long way to solving usual logging and metrics use cases.

At my current startup, I have embraced the idea of ‘SlackOps’: using Slack as a first tier support, monitoring, and even alarming tool. For example, we have platform integration such that high-value customer activities, such as signups, are reported in a slack channel. We also conduct customer support chat on our website using a 3rd party service that surfaces the chat to us through Slack. ALL team communication then ends up in Slack which is very convenient.

Combining the above two ideas, I extended the platform to report errors to slack as they occur. This is all well in good, but anyone who has used Lambda+Cloudwatch Logs knows that log diving is not always an easy experience. This prompted me to explore whether the platform could deep link to Cloudwatch Logs.

As it turns out, it was possible. The Lambda context object contains the name of the log group and log stream. Taking these, along with the current time, and forming a Cloudwatch Logs Console URL allows easy, real-time deep linking to the Cloudwatch Console. The demonstrative NodeJS code snippet below demonstrates the basic idea, using a fictitious ‘postNotification’ method to post a notification with the deeplink.

var AWS = require('aws-sdk');
module.exports.handler = function(event, context, callback)
{
  var link= https://console.aws.amazon.com/cloudwatch/home?region=REGION#logEventViewer:group=LOG_GROUP;stream=LOG_STREAM;start=START";
  link = link.replace(“REGION”, AWS.config.region); 
  link = link.replace(“LOG_GROUP”, context.logGroupName);
  link = link.replace(“LOG_STREAM”, context.logStreamName);
  link = link.replace(“START”, new Date().toISOString());
  
  postNotification("An Error Occurred:" + link);
  callback("error"); 
}

In my actual implementation, I wrap the toplevel callback with functionality that determines, for every call, whether or not to post to Slack and to generate the notification text itself. I also use Slack formatting so a nice link is used instead of the raw URL.

I hope this post helps you better leverage Cloudwatch Logs in your serverless application monitoring. Of course, this is usable in any sort of notification, not just Slack.

Do you have any tips for monitoring your serverless application? Are there facets of serverless application development that you find difficult or frustrating that you’d like me to write about? Let me know in a response below!

Disclaimer: The Cloudwatch Logs URL is reverse-engineered from my own use of the console. These deeplinks could theoretically go away or change at anytime