Logging, monitoring, and alerting with CloudWatch Events
It is essential to keep track of our application’s backend, know whether the product is working correctly, and examine its further growth. When necessary, we need to set up alarm systems from custom logs to notice errors or warning situations and take action in these cases.
For example, we have a dispersed architecture. We built our product from microservices written in multiple different programming languages. Their relations with each other are significant for the healthy operation of the product. In this case, following all these services from a common point will make our work much more accessible and save us more time to take action.
In this context, we can collect all our logs in a specific stream name we created with AWS CloudWatch, which is among the AWS services. We can write search queries according to the time interval we want, when necessary, and set up monitoring dashboards and alarms about them.
So what kind of logging architecture might we need here?
If we consider a simple structure;
- We can send our logging events from all our applications to an AWS SQS with AWS-SDK.
- From this AWS SQS, we can export data to AWS CloudWatch with a lambda invoked with batch packages.
- If we want, in case of any write error, we can send our event back to the same AWS SQS and thus reduce the log loss to zero.
- We can write custom queries and prepare AWS CloudWatch Dashboards with our data.
- We can search with our log parameters on AWS CloudWatch Insights for the period we want.
- We can set up AWS CloudWatch Alarms for matching custom words or sentences and receive these alarms as e-mail notifications via AWS SNS Topic.
Let’s have a look at the code and AWS Console screenshots that will bring this simple architecture to life together.
Step 1 — Sending messages to AWS SQS from our applications
We can continue with a few different programming language examples. You can check this AWS documentation for other language support if you want.
1- After importing AWS-SDK for the appropriate language, we need to use the required AWS SQS library.
2- The region and version information are essential for defining our SQS Service.
3- We can prepare a log payload thoroughly according to ourselves.
Tip: If we set up a parametric structure, it will be easier for us to write a query on the AWS CloudWatch Insights side.
4- We can enter MessageBody, and QueueUrl values and send.
Tip: The DelaySecond parameter is not a must. We should add this parameter to determine the time it takes for AWS Lambda invoked from AWS SQS to read the message. If our message load is too much, we can add this parameter to avoid throttle when sending to AWS CloudWatch.
PHP
You can find more AWS Documentation for the PHP language.
Javascript
You can find more AWS Documentation for the Javascript language.
Tip: If you proceed with credentials for javascript, you can continue from this document.
Step 2— Lambda invocation by AWS SQS messages, put events to stream
We are now going through how to send AWS CloudWatch events with a Javascript example. For other languages, see; CreateLogGroup, CreateLogStream, PutLogEvents.
1- First of all, we import AWS-SDK.
2- We set to lambda with AWS CloudWatch variables. This lambda will be invoked with batch message data from AWS SQS.
3- We call our logging method.
4- We first need to create the AWS CloudWatch client and register it in our logging method.
5- If you continue with AWS IAM User, we need to enter your user access credentials.
Tip: Not required if you continue without AWS IAM Role permission.
6- We must determine the log stream name according to our microservice or according to the features of our product. We will include every log message we send in this stream name.
For detailed information about Sequence Token, see. PutLogEvents
If you don’t want to deal with writing it yourself, you can use ready-made packages for this. Here are a few examples of NPM packages;
- https://www.npmjs.com/package/winston-cloudwatch
- https://www.npmjs.com/package/send-cloudwatch-log
Step 3— Following AWS CloudWatch Events
1- We enter the AWS CloudWatch service via the AWS Console, and we can see our group by searching for our group name in the log groups from the left panel. If you want, you can set the expiration date of your logs in the retention section.
2- We will see our log stream in the log group.
3- The log stream includes our log message as JSON.
Step 4— Queryable log messages in AWS CloudWatch Insights
1- We click on the log insights from the left panel.
2- We select our log group from the select box.
3- We can write a query by our log message fields.
fields @timestamp, @message
| filter `event-type` = 'info'
| sort @timestamp desc
| limit 20
Step 5— Export to AWS CloudWatch Dashboard by query
1- Let’s open the dashboard creation pop-up using the “Add to dashboard” button.
2- Let’s select “Create new dashboard” and enter the dashboard name, then press the “Create” button.
3- Let’s choose the chart type of our query result from the widget type section. You can see the chart preview in the “Preview” section.
4- Let’s press the “Add to dashboard” button, and we will be direct to our new dashboard.
Step 6— Create a custom AWS CloudWatch Metric and Alarms
First, we must create an AWS SNS topic to receive mail from AWS CloudWatch alarms.
1- We enter the AWS SNS service via the AWS Console. Press “Topics” on the left. Press the “Create Topic” button.
2- We can select the standard type and write the topic name to inputs.
3- On our topic page, we press to “Create subscription” button.
4- We can select the protocol and write our e-mail address.
5- We should confirm the subscription with the received mail to us.
6- We enter the AWS CloudWatch service and go to the “All metrics” in the left panel.
7- We search for our log group by name and choose which metric we want to create.
8- Then, we click on the alarm icon from the “Graphed metrics” tab.
9- We select metric conditions by period.
10- We can check our metric conditions, selected notification topics, and alarm name/description from the last preview step.
11- We click the “Create alarm” button and complete the process.
End of the logging structure
We have come to the end of our simple logging structure. We tried to create step-by-step logging, monitoring, and alerting architecture together. I hope it was an enlightening article. It is up to you to understand the logic and apply the best method for your application architecture.
Note: Even if all the steps are applied one-to-one, an integrated architecture will not occur. The code snippets and screenshots in the document are for informational purposes only. A certain level of AWS and development proficiency is required to build the architecture.
Thanks for reading.
