How to configure the Unified CloudWatch Agent for collecting Logs and Custom Metrics.

This Blog has moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

The crucial part of every log management system is alerting. With the AWS inbuilt service CloudWatch, you can do both log management as well as alerting using metrics. In CloudWatch, user has the option to set up alerts using default metrics as well as custom metrics. Default metrics like CPU Utilisation, InstanceStatusCheck, etc. may not be sufficient for predicting your infrastructure health especially if you are working with scalability scenarios. This is where custom metrics such as RAM utilization, disk utilization, etc. comes into picture. Earlier, the process of generating custom metrics used to be a cumbersome process especially in the case of Linux instances as you have to download packages, install and execute n number of commands. More details on the manual installation can be found in my other blog.

In this blog, I will be explaining the process of setting up the unified CloudWatch Agent to collect custom metrics and logs. As this is a fairly advanced topic, I am assuming that the user has a fair understanding on AWS and CloudWatch service. If not, to understand the capabilities of the service do check out our blogs on CloudWatch here and here.

Now let’s jump into the implementation details. Since we are using SSM (Systems Manager) to configure the log agent to the instance there are some prerequisites to the process.

Prerequisites

1. SSM agents should be installed. To install the SSM agent visit the AWS documentation here. For some instances like Amazon Linux, Ubuntu Server 16,18, etc. SSM agent is available by default.
2. IAM Role with EC2 read-only and SSM full access should be attached to the instance. To create roles visit the AWS documentation here.

Fig 1: Create IAM Role with following the permission

Configuring the Unified Agent

1. Login to your AWS account and browse to the service SSM. Once inside the service on the left pane, under actions click on ‘Run a Command’ option.

Fig 2: Click on Run a Command

2. You will get a list of Documents to choose from, in which select the document ‘AWS-ConfigureAWSPackage’. Continue to configure the parameters.

Fig 3: Select AWS-ConfigureAWSPackage

3. Keep the document version as ‘Default version at runtime’. Under the command parameters select the Action as ‘Install’ and for the Name field give the name ‘AmazonCloudWatchAgent’.

Fig 4: Add the command parameters

4. If the prerequisites are successfully configured you will be able to see the instances listed under the targets. Here in this example, I will be using a Linux based CentOS distribution. The procedure is the same for the Windows instances as well.

Note: Document can be pushed to multiple instances at the same time

Fig 5: Select the instance(s) you want to push the agent

5. You can also point the execution logs to an S3 bucket or CloudWatch logs, to inspect them in case of a failed execution but this step is not mandatory. Similarly, you can also configure SNS notifications for executions.

Fig 6: Enable writing to S3 bucket or CloudWatch logs (not mandatory)

6. Now that all this information is given, Run the command and wait for few seconds to get the success message and you are good to go.

Fig 7: Execution completed successfully

7. Now we have to log on to the instance and create the configuration file for the agent. For Linux instances run the following command to open the configuration wizard,

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

For Windows instances go to the following directory and run the file,

cd "C:\Program Files\Amazon\AmazonCloudWatchAgent"
amazon-cloudwatch-agent-config-wizard.exe

Once you launch the wizard, configure the settings as per your requirements. The wizard configuration is pretty straight-forward you can create both custom metrics and logs. For logs you need to configure the path, for this example I will be using ‘var/log/messages’.

Fig 8: Configure log agent as required

8. Once everything is configured push the data to ‘SSM Parameter Store’ and keep note of the name, as we will be using this configuration file as a source shortly. Here in this example I will be using the name ‘AmazonCloudWatch-Centos’.

Fig 9: Push to SSM parameter store

You can verify the parameter store to see if the file is reflected there. You should be in the same region to which you have pushed.

Fig 10: SSM parameter store

9. Now that you have successfully created the configuration file, we need to configure the agent using another SSM document. For this, go to the SSM service and click on ‘Run a Command’ option and select the document ‘AmazonCloudWatch-ManageAgent’ as we did earlier.

Fig 11: Select AmazonCloudWatch-ManageAgent

10. Under the Command parameters select the configuration required and make sure to enter the configuration file name under ‘Optional Configuration Location’. The configuration will be taken from the file we have created and stored in the parameter store.

Fig 12: Configure Command Parameters

11. Once the document is successfully executed you will be able to see the custom metrics as ‘CWAgent’ under CloudWatch Metrics. Logs will be populated under CloudWatch logs.

From here, alarms can be created using the generated custom metrics as per your requirements. To setup CloudWatch alarms visit my other blog.

Fig 13: Generated custom metrics

Similarly, the logs that are generated can be found inside the specified log group.

Fig 14: Generated log files

With the unified agent its a much easier and hassle-free process to configure custom metrics and logs for both linux and windows environment. One useful scenario that comes to my mind is that, when we are creating AMIs for mass deployments ‘unified agent’ can be pre-installed so that you can save lots of time in configuring logs and alarms.