How to configure AWS elastic beanstalk custom metrics on example of memory metrics

Hello, my name is Alex. Recently, I encountered a task where a business was using AWS Elastic Beanstalk but was struggling to understand the system state due to the lack of comprehensive metrics in CloudWatch. By default, CloudWatch only provides a few basic metrics such as CPU and Networks. However, it’s worth noting that Memory and Disk metrics are not included in the default metric collection.

Fortunately, each Elastic Beanstalk virtual machine (VM) comes with a CloudWatch agent that can be easily configured to collect additional metrics. For example, if you need information about VM memory consumption, which AWS does not provide out of the box, you can configure the CloudWatch agent to collect this data. This can greatly enhance your visibility into the performance and health of your Elastic Beanstalk environment, allowing you to make informed decisions and optimize your application’s performance.

To accomplish this, you’ll need to edit your Elastic Beanstalk zip bundle and include a cloudwatch.config file in the .ebextensions folder at the top of your bundle. Please note that the configuration file should be chosen based on your operating system, as described in this article. By doing so, you’ll be able to customize the CloudWatch agent settings and enable the collection of additional metrics, such as memory consumption, to gain deeper insights into your Elastic Beanstalk environment. This will allow you to effectively monitor and optimize the performance of your application on AWS.

Linux-based config

files:
  "/opt/aws/amazon-cloudwatch-agent/bin/config.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "agent": {
          "metrics_collection_interval": 60,
          "run_as_user": "root"
        },
        "metrics": {
          "append_dimensions": {
            "InstanceId": "$${aws:InstanceId}"
          },
          "metrics_collected": {
            "mem": {
              "measurement": [
                "mem_total",
                "mem_available",
                "mem_used",
                "mem_free",
                "mem_used_percent"
              ]
            }
          }
        }
      }
container_commands:
  apply_config_metrics:
    command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

Windows-based config

files:
  "C:\\Program Files\\Amazon\\AmazonCloudWatchAgent\\cw-memory-config.json":
    content: |
{
        "agent": {
          "metrics_collection_interval": 60,
          "run_as_user": "root"
        },
        "metrics": {
          "append_dimensions": {
            "InstanceId": "$${aws:InstanceId}"
          },
          "metrics_collected": {
            "mem": {
              "measurement": [
                "mem_total",
                "mem_available",
                "mem_used",
                "mem_free",
                "mem_used_percent"
              ]
            }
          }
        }
      }

container_commands:
  01_set_config_and_reinitialize_cw_agent:
    command: powershell.exe cd 'C:\Program Files\Amazon\AmazonCloudWatchAgent'; powershell.exe -ExecutionPolicy Bypass -File ./amazon-cloudwatch-agent-ctl.ps1 -a append-config -m ec2 -c file:cw-memory-config.json -s; powershell.exe -ExecutionPolicy Bypass -File ./amazon-cloudwatch-agent-ctl.ps1 -a start; exit

As you may have noticed, I enabled only a few memory-related metrics such as mem_total, mem_available, mem_used, mem_free, and mem_used_percent. However, you can enable more metrics as needed. The complete list of available metrics can be found here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html

Once you have updated your application, it would be beneficial to create a CloudWatch dashboard to visualize these metrics. To do so, navigate to the AWS CloudWatch console, select Dashboards, and click on Create dashboard. From there, you can create a widget by clicking the Add widget button and selecting Line to create a line chart that displays the desired metrics. Customizing a dashboard with relevant metrics can provide valuable insights into the performance and health of your Elastic Beanstalk environment, making it easier to monitor and optimize your application on AWS.

In case the example that you saw above we’ll see 5 new metrics in section CWAgent

Based on them we may configure memory widget and get something like that.

Feel free to explore the wide variety of metrics and AWS widgets available in CloudWatch to further customize your dashboard. If you have any questions or need assistance, feel free to reach out to me via Telegram at @kvendingoldo. I’m happy to help!