Automate CloudWatch Custom Metrics like Memory and Disk Utilization on Windows EC2 instance using Lambda function
As we are seeing, nowadays industries are growing and everyone wants to save their time, so everyone’s focus is on automation. As we know that AWS provided so many default metrics, but what if we want to monitor Disk and Memory utilization of our EC2 instance, As per AWS we have to manually configured and install CloudWatch agent on our EC2 instance and again we have to manually create a dashboard
What if I say that we can do all these things in just a single click and in 3 to 5 minutes?
Yes, it is possible……
So Let’s go for the 3 steps which you have to follow, in a single click, the EC2 instance will be ready with a Dashboard showing your Memory and Disk Utilization.
We are going to create this flow in our step Function
Let’s Start the Process
- Create A Windows EC2 instance
Step I: Create an IAM role with required permissions. We will use this role to Attach to our Windows EC2 instance.
Make sure to attach all these permission to your EC2 instance else your configuration of cloudwatch will not work as expected.
Step II: Create Lambda Function with these permissions
Step III. Create AWSLambda Basic customer managed policy with this code
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:eu-central-1:accountname:*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": [
"arn:aws:logs:eu-central-1:accountname:log-group:/aws/lambda/loggroupname:*"
]
}
]
}Make sure to Attach all these permissions because without these your Lambda will not Execute and start your CloudWatch Configurations.
Step IV: Create Lambda Function using below code
import boto3
def lambda_handler(event, context):
# Initialize the EC2 client
ec2 = boto3.client('ec2')
# Specify the parameters for creating the EC2 instance, including user data
instance_params = {
'ImageId': 'ami-id',
'InstanceType': 't3.large',
'KeyName': 'name', # Specify the key pair name (not the .pem file)
'MinCount': 1,
'MaxCount': 1,
'UserData': '''<powershell>
$url = "https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/amazon-cloudwatch-agent.msi"
New-Item -Path "C:\\Users\\Administrator\\Downloads" -ItemType Directory -Force
$outputPath = "C:\\Users\\Administrator\\Downloads\\amazon-cloudwatch-agent.msi"
Invoke-WebRequest -Uri $url -OutFile $outputPath
Start-Process -FilePath "msiexec.exe" -ArgumentList "/i", $outputPath, "/quiet" -Wait
& "C:\\Program Files\\Amazon\\AmazonCloudWatchAgent\\amazon-cloudwatch-agent-ctl.ps1" -m ec2 -a start
</powershell>''',
'NetworkInterfaces': [
{
'DeviceIndex': 0,
'SubnetId': 'subnet-id',
'Groups': ['sg-id']
}
],
'IamInstanceProfile': {
'Name': 'already-created-role-name'
},
'TagSpecifications': [
{
'ResourceType': 'instance',
'Tags': [
{
'Key': 'Name',
'Value': 'insatnce-name'
}
]
}
]
}
# Create the EC2 instance
response = ec2.run_instances(
ImageId=instance_params['ImageId'],
InstanceType=instance_params['InstanceType'],
KeyName=instance_params['KeyName'],
MinCount=instance_params['MinCount'],
MaxCount=instance_params['MaxCount'],
UserData=instance_params['UserData'],
NetworkInterfaces=instance_params['NetworkInterfaces'],
IamInstanceProfile=instance_params['IamInstanceProfile'],
TagSpecifications=instance_params['TagSpecifications']
)
# Extract the instance ID
instance_id = response['Instances'][0]['InstanceId']
return {"instance-name_Instance_id":instance_id}2. Considering the Wait time of 5 minutes
As we know that after the agent starts it will take at least 3 to 5 minutes to reflect metrics in our CloudWatch Dashboard, so for waiting we have to add one wait step function which will wait at least 5 minutes before creation of CloudWatch Dashboard.
3. Creating CloudWatch Dashboard
Step 1: Now create one more Lambda Function with these code to create Dashboard
import json
import boto3
def create_cloudwatch_dashboard(cloudwatch,dashboard_name,instance-name_instance_id):
dashboard_body = {
"widgets": [
{
"type": "metric",
"properties": {
"metrics": [
[ "AWS/EC2", "CPUUtilization", "InstanceId", instance-name_instance_id, { "label": "instance-name - CPUUtilization", "region": "eu-central-1" } ], [ ".", "NetworkIn", ".", ".", { "label": "instance-name - NetworkIn", "region": "eu-central-1" } ], [ ".", "CPUUtilization", ".", "." ],
],
"view": "timeSeries",
"stacked": True,
"region": "ap-south-1",
#"title": "instance-name EC2 CPU {}".format( ),
"period": 300
}
},
#i have updated from this
{
"type": "metric",
"properties": {
"metrics": [
[ "CWAgent", "LogicalDisk % Free Space", "instance", "C:", "InstanceId", instance-name_instance_id, "objectname", "LogicalDisk", "ImageId", "ami-0fb974a4772b174a5", "InstanceType", "t3.large" ], [ ".", "Memory % Committed Bytes In Use", "InstanceId", instance-name_instance_id, "objectname", "Memory", "ImageId", "ami-0fb974a4772b174a5", "InstanceType", "t3.large" ]
],
"view": "timeSeries",
"stacked": False,
"region": "ap-south-1",
#"title": "DetectionEngine EC2 - Disk {}".format( ),
"period": 300
}
},
#to this
{
"type": "metric",
"properties": {
"metrics": [
[ "CWAgent", "LogicalDisk % Free Space", "instance", "C:", "InstanceId", instance-name_instance_id, "objectname", "LogicalDisk", "ImageId", "ami-id", "InstanceType", "t3.large" ]
],
"view": "timeSeries",
"stacked": False,
"region": "ap-south-1",
#"title": "instance-name EC2 - RAM {}".format( ),
"period": 300
}
},
]
}
cloudwatch.put_dashboard(DashboardName=dashboard_name, DashboardBody=json.dumps(dashboard_body))
def lambda_handler(event, context):
# Parameters
# auth_name = "cust-" + str(event["customer_number"]) +"-"+str(event["name"])
dashboard_name = 'Cloudwatch-Dashboard'
instance-name_instance_id = event["Detectionengine_Instance_id"]
# Create aws session
session = boto3.Session(region_name="ap-south-1")
# Create cloudwatch client
cloudwatch = session.client('cloudwatch')
create_cloudwatch_dashboard(cloudwatch, dashboard_name, instance-name_instance_id)
return {
"message": "EC2 instance with ID instance-name_instance_id has been created."
}Note: In Above code replace your instance name, region name and ami-id with respective details.
Check your dashboard, it will look exactly like this
Conclusion: We used Lambda to Automatically install and Configure CloudWatch Agent on Windows EC2 instance and automation to create Dashboard with Memory and Disk Utilization. The complete process will save your time and rework of manually configuring CloudWatch on EC2 instance.
