Sending Windows 2012 logs to CloudWatch

This Blog has moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

Why should we send logs to Cloudwatch?

This helps to centralise logs to a secure, durable and controllable place. Centralising logs has following benefits.

Reduces the risk of losing data

If the instance of interest is down, local log files won’t be accessible. Cloud watch log agent ensures that logs are continuously shipped from the instances and stored in a secure and durable place. Also, a cloud watch log can be migrated to S3 for long term retention.

Easier Troubleshooting

When your system grows to multiple hosts, managing the logs and accessing them can get complicated. Searching for a particular error across hundreds of log files on hundreds of servers is difficult and cloudwatch logs provides an easy interface to search logs across the instances.

Better Forensic Ability

When a system is compromised logs are no longer trusted. Centralized logs give you the forensic ability to determine what happened right before the compromise, including any user activity.

Enhanced Security

Since this log is accessible to through AWS Console so we can control permissions using AWS IAM, an only relevant administrator can be delegated for view and analyze the log.

Configuration of Windows 2012

You can configure your Amazon EC2 instances to send Windows Server logs, events, and performance counters to Amazon CloudWatch Logs. Amazon EC2 offers several methods for configuring your instances to export this data. Amazon EC2 instances use SSM agents to send log data to CloudWatch. Create a .json document and copy it to each instance. Updates in the configuration files should be copied to each instance manually. The SSM agent on the instance starts sending data to CloudWatch within a few minutes.

I have created a sample .json file copied to the instance in the specified path to perform the following tasks.

AMI: Windows_Server-2012-R2_RTM-English-64Bit-Base-2018.03.24 (ami-d8b9e2b7)

EC2Config:4.9.2565.0

SSM Agent: 2.2.355.0

1) Send the logs from the Microsoft Windows 2012 EventViewer to cloud watch. Here we consider System, Security, Application and EC2Config Logs. This can be customized further to accommodate more EventViewer based logs.

2) Send Windows Performance Monitor metric to cloud watch. We considered Available Memory, Free Space in C:\ and Free Space in D:\.

3) Send the logs from the custom application which is not-an- EventViewer based. Tested to send logs from a directory C:\CustomLogs to Cloudwatch.

Ref: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartWindows20082012.html

Configurations involves the following steps:

1. Make sure that your instance is added to log relevant IAM role.

2. Edit the supplied .json file based on the requirement. See sample .json file.

3. Copy the sample .json file to the instance as C:\Program Files\Amazon\SSM\Plugins\awsCloudWatch\ AWS.EC2.Windows.CloudWatch.json

4. Execute the following command to start the service or restart the service after the modification of the file. This has to be done on a Powershell command prompt.

PS C:\> Restart-Service AmazonSSMAgent

Syntax: AWS.EC2.Windows.CloudWatch.json File

The AWS.EC2.Windows.CloudWatch.json file has five Sections.

1. Log source section: Settings to input Windows EventViewer log to a log data stream.

2. Log destination section: To configure settings to output the captured log data to CloudWatch Logs.

3. Metric source section: Settings to input Windows Performance Monitor data to a metric-data stream.

4. Metric destination section: To configure settings to captured output metric-data to CloudWatch Logs.

5. Configure Flow Control. Each source input log/metric stream must have a corresponding destination in the Flow control section. Multiple sources can be mapped to a single relevant destination.

The below sample .json file has Five log types and three metrics sent to cloud watch. Log types are EC2ConfigService, Security, System, Application and Custom Application Logs(Non-EventViewer). Metrics are free memory, C:\ free space and D:\ free space.

Sample .json File

{
  "IsEnabled": true,
  "EngineConfiguration": {
    "Components": [
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "EC2ConfigServiceEventLog",
        "Parameters": {
          "Levels": "4",
          "LogName": "EC2ConfigService"
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "ApplicationEventLog",
        "Parameters": {
          "Levels": "4",
          "LogName": "Application"
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "SystemEventLog",
        "Parameters": {
          "Levels": "7",
          "LogName": "System"
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.EventLog.EventLogInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "SecurityEventLog",
        "Parameters": {
          "Levels": "7",
          "LogName": "Security"
        }
      },
{
        "FullName": "AWS.EC2.Windows.CloudWatch.CustomLog.CustomLogInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "CustomLogs",     
         "Parameters": {
        "LogDirectoryPath": "C:\\CustomLogs\\",
        "TimestampFormat": "MM/dd/yyyy HH:mm:ss",
        "Encoding": "UTF-8",
        "Filter": "",
        "CultureName": "en-US",
        "TimeZoneKind": "Local",
        "LineCount": "5"
    }
      },
{
        "FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "PerformanceCounter",
        "Parameters": {
          "CategoryName": "Memory",
          "CounterName": "Available MBytes",
          "DimensionName": "InstanceId",
          "DimensionValue": "{instance_id}",
          "InstanceName": "",
          "MetricName": "AvailableMemory",
          "Unit": "Megabytes"
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
        "Id": "CloudWatchLogs-System",
        "Parameters": {
          "AccessKey": "",
          "LogGroup": "Tensult/System",
          "LogStream": "{instance_id}",
          "Region": "ap-south-1",
          "SecretKey": ""
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
        "Id": "CloudWatchLogs-Application",
        "Parameters": {
          "AccessKey": "",
          "LogGroup": "Tensult/Application",
          "LogStream": "{instance_id}",
          "Region": "ap-south-1",
          "SecretKey": ""
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
        "Id": "CloudWatchLogs-Security",
        "Parameters": {
          "AccessKey": "",
          "LogGroup": "Tensult/Security",
          "LogStream": "{instance_id}",
          "Region": "ap-south-1",
          "SecretKey": ""
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
        "Id": "CloudWatchLogs-EC2config",
        "Parameters": {
          "AccessKey": "",
          "LogGroup": "Tensult/EC2Config",
          "LogStream": "{instance_id}",
          "Region": "ap-south-1",
          "SecretKey": ""
        }
      },
{
        "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatchLogsOutput,AWS.EC2.Windows.CloudWatch",
        "Id": "CloudWatchLogs-CustomLogs",
        "Parameters": {
          "AccessKey": "",
          "LogGroup": "Tensult/CustomLogs",
          "LogStream": "{instance_id}",
          "Region": "ap-south-1",
          "SecretKey": ""
        }
      },
{
        "FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "PerformanceCounterDiskC",
        "Parameters": {
          "CategoryName": "LogicalDisk",
          "CounterName": "% Free Space",
          "DimensionName": "InstanceId",
          "DimensionValue": "{instance_id}",
          "InstanceName": "C:",
          "MetricName": "C.FreeDiskPercent",
          "Unit": "Percent"
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.PerformanceCounterComponent.PerformanceCounterInputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "PerformanceCounterDiskD",
        "Parameters": {
          "CategoryName": "LogicalDisk",
          "CounterName": "% Free Space",
          "DimensionName": "InstanceId",
          "DimensionValue": "{instance_id}",
          "InstanceName": "D:",
          "MetricName": "D.FreeDiskPercent",
          "Unit": "Percent"
        }
      },
      {
        "FullName": "AWS.EC2.Windows.CloudWatch.CloudWatch.CloudWatchOutputComponent,AWS.EC2.Windows.CloudWatch",
        "Id": "CloudWatch",
        "Parameters": {
          "AccessKey": "",
          "NameSpace": "Windows/Default",
          "Region": "ap-south-1",
          "SecretKey": ""
        }
      }
    ],
    "Flows": {
      "Flows": [
        "(EC2ConfigServiceEventLog),CloudWatchLogs-EC2Config",
        "(SystemEventLog),CloudWatchLogs-System",
        "(SecurityEventLog),CloudWatchLogs-Security",
        "(ApplicationEventLog),CloudWatchLogs-Application",
         "(CustomLogs),CloudWatchLogs-CustomLogs",
"(PerformanceCounter,PerformanceCounterDiskC,PerformanceCounterDiskD),CloudWatch"
      ]
    },
    "PollInterval": "00:00:15"
  }
}

Screenshots