Centralised Logs and Alarms from Multiple AWS Accounts

If you have multiple AWS accounts and they are all managed by the same team, centralising logs and alarms could improve operational efficiency. It could also improve security by reducing the number of people who need access to workload accounts.

As is often the case, there are different ways to solve this problem. AWS provides a central logging reference architecture. If this suits your requirements it’s the best place to start; it is well thought out, comes with implementation guidance, a cost estimate, and source code. It streams logs from source accounts into Amazon OpenSearch where they can be explored using Kibana.

This article describes a different approach that also allows alarms to be centralised. In complex environments, it’s easy to lose track of alarms distributed across many accounts. Proliferation of alarms can lead to:

• Time wasted handling false positive alarms,
• Delayed resolution of genuine incidents lost in the noise, and
• Inconsistent alarms across source accounts.

Solution Overview

The solution makes logs, events and metrics available to a single central account from multiple source accounts. These can then be used to trigger alarms or integrate with third-party systems via SNS or EventBridge.

• Logs are streamed through Kinesis Data Streams, ingested into CloudWatch, and can be the subject of metric filters in the central account
• Events are pushed using cross-account delivery (Sending and Receiving Events Between AWS Accounts — Amazon CloudWatch Events)
• Metrics remain in the source account but are made visible to the central account using cross-account cross-region metrics (Cross-account cross-Region CloudWatch console — Amazon CloudWatch). They can then be built into dashboards and trigger alarms in the central account.

Logs

A subscription filter is configured per log group in each source account. These let you specify a destination for the logs, along with a filter pattern, controlling which logs will be sent to the central account. See: Setting up a new cross-account subscription — Amazon CloudWatch Logs. There’s the option of filtering out lower logging levels to reduce costs, for example, TRACE, DEBUG and possibly INFO, depending on how the central repository will be used.

Note: there is a limit of two subscription filters per log group.

The logs are sent to a Kinesis Data Stream and onwards to a Lambda function. The Lambda context contains both the source log group and log stream names, which can derive the desired target stream and log group. If the target stream and group don’t already exist they can be created and the message and timestamp written to them unchanged.

So that we don’t lose track of which source account logs came from, we can pre-fix the target log group name with the source account ID, allowing the source of logs to be easily identified in the central account.

If central logs are to be used purely for alerting, a short retention period can be set in the central account to avoid duplicating data and incurring unnecessary costs.

Alarms can be triggered indirectly from logs by building metric filters.

Log Costs

The main cost incurred for centralising logs in this solution is from CloudWatch, which is billed at $0.57 per GB for data ingestion and $0.03 per GB stored per day. This can quickly add up for larger use cases and will likely exceed other costs combined. There is no charge for subscription filters. Data streams are billed at $0.02 per shard hour and $0.02 per million PUT units. Lambda is billed per millisecond with the total cost likely within the same order of magnitude as the data stream.

Costs can be reduced by selectively filtering logs in the subscription filter, for example removing INFO logs that shouldn’t be triggering alarms.

Costs are for eu-west-1 in June 2022.

Metrics

Metrics are not moved from the source accounts, instead, access is provided from the central account. See Cross-account cross-Region CloudWatch console — Amazon CloudWatch.

Central alarms can then be triggered based on the source account’s metrics.

If you’re deploying alarms using CloudFormation, note that CloudFormation’s metric properties do not include account information, however, metric data queries do allow an AccountId parameter. Cross-account metrics are relatively new so hopefully updating metric properties is on the AWS roadmap.

Since nothing is moved there is no additional cost.

Events

Events can be sent between the event buses of accounts, see: Sending and Receiving Events Between AWS Accounts — Amazon CloudWatch Events. They don’t directly trigger alarms, but if required can be used to trigger notifications or other actions using EventBridge or SNS.

Event Costs

The AWS user guide says:

Events sent from one account to another are charged to the sending account as custom events. The receiving account is not charged. For more information, see Amazon CloudWatch Pricing.

Custom events are charged at $1.00 per million events.

About the Author:
Ed Eastwood is an AWS Architect here at Version 1.