Insufficient data: CloudWatch alarm based on custom metric filter

This is one of those things that were difficult to google, so leaving a note here.

TL;DR — if CloudWatch alarm based on custom metric filter is stuck in “INSUFFICIENT DATA” state, make sure you don’t use UNIT nor DIMENSION settings in its definition.

AWS CloudWatch gives you a very useful capability to define “metric filters” — basically you define a regular expression, and a value that corresponds to finding a match in a log. One of common cases could be counting occurrences of a term (“ERROR”, for example) in your log — metric value is then 1 and total value of the metric filter informs you how many occurrences of the term there were in your logs. AWS cli for defining such metric filter can look like:

aws logs put-metric-filter
 — log-group-name /aws/lambda/example
 — filter-name ERROR
 — filter-pattern ERROR
 — metric-transformations metricName=allErrors,metricNamespace=logMetrics,metricValue=1,defaultValue=0

Once you define a metric, you may want to define an alarm based on it. In my case, I tried to do so using ansible ec2_metric_alarm module. I copied definition from another project and filled in the template:

ec2_metric_alarm:
  region: ...
  aws_access_key: ...
  aws_secret_key: ...
  security_token: ...
  state: present
  alarm_actions: ...
  name: ...
  metric: ...
  namespace: ...
  dimensions: ...
  statistic: ...
  comparison: ...
  threshold: ...
  period: ...
  evaluation_periods: ...
  unit: ...

However, I ended up with an alarm in “INSUFFICIENT DATA” state and couldn’t figure out what was wrong. Metric filter had reasonable values and all the graphs in AWS Console looked OK.

Turns out (after reading some forums) that if you define an alarm based on metric filter, it should not have UNIT nor DIMENSIONS defined. Getting rid of those two fields from list of properties fixed the definition and alarm started working as I’d expect.