Sending AWS metrics into Prometheus using operator and cloudwatch-exporter in Kubernetes

Add permissions to our AWS user with credentials:

aws iam attach-group-policy \
--policy-arn arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess \
--group-name kops-dev

Create values file for exporter:

cat << EOF > values-cloudwatch-exporter.yml
service:
  annotations:
    prometheus.io/scrape: "true"
EOF

Install exporter:

helm install stable/prometheus-cloudwatch-exporter \
--name prometheus-cloudwatch-exporter \
--namespace kube-system \
--set aws.aws_access_key_id=${AWS_ACCESS_KEY_ID} \
--set aws.aws_secret_access_key=${AWS_SECRET_ACCESS_KEY} \
-f values-cloudwatch-exporter.yml

Create values file for the operator:

cat << EOF > values-operator.yml
prometheus:
  additionalServiceMonitors:
  - name: prometheus-operator-cloudwatch
    endpoints:
    - port: http
      path: /metrics
      interval: 30s
    namespaceSelector:
      matchNames:
      - kube-system
    selector:
      matchLabels:
        app: prometheus-cloudwatch-exporter
serviceMonitors:
  - name: ""
    namespaceSelector:
      any: true
EOF

Install prometheus-operator with Alertmanager and Grafana:

helm install stable/prometheus-operator \
--name prometheus-operator \
--namespace monitoring \
-f values-operator.yml

Go to Prometheus dashboard targets page and check targets:

New target

And now you can get some metrics:

New metrics from exporter