Using OTEL community edition to leverage logging into AWS CLOUDWATCH
In the previous story we understood how to use opentelemetry and we integrated aws cloudwatch with otel collectors .
But we also talked about one drawback there if you remember ,aws otel collector right now don’t support pushing logs to cloudwatch , only metrics and traces are allowed . We also discussed that community edition can be used to leverage shipping logs to cloudwatch . So I have tried it out to test it out and yeah it worked out.
. Here is the community repo for opentelemetry
. In the repository u will be able to see configurations for exporters of different vendors here is the exporter that I am using for cloudwatch,similarly you will be seeing configurations setup for receivers , processors and extensions too.
So if u have read through the previous article , you already know how to install the otel on eks , so lets just see how to utilize otel-contrib edition for exporting cloudwatch logs .
All you need to do for exporting logs is just reconfigure the collector with community edition and also add right exporters and pipelines
otelcollector.yaml
---
$ kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: sidecar-for-my-app
spec:
image: public.ecr.aws/aws-observability/aws-otel-collector:latest
mode: sidecar
config: |
extensions:
health_check:
pprof:
endpoint: 0.0.0.0:zzzz
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:xxxx #(grpc endpoint on which my pod/api sends logs, so our sidecar listens to this port and collects from here)
http:
endpoint: 0.0.0.0:yyyy #(http endpoint on which my pod/api sends logs, so our sidecar listens to this port and collects from here)
processors:
batch:
exporters:
logging:
loglevel: debug
awsxray:
region: <region>
awsemf:
log_group_name: <log-group-name>
log_stream_name: <log-stream-name>
namespace: <namespace>
region: <region>
awscloudwatchlogs:
log_group_name: <container_log_group_name>
log_stream_name: <container_log_stream_name>
service:
pipelines:
traces:
receivers: [oltp]
exporters: [awsxray]
metrics:
receivers: [otlp]
exporters: [awsemf]
logs:
receivers: [otlp]
processors: [batch]
exporters: [awscloudwatchlogs]
extensions: [pprof]
telemetry:
logs:
level: debug
EOF
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: myapp
annotations:
sidecar.opentelemetry.io/inject: "true"
spec:
containers:
- name: myapp
image: someimage:latest
ports:
- containerPort: 8080
protocol: TCP
EOF
So using this sidecar configuration and adding the sideacar to ur pod/deployment should be goood enough and you will be able to see logs going into loggroup you mentioned in aws cloudwatch