Setup Loki Storage to AWS S3 and DynamoDB in Kubernetes with Kiam
Grafana + Prometheus + Loki is used by us as a monitoring system with dashboards, monitoring, alerting, and logging for our Kubernetes clusters.
At first, Loki stores its data (logs and index data) on the persistent volume (PV) inside the Kubernetes clusters where an EBS with 1TB (in the beta environment) is attached to the cluster assigned to the PV.
Why Changing to S3?
After running Loki for a while in our clusters, it is considered to move the storage of Loki from K8s PV (EBS) to S3. There are some points being considered.
Flexibility in Scale
In these days I worked with Loki, the full disk issue happened several times, which means the Loki data reaches the limit of disk definition of the PV (1TB). Then, the Loki service stops working completely.
Loki provides configuring retention period¹ but unfortunately, it doesn’t provide a retention logic by disk size² yet. Therefore, a week-long retention period does make sense for troubleshooting issues.
When using PV with EBS in the Kubernetes clusters, it is treated as a local mounted disk, so a target disk size has to be provided when claiming the PV. Then, it is hard to tell how big to fit Loki data with the…
