At LogDNA, we’re all about speed. We need to ingest, parse, index, and archive several terabytes of data per second. To reach these speeds, we need to find and implement innovative solutions for optimizing all steps of our pipeline, especially when it comes to storing data.
Our architecture makes heavy use of Kubernetes since it lets us deploy, manage, and scale quickly. However, we quickly ran into bottlenecks when attempting to store data for use in pods. To maximize efficiency and reduce costs, we needed to make use of local storage.
Kubernetes makes it easy to manage containers at scale. It also introduces complexity in the number of moving parts; containers, backend services, and even nodes change constantly.
When the pods are evicted, crashed, deleted or scheduled on a different node, the logs from the container is gone. This is different than logging on traditional servers or virtual machines. When your apps die on your virtual machine, you don’t lose the logs until you delete it. In Kubernetes, it cleans up after itself and the logs will not persist. Understanding the ephemeral nature of default logging on Kubernetes is important because…