How Logs Work

Distributed Services with Go — by Travis Jeffery (25 / 84)

👈 The Log Is a Powerful Tool | TOC | Build a Log 👉

A log is an append-only sequence of records. You append records to the end of the log, and you typically read top to bottom, oldest to newest — similar to running tail -f on a file. You can log any data. People have historically used the term logs to refer to lines of text meant for humans to read, but that’s changed as more people use log systems where their “logs” are binary-encoded messages meant for other programs to read. When I talk about logs and records in this book, I’m not talking about any particular type of data. When you append a record to a log, the log assigns the record a unique and sequential offset number that acts like the ID for that record. A log is like a table that always orders the records by time and indexes each record by its offset and time created.

Concrete implementations of logs have to deal with us not having disks with infinite space, which means we can’t append to the same file forever. So we split the log into a list of segments. When the log grows too big, we free up disk space by deleting old segments whose data we’ve already processed or archived. This cleaning up of old segments can run in a background process while our service can still produce to the active (newest) segment and consume from other segments with no, or at least fewer, conflicts where goroutines access the same data.