Write-Ahead Log: Ensuring Data Durability and Atomicity
This is my very first article, and I am excited to embark on this new adventure. After several attempts, I have finally managed to start doing something, and I hope it will provide an opportunity to learn new things and review others. Not to mention, it is part of a cool project I am involved in with some colleagues at work, where we discuss more about tech since we are a data product team.
This first article will be about a topic that came up in a discussion at work a few weeks ago at work, and I thought it would be a good thing to discuss. So, let’s get down to business! 😄
🔒 Safeguarding Data with Write-Ahead Logging
Write-ahead logging (WAL) is a database technique used to ensure atomicity and durability, which are two key ACID properties. In addition to ensuring that all changes to a database will be committed even in case of system failures, WAL also plays a crucial role in maintaining data durability. By recording modifications in a log before writing them to the database itself, WAL ensures that the database can recover to a consistent state after a failure. This process of logging changes before applying them helps safeguard against data inconsistencies and supports the integrity of the database. WAL ensures that every modification is captured and correctly applied, giving users confidence in the durability of the data. This topic is a big deal these days, especially in data engineering teams. It’s sort of like an extra layer of protection that is found in many databases.
Atomicity means that all changes are successfully committed to the database, or none of them are applied in case of failure.
Consistency signifies that if a database tries to do something against the rules that were set, it will stop the process and go back to the way things were before.
Isolation treats each transaction as a separate unit and won’t interfere with other transactions.
Durability ensures the changes made to the data are stored and can be recovered if needed.
💪 Crash Recovery
The primary purpose of write-ahead logging is to serve as a crash recovery mechanism, ensuring a database can maintain a consistent state even after a system failure. WAL achieves this by writing the logged data to a durable storage medium, such as a hard disk, in append mode and a sequential format. However, WAL offers more than just recovery capabilities. It also optimizes performance by batching multiple transactions into a single log file, rather than creating a separate file for each transaction. This batching approach prevents potential performance issues when reading the log files during the recovery process. By consolidating transactions into log files, WAL streamlines the recovery process and improves overall system performance. It is worth saying that these log files can even be used in CDC capability.
🚀 Checkpointing
To further enhance efficiency and maintain data consistency, WAL incorporates a technique called checkpointing. Checkpointing involves creating a marker in the log files to indicate a known consistent state of the database. This marker, stored as a checkpoint record, contains crucial information such as the Log Sequence Number (LSN) indicating the stable state. Checkpoints are created at regular intervals and durably stored, typically on a hard disk. During recovery, the system starts applying changes from the checkpoint onward, reducing the need to replay all logged changes to the database, which could cause performance issues. Checkpointing improves recovery efficiency, ensuring data consistency, and minimizing log scanning during the process of applying log transactions to the database.
⏮️ Recovery Process and Log Management
In the event of a system failure after creating a WAL file but before committing it to the main storage, the recovery process comes into play. During recovery, the system reads the WAL files and starts from the last checkpoint to reapply the uncommitted changes. By analyzing the log files and using the checkpoint as a reference point, the recovery process selectively applies the necessary modifications to bring the database back to a consistent state. Redundant modifications are ignored to prevent duplication in this case. Once the recovery is complete, a new checkpoint is typically created to establish a new reference point for future recoveries, optimizing the process and maintaining data consistency even in the face of multiple system failures. This part of the process can be known as the redo phase, where the database applies the changes made before the crash using log files. You can get a high-level understanding of how it works by referring to the visual representation provided in Image 1. In addition to that, WAL can also be utilized during the undo phase, wherein the log files are read, and the transactions are rolled back in the database.
A good demonstration of how it works, along with detailed steps, can be found in an article written by Kevin Sookocheff. This article also brings important concepts like stealing and forcing.
🗂️ Managing Log Files for Efficiency
Effective log file management is essential in WAL-based database systems. Two important techniques come into play: log segmentation and log cleaning with a low-water mark. Log segmentation involves periodically switching to a new log file to prevent excessive file growth and optimize storage utilization. It ensures that log files remain manageable in size and simplifies log management. On the other hand, log cleaning with a low-water mark involves removing obsolete log records that are no longer needed for recovery. By reclaiming disk space occupied by these unnecessary records, it improves storage efficiency and system performance. These techniques play a crucial role in the effective management of log files in WAL-based database systems.
✨ The end
To sum it up, write-ahead logging keeps your data safe, ensuring that even in case of system failures, your database remains consistent and durable.
That’s all, thanks for reading 😊!
