Object Stores: The Case for Snapshots vs Object Versioning

6 min readJun 25, 2024

Overview

The purpose of this blog post is to highlight the snapshot feature for object stores while comparing it to a somewhat similar feature called object versioning. Contemporary object stores provide versioning for individual objects. While Object versioning serves use cases that require maintaining older versions of the objects, it creates its own set of new challenges for the users and applications. This blog will highlight some of these challenges associated with individual object versioning and then move on to discussing the object store snapshot feature and how it addresses those challenges. This blog is part of the blog series on Apache Ozone Snapshots. Subsequent blogs will discuss the snapshot feature in the context of Apache Ozone object store and will provide more details.

Object Versioning

Object versioning is a feature where every update to an object creates a new version of the object. All versions of the objects are preserved by the object store. If the object gets deleted accidentally, it is possible to restore the object to any of its older versions. Some object stores allow the ability to auto expire the older versions of the objects based on some criteria.

What’s wrong with Object Versioning

Object Versioning is a great feature for preserving individual objects and keeping the history of the objects around. Object versioning works well for the use cases where the application is looking at objects in their individuality. However there are real world applications where its utility is limited. Let’s look at some of these challenges posed by this feature.

Object Namespace Explosion

Over time an object can end up having multiple versions. If an object store was designed to handle a billion objects in a bucket, a versioning-like feature may easily demand a 10–100x scalability to be able to handle different versions of the objects over time.

Garbage Collecting Older Versions

In order to limit the namespace explosion as well as provide better storage space management, the older version of the objects need to be deleted and garbage collected regularly. For users it is challenging because they may need to explicitly track which versions of the objects are still useful. A typical solution employed today, is to blindly remove relatively older versions of the objects based on some time window criteria. For example in AWS S3 you can remove older versions of an object by setting up a lifecycle configuration on the bucket.

Referential Integrity and Consistency

An application built on top of an object store, manages several objects and updates different objects based on the application logic and current state. These different objects typically end up referring to each other according to the application logic. Now, over time we could have different versions of the objects referring to specific versions of some other objects.

This creates a complex dependency graph between different versions of these hundreds of objects. Further, auto expiry of some versions of these objects adds a referential integrity problem. This is because some object versions end up auto deleted while another object, that’s still referring to it, is still alive. The figure below explains this.

Referential Integrity and Correlation Explosion

An application that is managing thousands of objects, can experience correlation explosions with multiple versions of one object referring to a specific version of another object. With policy driven automatic object deletion, this only complicates an application state. This is explained below in the figure that shows an application managing only 3 objects with some interdependency between these objects. You can see that this situation for an application only gets worse with thousands of objects and with more interdependencies between objects.

Application Inconsistency

Most cloud native applications periodically checkpoint their old state to recover from various failure cases e.g. application crashes or malware attacks. As we saw above, auto expiry of versioned objects based on the lifecycle configuration at bucket level, can result in objects referring to other object versions that are deleted. This can result in some older checkpoints of the application in invalid state. Effectively this results in applications inability to recover from various failures. It’s not just the older application checkpointed state that can go invalid, even the current state of the application can be invalid if auto expiry of object versions are not set up correctly.

Administrative Overhead

While the Object version feature makes it convenient for users to keep older versions of the objects around, it also makes it difficult to manage these versions. It also makes the job of an application writer difficult with added complex logic to manage application state over various objects, different versions of these objects and a complex relationship between these objects. This only gets worse with independent lifecycle management policies on the object store buckets that hold these objects,

Object Store Snapshots

Ozone’s Snapshot feature solves all of the problems associated with object versioning and yet provides the ability to keep older state of objects around when they are updated. The snapshot feature achieves this by providing the same versioning concept but for a group of objects as opposed to an individual object. This is explained in the picture below.

As you can see in the picture above, a group of objects i.e. {obj1, obj2, obj3} are captured at a consistent application state in a snapshot. You can call this snapshot as a “v1” of this group. The application can keep on updating these objects. Here, individual objects can go through multiple updates. The application or the user can decide to take another snapshot at this point. When we take the snapshot it captures the same group of objects in their current state and we can call it a version “v2” of the application state. After the snapshot is taken, the application can continue to make changes to individual objects in this group again. At any point, the application can decide to restore its state back to version “v1” or “v2” of this group of objects.

Avoiding the Namespace Explosion

The snapshot always captures the state of the group of objects. Within this group some objects can go through 1000 updates yet we need to capture only the final state of the object when the snapshot operation was issued on this whole group. This avoids the namespace explosion that we saw earlier with the object versioning. Here we are capturing the version of the object that is meaningful from an application’s perspective while ignoring to preserve other intermediate versions of the object that could have caused unnecessary namespace explosion.

Maintaining the Referential Integrity

A snapshot captures all the objects in the group at their current application consistent state. All snapshots are inherently ReadOnly. No modification or deletions are allowed in a snapshot. The only way to delete objects in the snapshot is to delete the whole snapshot. Due to its ReadOnly nature, referential integrity is an inherent part of the Snapshot feature.

Snapshot Captures Application Consistent View

An application typically transforms multiple objects depending on the application logic. There is no need to capture intermediate states of various objects while the application is carrying out a specific transaction. What is meaningful from an application’s perspective is the state of the group of objects at the beginning of the transaction and again at the end of the transaction after it is fully committed. Snapshot feature provides that primitive to applications where a captured state of the object store bucket is always application consistent. Restoring the application from any of its older snapshots would be guaranteed to not cause any data corruption or undesired application behavior.

What’s next ?

The next blog in this blog series will go over the snapshot feature for Apache Ozone object store. It will discuss details about how the feature can be used and its storage footprint implications.