Apache Ozone: Using the Snapshot feature

5 min readJun 26, 2024

What is a snapshot?

In Apache Ozone, a snapshot is a read-only copy of the object store at a particular point in time. It allows users to capture the state of the object store instantaneously. After the snapshot creation, the users can continue to make changes to the active object store while the captured image in the snapshot remains immutable forever. Snapshots are useful for data protection, disaster recovery, and data management purposes. They provide a way to revert to a previous state of the object store in case of accidental data loss or corruption. Snapshots can also be used for data analytics, testing, and auditing without affecting the live data in the active object store.

Using the Ozone Snapshot feature:

Snapshot feature is available through ozone fs and ozone sh CLI. This feature can also be programmatically accessed from the Ozone ObjectStore Java client. The feature provides the following functionalities:

Create Snapshot:

Users can create new Snapshots at the bucket level granularity. This operation requires owner privilege of the bucket or the volume. Snapshot create operation is an instantaneous operation regardless of the size of the object store. Apache Ozone object store may contain billions of keys and yet the snapshot create operation is designed to complete instantaneously.

ozone sh snapshot create [-hV] <volume-bucket-path> [<snapshotName>]
ozone fs [generic options] -createSnapshot <volume-bucket-uri> [<snapshotName>]

Reading from Snapshots:

Users can start reading from the snapshot as soon as the snapshot creates operation returns. Snapshots are created under a reserved path “.snapshot” inside a bucket. To access a snapshot, the “.snapshot” hidden namespace is used.

List snapshots:

Users can list all the Snapshots for a given bucket. This operation requires read privilege on the bucket.

ozone sh snapshot list [-hV] <volume-bucket-path>
ozone fs [generic options] -ls /<volume-bucket-uri>/.snapshot

List keys in a snapshot:

Users can list all the keys of the Snapshot. This operation requires read privilege on the bucket.

ozone sh key list [-hV] /<volume-bucket-path>/.snapshot/<snapshot>
ozone fs [generic options] -ls /<volume-bucket-uri>/.snapshot/<snapshot>

Read key from a snapshot:

Users can read a key from the Snapshot. This operation requires read privilege on the bucket.

ozone sh key get [-hV] /<volume-bucket-path>/.snapshot/<snapshot>/<key> <get-key-to-path>
ozone fs [generic options] -cat /<volume-bucket-uri>/.snapshot/<snapshot>/<key>

Delete snapshot:

Users can delete a specific snapshot for a given bucket. This operation requires owner privilege of the bucket. Snapshot delete is an instantaneous operation as well. When a user deletes a snapshot, it is marked deleted and immediately removed from the visible namespace. Actual key reclamation and garbage collection is done in the background by a dedicated service called SnapshotDeletingService.

ozone sh snapshot delete [-hV] <volume-bucket-path> <snapshotName>
ozone fs [generic options] -deleteSnapshot <volume-bucket-uri> <snapshotName>

Snapshot Diff:

Given any two snapshots on the same bucket, users can list all the keys that are different between these two snapshots. Snapshot diff is calculated efficiently. An efficient snapdiff is proportional to the actual difference between the two snapshots rather than the number of objects in source and destination snapshots. For example, the bucket may contain 10 million objects but if there are only 1000 objects that changed between the two snapshots, then the snapdiff operation will take time only proportionate to iterating over 1000 objects.

Efficient snapdiff operation comes with a memory and space overhead to maintain the RocksDB compaction mechanism. Therefore efficient snapdiff can only be computed for a configurable time. This configuration parameter is set by default to 30 days and users can change it through ozone.om.snapshot.compaction.dag.max.time.allowed. If needed to compute a snapshot diff for snapshots older than the configured time, Ozone does a metadata scan based diff which goes over each key in the source and destination snapshots and computes the diff. Such a metadata scan based diff is propionate to iterating over the metadata of all keys in the bucket.

ozone sh snapshot diff [-chV] [-p=<pageSize>] [-t=<continuation-token>] <volume-bucket-path> <fromSnapshot> <toSnapshot>

SnapshotDiff is an asynchronous operation.It is designed that way to account for a very large churn between two snapshots and have the ability to return paginated responses. The first time it is invoked, Ozone Manager starts a background thread to calculate the SnapshotDiff. The API also returns the suggested duration before retrying the snapdiff operation. Once the SnapshotDiff is computed, it returns the diff report in a paginated fashion. This API is safe to be called multiple times for a given snapshot source and destination pair. Ozone manager computes Snapdiff only once and stores it for a configurable time for future invocations of the SnapshotDiff with the same request parameters.

Snapshot diff report consists of entries with the following result types:

+ The file/directory has been created.
- The file/directory has been deleted.
M The file/directory has been modified.
R The file/directory has been renamed.

Note, that the snapshot diff report does not guarantee the same operation sequence. For example, if we rename the directory “/foo” to “/foo2”, and then append new data to the file “/foo2/bar”, the difference report will be:

R ./foo -> ./foo2
M ./foo/bar

I.e. The changes on the files/directories under a renamed directory are reported using the original path before the rename (“/foo/bar” in the above example).

Snapshot Rename:

Users can rename an existing snapshot. This operation requires owner privilege on the bucket.

# ozone shell command:
ozone sh snapshot rename [-hV] <volume-bucket-path> <snapshotOldName> <snapshotNewName>
# ozone fs command:
ozone fs [generic options] -renameSnapshot <volume-bucket-uri> <snapshotOldName> <snapshotNewName>

Snapshot Info:

This API Returns detailed information of an existing snapshot. The details include snapshot’s creation time, status, reference size and exclusive size along with basic info like snapshot name, id, bucket path, etc. Exclusive size is the size of keys exclusively referenced by this snapshot. It is used to know how much space (times replication factor) can be released from the cluster when the snapshot is deleted and garbage collected.

ozone sh snapshot info [-hV] <volume-bucket-path> <snapshotName>

List SnapshotDiff Jobs:

List all active snapshotDiff jobs on a given bucket. This operation is mostly for operational purposes to get information about the running snapshot diff operation. If you need to cancel some SnapshotDiff operations due to memory constraints or stuck operations in the system.

ozone sh snapshot listDiff [-ahV] [-s=<jobStatus>] <volume-bucket-path>

What’s next?

This blog highlighted a quick hands on session with Apache Ozone snapshot feature. Now that we have gone over the Ozone Snapshot feature high level concepts, use cases and have familiarized ourselves with the day to day working of Apache Ozone snapshots, we will conclude this blog series here with this blog. We have another blog series planned to cover the in-depth system level design details of Apache Ozone object store snapshot feature. You can stay tuned for more.