Data Consistency Across Bigtable Clusters

While working with large amounts of data, it is important to have high throughput as users request data frequently and in large quantities. Cloud Bigtable is a sparsely populated table that can scale to billions of rows and thousands of columns, enabling customers to store terabytes or even petabytes of data. Bigtable is ideal for storing very large amounts of single-keyed data with very low latency. Bigtable instances consist of one or more clusters, storing data. Single-cluster Bigtable instances provide strong consistency. By default, instances that have more than one cluster provide eventual consistency, but for some use cases they can be configured to provide read-your-writes consistency or strong consistency, depending on the workload and app profile settings. Before we dive deep into the details of Bigtable consistency, let us briefly visit the CAP theorem and discuss tradeoffs to consider while handling consistency.

CAP Theorem

The CAP theorem states that in a distributed database system, you can only simultaneously guarantee two out of three desirable properties:

• Consistency: All nodes see the same data at the same time.
• Availability: Every request receives a response, even if some nodes have failed.
• Partition Tolerance: The system continues to operate despite network partitions (e.g., temporary loss of connection between nodes).
• This means you often have to choose between high consistency and high availability, especially in systems that need to tolerate network disruptions.
• Cloud Spanner is a great choice for workloads that require strong consistency along with the availability and horizontal scalability.

Tradeoffs to Consider:

• Latency vs. Consistency: Strict consistency often requires waiting for data to be replicated across all nodes, which can increase latency. Relaxing consistency can improve response times.
• Availability vs. Consistency: Prioritizing consistency might mean some operations fail if not all nodes can be reached, potentially reducing availability. Prioritizing availability might mean accepting some inconsistencies.
• Complexity vs. Consistency: Achieving strong consistency can require more complex algorithms and coordination, which can be harder to implement and maintain.

Choosing the Right Balance:

The ideal balance depends on your specific application’s needs:

• High-Consistency Applications: Financial systems, e-commerce platforms, and other applications where data accuracy is critical might prioritize strong consistency.
• High-Availability Applications: Social networks, real-time analytics systems, and applications where continuous operation is paramount might favor eventual consistency.
• Mixed Requirements: Many applications have varying consistency needs for different data or operations. They might use different consistency models for different parts of the system.

Now let us get started with understanding how to always serve consistent data from your multi-cluster Bigtable instance. In this blog, we’ll be discussing how to ensure data consistency in a multi-cluster Bigtable instance. Let’s get started!

Bigtable Basics

In Cloud Bigtable, you create instances, which consist of clusters that your applications can connect to. Each cluster contains nodes, which are the compute units that manage your data and perform maintenance tasks.

What is a Bigtable Instance?

A Bigtable instance is a container for your data. Instances have one or more clusters, located in different zones. Each cluster has at least 1 node.

What is a Bigtable Cluster?

A cluster in Bigtable represents the Bigtable service in a specific location. Each cluster belongs to a single Bigtable instance. When your application sends requests to a Bigtable instance, those requests are handled by one of the clusters in the instance.

What is a Bigtable Node?

Each cluster in an instance has 1 or more nodes, which are compute resources that Bigtable uses to manage your data. Behind the scenes, Bigtable splits all of the data in a table into separate tablets. Tablets are stored on disk, separate from the nodes but in the same zone as the nodes. A tablet is associated with a single node.

What is Strong Consistency?

Traditional relational databases have been designed based on the concept of strong consistency. This means that data viewed immediately after an update will be consistent for all observers of the entity. This characteristic has been a fundamental assumption for many developers who use relational databases. However, to have strong consistency, developers must compromise on the scalability and performance of their application. Simply put, data has to be locked during the period of update or replication process to ensure that no other processes are updating the same data.

What is Eventual Consistency?

Eventual consistency is a theoretical guarantee that, provided no new updates to an entity are made, all reads of the entity will eventually return the last updated value. This means that if there was a write in a database, the data will not be available to read if there is a replication setup in the database and the read might not reflect the most recent changes.

What is Read-Your-Writes Consistency?

RYW (Read-Your-Writes) consistency is a state when the database guarantees that once a client writes a record, all reads to that record from the same client will return the written data. For example, if you have a profile at an e-commerce website and you updated your contact number, read-your-writes would mean you as the client would see the change reflected in the webpage without any delay.

What is a ConsistencyToken?

A multi cluster Bigtable instance will provide eventual consistency. To check data replication across clusters, Bigtable provides a ConsistencyToken which can be utilized to validate data replication across clusters in a Bigtable instance. You create a consistency token either after a batch of writes has been sent or after a certain interval, such as an hour or ten minutes. Then the token can be handed off to be used by another process, such as a client making a read request, which uses the token to check to ensure all the data has been replicated across all clusters before it attempts to read.

Bigtable High Availability and Data Consistency

Sometimes you need the benefits of a NoSQL database such as columnar storage, high throughput, low read write latency and high availability but might have a requirement for providing strongly consistent data to your end users. Following are certain points to keep in mind while utilizing Bigtable for HA and Strong Consistency:

• High Availability: Cloud Bigtable provides high availability via multi regional clusters in an instance, supported by replication across clusters
• Multi-Cluster Routing: Failovers are never easy to handle, Cloud Bigtable Multi-Cluster Routing makes life easier by automatically routing requests to the nearest cluster in an instance. If a cluster becomes unavailable, traffic automatically fails over to the nearest cluster that is available.
• Strong Consistency: With High Availability and Multi-Cluster Routing comes eventual consistency, if your application use case needs strongly consistent data reads, you can achieve that using the ConsistencyToken.

Strongly Consistent Data Across Bigtable Clusters

Below is an example architecture where we review the flow of achieving strongly consistent data across replicated Bigtable clusters:

Architecture Flow

The above architecture is a solution for achieving strongly consistent data across multiple Bigtable clusters in an instance. Detailed steps are outlined below:

1. Based on a schedule, something to the order of 5 minutes, Cloud Scheduler triggers Cloud Functions to create a ConsistencyToken to be used for providing strong consistency data across Bigtable clusters
2. Cloud Functions creates a ConsistencyToken using the generateConsistencyToken API call, which will provide a ConsistencyToken to be utilized for checking consistency. Upon receiving a token via the API call, Cloud Functions will perform the checkConsistencyToken API call to check if data replication has caught up, the call returns boolean true or false.
3. After the ConsistencyToken has been created and verified, Cloud Functions will store it in a “Consistency Ledger” in Firestore in the format:{"tokenCreatedTime":"<timestamp>","token":"<token string"}
4. As a data consumer, Cloud Run will check the latest token entry from the Consistency Ledger stored in Firestore.
5. After fetching the latest token from the Consistency Ledger in Firestore, Cloud Run will read data from Bigtable based on the tokenCreatedTime timestamp received from Firestore. The data read by Cloud Run will be consistent across clusters in the Bigtable instance.

NOTE:

1. With the above architecture, there is a potential risk of clock skew between data mutations and data reads. Hence, it is important to be careful about how clock skews could affect the business use case. You could potentially get around the clock skew by choosing some expected maximum clock skew, skew_max. The value skew_max should be substantially higher than the data mutation deadline to avoid slow-writes problem. Then if you save the current time T before creating a token, a CheckConsistency call can “guarantee” that any value with timestamp < T — skew_max is safe. Now, you can then read data using a filter that excludes any values newer than this value. The idea of skew_max would be set based on your specific business use case.
2. Another thing to keep in mind is data delete operations. By utilizing the above solution, you can achieve strong consistency across a multi-cluster Bigtable instance, but you have to keep in mind that the data consistency is based on tokens and timestamps. In a situation where a cell is written at time T0, ConsistencyToken is created at time T1 and CheckConsistency is successful at time T2, if there is a data deletion at time T3 then if data were to be read based on T1 consistency, the deleted data may or may not be present. The best way to avoid this issue, you should always write data using current timestamp, don’t perform deletes yourself and only use an age-based garbage collection policy. For example, if the data is only deleted after 7 days or so, you won’t run into the case of the latest consistent data being dropped.

Conclusion

In this post, we discussed how to ensure Strong Data Consistency in a Multi-Cluster Bigtable Instance, using a ConsistencyToken in combination with Cloud Scheduler, Cloud Functions and Firestore.

We are able to achieve strongly consistent data reads with the help of ConsistencyToken. The usage of the ConsistencyToken is designed in such a way that the data consumer does not expect near real time data. As a recommendation, it would be realistic to create a ConsistencyToken every 5 minutes to ensure strong consistent data reads as the API call does take time to check for data consistency. If your use case demands high throughput, low latency and strongly consistent data reads, this solution will help you reach those objectives.