Open in app

Sign in

Write

Sign in

Cassandra Read / Write Consistency & Replication Factor

Environment:

Guru Datta
6 min readAug 10, 2020

Running on docker, with Total of Cassandra Nodes (n) = 3 in one Data Center.

(This Simulation was on Cassandra version 3.x — 3.11.7)

>> docker ps
CONTAINER ID
<id3>
<id2>
<id1>// Logging into one of the container <id1>
> docker exec -it <id1> bash

Cassandra partitions data over storage nodes using a special form of hashing called consistent hashing. Cassandra maps every node to one or more tokens (vnodes) on a continuous hash ring. (For simplicity purpose we will use only one vnode for each node and using Replication strategy of Simple Strategy for data replication within one datacenter)

Cassandra Cluster with 3 nodes

Cassandra supports a per-operation tradeoff between consistency and availability through Consistency Levels (refer Tunable consistency).

Write operations are always sent to all replicas, regardless of consistency level. The consistency level simply controls how many responses the coordinator waits for before responding to the client.

For read operations, the coordinator generally only issues read commands to enough replicas to satisfy the consistency level. The one exception to this is when speculative retry may issue a redundant read request to an extra replica if the original replicas have not responded within a specified time window.

Let’s Observe how cassandra Reads behave with Different Tunable Consistencies (Testing with Consistency level = 1 and QUORUM ((replication_factor/2) + 1), as remaining options mostly fall under this number for current setup.)

Note: Also Writes Consistency behavior is similarly to Reads, hence providing observations on Reads.

> cqlsh
cqlsh > CREATE KEYSPACE bookstore WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : '1'};// get the keyspace details in the cluster
cqlsh > DESC keyspaces;// selecting bookstore keyspace
cqlsh > USE bookstore;// creating Table musics_by_genre, we will be using books table in bookstore throughout this process.
cqlsh > CREATE TABLE books (
name VARCHAR,
category VARCHAR,
year INT,
title VARCHAR,
PRIMARY KEY ((name), category, year, title)
) WITH CLUSTERING ORDER BY (category ASC, year DESC, title ASC);// get details of table
cqlsh > DESC TABLE books;// insert some data to this table
cqlsh > INSERT INTO books (name, category, year, title) VALUES ('in to the woods', 'Fiction', 2020, 'In To The Woods');// Based on the partiton hash, the above record got stored in node container <id2>

With Replication Factor 1 (Not to be Used in Production):

Based on Partition Key Hash, node/s (<id2>) is chosen in clockwise direction.
> cqlsh
cqlsh > CREATE KEYSPACE bookstore WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : '1'};// get the keyspace details in the cluster
cqlsh > DESC keyspaces;// selecting bookstore keyspace
cqlsh > USE bookstore;// creating Table books, we will be using books table in bookstore throughout this process.
cqlsh > CREATE TABLE books (
name VARCHAR,
category VARCHAR,
year INT,
title VARCHAR,
PRIMARY KEY ((name), category, year, title)
) WITH CLUSTERING ORDER BY (category ASC, year DESC, title ASC);// get details of table
cqlsh > DESC TABLE books;// insert some data to this table
cqlsh > INSERT INTO books (name, category, year, title) VALUES ('in to the woods', 'Fiction', 2020, 'In To The Woods');// Based on the partiton hash, the above record got stored in node <id2>cqlsh> consistency;
Current consistency level is ONE.> docker stop <id3> // <id1> and <id2> are up and data is present in node <id2> and hence able retrieve record.
cqlsh > SELECT * FROM bookstore.books WHERE name='in to the woods';
name            | category | year | title-----------------+----------+------+-----------------in to the woods |  Fiction | 2020 | In To The Woods// Stopping the container which has record> docker stop <id2>cqlsh> SELECT * FROM bookstore.books WHERE name='in to the woods';NoHostAvailable:

With Replication Factor 2 :

After re-creating keyspace with replication factor = 2 & table with data (or) altering replication factor of keyspace to 2.

Based on Partition Key Hash, node/s (<id2>, <id3>) is chosen in clockwise direction.
cqlsh > CONSISTENCY QUORUM;
Consistency level set to QUORUM.// Since replication factor=2 , The inserted record with name='in to the woods' got replicated to nodes <id2> and <id3>.// stopping nodes <id2> and <id3>, to test READ/ WRITE CONSISTENCY.> docker ps
CONTAINER ID
<id1>cqlsh > SELECT * FROM bookstore.books WHERE name='in to the woods';
NoHostAvailable://Bringing up container node <id3>// Consistency level is still quorum which is 2 in this case.
> docker ps
CONTAINER ID
<id3>
<id1>// Not able to retrieve record, since CONSISTENCY of QUORUM was not achieved for queried record.
cqlsh> SELECT * FROM bookstore.books WHERE name='in to the woods';
NoHostAvailable://Bringing up container node <id3>, <id2> (Data is present in <id3> and <id2> node) and stopping <id1>
> docker ps
CONTAINER ID
<id3>
<id2>cqlsh> SELECT * FROM bookstore.books WHERE name='in to the woods';
SELECT * FROM bookstore.books WHERE name='in to the woods';name            | category | year | title-----------------+----------+------+-----------------in to the woods |  Fiction | 2020 | In To The Woods//Bringing up container node <id1>, <id3> and stopping <id2> for READ CONSISTENCY = QUORUM, we were not able to retrieve record before. Now let's change CONSISTENCY to 1//After changing consistency Level to One, able to retrieve record, it doesn't matter which node was down, since we have REPLICATION_FACTOR = 2 and two nodes up.cqlsh> consistency one;
Consistency level set to ONE.cqlsh> SELECT * FROM bookstore.books WHERE name='in to the woods';
name            | category | year | title-----------------+----------+------+-----------------in to the woods |  Fiction | 2020 | In To The Woods

With Replication Factor = 3

After re-creating keyspace with replication factor = 3 & table with data (or) altering replication factor of keyspace to 3.

Based on Partition Key Hash, node/s (<id2>, <id3>, <id1>) is chosen in clockwise direction.
cqlsh> CONSISTENCY QUORUM;
Consistency level set to QUORUM.// with 2 nodes up, consistency level =quorum (2)
> docker ps
CONTAINER ID
<id3>
<id1>// will be able to retrieve data
cqlsh> SELECT * FROM bookstore.books WHERE name='in to the woods';
name            | category | year | title-----------------+----------+------+-----------------in to the woods |  Fiction | 2020 | In To The Woods// After stopping <id3> and keeping only one node up, will not be able to retrieve data.
> docker ps
CONTAINER ID
<id1>cqlsh > SELECT * FROM bookstore.books WHERE name='in to the woods';
NoHostAvailable://With one container, if consistency changed to 1, will be able to retrieve data.
cqlsh> consistency one;
Consistency level set to ONE.cqlsh> SELECT * FROM bookstore.books WHERE name='in to the woods';name            | category | year | title-----------------+----------+------+-----------------in to the woods |  Fiction | 2020 | In To The Woods

Note:

In general Consistency level can be defined per query basis, for example:

cqlsh > SELECT * FROM bookstore.books USING CONSISTENCY QUORUM WHERE name='in to the woods';
//similarly for Insert/ Update/ Delete.
//Refer cql

Observations:

Cassandra cluster is more Fault tolerant, if chosen Replication Factor is higher and tradeoff is it needs more storage allocation.

Similarly, Strong READ/ WRITE Consistency can be achieved, if higher READ/ WRITE CONSISTENCY is chosen, but tradeoff is increase in latency.

Replication Strategy, Replication Factor and READ/ WRITE consistency levels has to be chosen based on the needs of application, nodes available, data centers available.

Edge cases: I have not covered all edge cases, please try mixed scenarios of WRITE CONSISTENCY = x and READ CONSISTENCY = y of same record (where x != y), to discover the behavior.

Below are the observations of above simulation.

Cassandra Cluster docker-compose.yml file

version: "3.3"

networks:
  app-tier:
    driver: bridge

services:

  cassandra-seed:
    image: cassandra:latest
    hostname: cassandra0
    ports:
      - "9042:9042"
    volumes:
      - "cassandra_data_seed:/var/lib/cassandra"
    environment:
      - "CASSANDRA_SEEDS=cassandra-seed"
      - "CASSANDRA_CLUSTER_NAME=Test Cluster"
      #      needed for setting up custom cluster name
      - "CASSANDRA_DC=se1"
      - "CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch"
  #    restart: always
    networks:
      - app-tier

  cassandra1:
    image: cassandra:latest
    hostname: cassandra1
    ports:
      - "9041:9042"
    volumes:
      - "cassandra_data_1:/var/lib/cassandra"
    environment:
      - "CASSANDRA_SEEDS=cassandra-seed"
      - "CASSANDRA_CLUSTER_NAME=Test Cluster"
      #      needed for setting up custom cluster name
      - "CASSANDRA_DC=se1"
      - "CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch"
    depends_on:
      - cassandra-seed
  #    restart: always
    networks:
      - app-tier

  cassandra2:
    image: cassandra:latest
    hostname: cassandra2
    ports:
      - "9040:9042"
    volumes:
      - "cassandra_data_2:/var/lib/cassandra"
    environment:
      - "CASSANDRA_SEEDS=cassandra-seed"
      - "CASSANDRA_CLUSTER_NAME=Test Cluster"
      #      needed for setting up custom cluster name
      - "CASSANDRA_DC=se1"
      - "CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch"
    depends_on:
      - cassandra-seed
#    restart: always
    networks:
      - app-tier

volumes:
  cassandra_data_seed:
  cassandra_data_1:
  cassandra_data_2:
  Users:

Sources:

https://cassandra.apache.org/doc/latest/architecture/dynamo.html#

https://cassandra.apache.org/doc/latest/cql/index.html

Cassandra
Consistency
Replication Factor
Cassandra Reads
Cassandra Writes

--

--

Guru Datta

Written by Guru Datta

1 Follower

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams