Simple Things You Can Learn From Cassandra Nodetool (Monitor/Manage) For DC/OS

As we know, DC/OS gave us minimal packages of services for running on apache mesos like cassandra. Cassandra services only give us the functionality for storing data , not other things like native tools as `nodetool`, `cqlsh`, `cassandra-stress` and more. Today, we are looking into the way for using `nodetool`, for monitoring DC/OS cassandra services nodes or cluster.

Brief:

Nodetool is a native command line interface for managing and monitoring cassandra cluster. While our cassandra services are running on DC/OS, we can manage and monitor cluster using nodetool. We need to execute nodetool on DC/OS using cassandra docker image on any of the DC/OS cassandra nodes by performing below steps:

1. First, need to ssh any of DC/OS cassandra node: ssh @
2. Second, execute the docker command for download cassandra docker image and pass nodetool command for execution as below:

$ sudo docker run -t --net=host cassandra:3.0 nodetool tablestats -H .

Note: Sometime nodetool commands may not work if Cassandra version and Nodetool version is incompatible e.g: Cassandra Version 3.0 and Nodetool version 3.10, nodetool will not be able to run all commands.

Below, we are looking into some of the important nodetool commands for monitoring perspective of cassandra cluster. But still, nodetool have lots of commands, which we are not discussing now. For reference please click on this link.

1. CASSANDRA CLUSTER STATUS

For looking into cassandra cluster health or status using node tool, we need to execute below command:

sudo docker run -t --net=host cassandra:3.0 nodetool -p 7199 status

OUTPUT:







































— AddressLoadTokensOwns (effective)Host IDRackUN23.26 GB25621.7%rac1UN20.48 GB25619.7%rac1UN21.62 GB25619.7%rac1

• U indicate whether the node is UP.
• N indicates whether the node is NORMAL.
• DC/OS cassandra node IP address.
• Load: the amount of filesystem data under the cassandra data directory and updated every 90 seconds.
• Tokens: number of tokens set for the node.

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsStatus.html

2. MONITOR CASSANDRA CLUSTER SINGLE NODE INFO

If we need to require cluster node stats including:

• Load
• Uptime
• Heap Memory
• Exceptions Count
• Key Cache Hit Rate

and more, for complete we need to execute below command. But this command gave us the stats of current node in which we are making ssh.

sudo docker run -t --net=host cassandra:3.0 nodetool info

OUTPUT



































































IDGossip activetrueThrift activefalseNative Transport activetrueLoad22.57 GBGeneration No1506714058Uptime (seconds)1089322Heap Memory (MB)1594.28 / 4086.00Off-Heap Memory (MB)3.42Data Centerdc1Rackrac1Exceptions0Key Cacheentries 158173, size 56.76 MB, capacity 100 MB, 9092924 hits, 9352093 requests, 0.972 recent hit rate, 14400 save period in secondsRow Cacheentries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in secondsCounter Cacheentries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in secondsToken(invoke with -T/ — tokens to see all 256 tokens)
For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsInfo.html

3. MONITOR CLUSTER STATISTICS OF KEYSPACES AND TABLES

If required to monitor statistics of all keyspaces and tables like:

• Read Latency
• Write Latency
• Total Disk Used

and more. For complete we need to execute below command.

sudo docker run -t --net=host cassandra:3.0 nodetool tablestats // Show stats for all keyspaces and tables
sudo docker run -t --net=host cassandra:3.0 nodetool tablestats . … . // show stats for specific keyspace and tables
OUTPUT
Keyspace: 
Read Count: 75738
Read Latency: 1.0500195410494073 ms.
Write Count: 34312
Write Latency: 0.12817600256470038 ms.
Pending Flushes: 0
Table: 
SSTable count: 6
Space used (live): 3390578917
Space used (total): 3390578917
Space used by snapshots (total): 0
Off heap memory used (total): 423687
SSTable Compression Ratio: 1.003568418469647
Number of keys (estimate): 3967
Memtable cell count: 94
Memtable data size: 11470945
Memtable off heap memory used: 0
Memtable switch count: 75
Local read count: 75738
Local read latency: 1.032 ms
Local write count: 34312
Local write latency: 6.078 ms
Pending flushes: 0
and more...
For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablestats.html

4. MONITOR CLUSTER DISK SPACE RECLAIM

If we require checking, how many garbage value we have and need to monitor how much disk is available for re-claim, we need to execute below command.
sudo docker run -t --net=host cassandra:3.0 nodetool gcstats
OUTPUT: 





















Interval (ms)Max GC Elapsed (ms)Total GC Elapsed (ms)Stdev GC Elapsed (ms)GC Reclaimed (MB)CollectionsDirect Memory Bytes1091695368530135316012460376473472056632-1
For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsGcstats.html

5. MONITOR CLUSTER THREADS AND PENDING PROCESS

Cassandra is based on a Staged Event Driven Architecture (SEDA). Different tasks are separated into stages that are connected by a messaging service. Stages have a queue and thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. The queues can back up if executing at the next stage is too busy and cause performance bottlenecks.
So, if we require monitoring cluster threads, active task, pending task, completed blocked and more. It's updated when SSTables change through compaction or flushing. we need to execute below command:
sudo docker run -t --net=host cassandra:3.0 nodetool tpstats

6. MONITOR CLUSTER THREADS AND PENDING PROCESS

If required statistics about the table like:

• Read/Write Latency
• Partition Size
• Column Count
• Number of SSTable

and more... This command is kind of tablestats command, but its output gave us histogram matrices for stats. Below is the command:
EXAMPLE:
sudo docker run -t --net=host cassandra:3.0 nodetool tablehistograms
OUTPUT



































































PercentileSSTablesWrite Latency (micros)Read Latency (micros)Partition SizeCell Count50%0.00182.790.001331475%1.00182.790.001331495%1.00219.340.001597498%1.00379.020.001597499%1.00379.020.0015974Min0.0051.010.004474Max1.001629.720.0019164
For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablehisto.html
As we mention above there are lots of commands are available for monitoring and manage cluster by using nodetool. So, as per our requirements, we will use nodetool for manage or monitor.
References:
https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsNodetool.html