Cassandra CAP Theorem Curse
Cassandra, as a distributed database, is affected by the CAP theorem eventual consistency consequence. And, sometimes, eventually means a long long time, if you are not taking any action. This article is our first telling on our adventures and challenges with Cassandra and how we faced them. This one is about Cassandra Repair System.
Let me start with a big, loud, imperative and truthful statement:
Cassandra Repair System isn’t an “I’ll study it later” feature. It is a requirement for production environments.
While writing or removing data from it, the cluster’s nodes must communicate among themselves to synchronize replicas and ensure consistency. This process is what Cassandra calls anti-entropy.
Besides anti-entropy mechanics, two other processes build up Cassandra’s repair system: hinted handoff and read repair. As anti-entropy, their goal is to improve Cassandra’s consistency by taking action on specific occasions; the former is when a node is down for some time and has lost some writes, the latter is during some reads.
You might be wondering why I have written about subjects that already are present on Cassandra’s official documentation. Bear with me.
As you already know — just in case you don’t — In Loco’s main technology is to provide beaconless indoor location intelligence. We believe in being able to provide services by anonymously detecting our clients’ interaction with the world around them. The team I work on was built to develop solutions related to this vision.
Behavior is our first attempt to develop privacy-friendly authentication / authorization products through geolocation. To summarize our current vision in a question, it would be: can we authorize / authenticate a person’s action without knowing exactly who is it?
Our first authentication product is currently used by a few digital banks in order to accelerate their onboarding process while reviewing user information. Through our technology, clients’ addresses documentation turns to be obsolete, thus enabling the whole onboarding process to be frictionless for them.
To construct this product, we adopted Cassandra to anonymously store aggregated devices’ geolocation data. Currently, we have a Spark pipeline processing device’s daily visits and feeding our inference engine.
Everyday, In Loco’s integrated devices, generate approximately 50 million visits, creating new or updating an existing device’s frequent locations. This is where consistency comes to play; as we have said before, inconsistencies happen every time we write to Cassandra, although repair systems try to take care of it.
But it is not perfect…
Whilst analysing a reported issue within our Cassandra data, we had a big surprise. Two nodes returned a very different set of answers, one of which was missing new data. Well, we knew about Cassandra eventual consistency property, but no one in the company ever had a problem with it. Until now.
Just to be sure, we queried both nodes shortly after. At this time the data was the same! How could it be? We had just queried the nodes and they had different data!
This event taught us about Cassandra’s read repair… But a bit late.
After this “joyful” ride, we started reading about Cassandra’s repair system. The documentation has a section dedicated to teaching about when to repair nodes. Two of the situations listed are very important to keep in mind:
- To routinely maintain node health;
- To update data on a node containing data that is not read frequently, and therefore does not get read-repair.
We did not have a routine repair and we certainly had data that wasn’t queried frequently enough so read-repair could make its magic. It was about time to start this repair policy, but how?
A few options were suggested:
- Create a cron job to run nodetool repair command line with proper configurations;
- Create a kubernetes Cron Job deployment to execute repair and take care of failure for us;
- Use our Airflow deployment to schedule repairs.
Although they were simple and doable alternatives, they missed a key feature we wanted: a more automatic and less laborious way to repair Cassandra according to a schedule. Given that, we decided to check out existing projects related to this and find out if they could be a more robust alternative.
Two projects came out quite frequently:
Priam is more along the lines of a Cassandra cluster manager. It is able to perform token and backup management, seed discovery and cluster configuration. Cassandra-reaper is “a centralized, stateful, and highly configurable tool for running Apache Cassandra repairs against single or multi-site clusters”.
We had just found our hero. With Cassandra-reaper we could not only get our beloved repair working automatically but also we could check nodes’ health in a friendly UI.
It was very simple to set a kubernetes deployment for it. Even if you are not familiar with Kubernetes, a similar effort to set up Cassandra-reaper can be accomplished using Docker (docker-compose or a dockerfile). Beware of the storage system you choose for Cassandra-reaper. We opted to store within Cassandra as it wraps the whole cycle in a single place, so we just have to watch one database.
The “hardest” part is to set Cassandra’s JMX. This is the way Cassandra-reaper communicates with the cluster and operates over it. For test purposes, avoid setting authentication / authorization, just make sure JMX_LOCAL=no and you should be good to go.
When all is done, you should see this screen when you visit Cassandra-reaper web server. We have already added our clusters. Any information related to how you can use it, can be found in its documentation.
One of Cassandra-reaper’s major features is its simple web UI with quick configuration and very clean layout. It is very easy to use and configure any repair and check the cluster’s health. It also comes with an authentication / authorization mechanism, which is as simple to set as the deployment itself.
The other one is the split of token ranges into smaller segments. This mechanism enables a smoother repair; node’s CPU usage can increase during repair, which impacts query latency. Be aware that its impact is strongly related to the repair intensity configuration.
Cassandra-reaper has a whole lot of other features and concepts which can be found in its documentation. It is now integrated into our system to watch Cassandra status and keep nodes healthy. Hopefully, we won’t have more surprises with inconsistencies.
You can checkout our deployment file here.
Are you interested?
If you are interested in building context-aware products through location, check out our career page. Also, we’d love to hear from you. Leave a comment.
