Querying Neo4j Clusters
With Neo4j as with most databases, if you want to just query the database it’s simple. You use a driver, create a connection, send a query and get back some results. That’s all there is to it!
Behind the scenes though, if you’re working with any clustered database, there’s a whole lot more going on than that. To begin with, the database isn’t in a single place but is composed of multiple servers. In this article, we’ll explore how Neo4j Clusters work, and how Neo4j drivers get your query executed.
Before describing the drivers, we need a brief overview of how Neo4j clusters work, and what the cluster roles are. This, in turn, will help you understand what a driver is doing.
Questions or comment on any of this? Come ask on the Neo4j Community Site!
Neo4j Causal Clustering
A cluster is composed of three or more Neo4j instances that communicate with one another to provide fault-tolerance and high-availability using a consensus protocol (RAFT). In Neo4j clustering, each database has a perfect, complete copy of the entire database (the graph is not sharded). Each machine in the cluster has a “role”. It can either be the leader or a follower.
Cluster Roles
The leader is responsible for coordinating the cluster and accepting all writes. Followers help scale the read workload ability of the cluster and provide for high-availability of data. Should one of the cluster machines fail, and you still have a majority, you can still process reads and writes. If your cluster loses the majority it can only serve (stale) reads and has no leader anymore.
Optionally, you can have any number caches in the form of read replicas. They are read-only copies of your database for scaling out read-query load. They are not officially part of the cluster, but rather are “tag along” copies that get replicated transactions from the main cluster.
Topology Changes
In the lifecycle of a cluster, cluster roles are temporary, not things you configure. Suppose you have machines A, B, and C. If A fails, then the remaining nodes (B and C) will elect a new leader amongst themselves. When A restarts, later on, it will rejoin the cluster, but probably as a follower. So roles can change through the lifecycle of the cluster. There are various other reasons where everything is working fine, where the cluster might elect a new leader — and so by themselves, role changes are not a cause for concern.
Neo4j uses the RAFT consensus algorithm to coordinate the cluster. Quite a lot is published on that topic if you’d like to go deeper.
Neo4j Drivers
The Driver API consists of 4 key parts illustrated below. Whether talking to a Neo4j cluster or single instance, all of these concepts apply.
In the cluster world though, because we are talking to a group of machines, how the transactions get executed and where they go is the subject of how “Routing Drivers” work.
Routing Drivers
When you use one of the supported Neo4j drivers (Java, Javascript, Python, .Net and Go), there is an option to use the bolt+routing protocol. You’ll know you’re using it because it’s in the URI of the connection string. For example, you can connect to bolt+routing://neo4j.myhost.com.
Tip
For clusters, set up a single DNS record with multiple A entries, each pointing to the cluster members.
This way, all clients can connect to the same DNS address but may have different machines as points of entry depending on cluster state and what is up. You don’t have to do it this way; you can connect directly to any of the server member’s IP addresses, but this way is more flexible should query topology change.
If you’re connecting to any single host or address, the driver handles the routing smarts for you. The driver will first check if the database has routing capabilities, and if so will fetch that holds a full or partial snapshot of the read and write services available. After that moment the initial host will not be used unless the driver loses contact with the cluster and has to re-initialize routing.
Routing Tables
The routing table holds a list of servers that provide ROUTE, READ and WRITE capabilities. This routing information is considered volatile and is refreshed regularly.
Drivers refresh this regularly because the cluster topology could change according to runtime events (like a machine failing). Generally, the WRITE node is going to be the leader, and the READ nodes are going to be the followers in the cluster or read-replicas.
Important Tip
Read replicas do not participate in cluster topology management, and as such they do not provide routing tables. Only core nodes provide routing tables. This may change in subsequent releases of Neo4j.
If you’d like to try this out for yourself, just execute this statement on any clustered Neo4j instance:
CALL dbms.cluster.routing.getRoutingTable({})
YIELD ttl, servers
UNWIND servers as server
RETURN ttl, server.role, server.addresses;and you can see your routing table
This particular cypher query is only intended for internal use and may change in future versions of Neo4j so don’t write your code to it, but it gives you the idea of what the driver is actually doing.
Advertised Addresses
Where did those addresses in the routing table example come from? These are set by the user in the neo4j.conf file, as dbms.connector.bolt.advertised_address . In this way, Neo4j knows what address to publish to the external world where it can be contacted. (Configuration Reference)
Important Tip
The advertised address setting is crucial for external clients to know how to contact your cluster. Set it explicitly in neo4j.conf
Make sure it is an address that can be resolved by external clients (and not an internal private address, such as 192.168.0.5). A very common error with Neo4j connectivity is to fail to set this, or set it to an internal private address, and have external clients on the Internet fail to connect because they cannot figure out how to reach your database!
Connection Management
Once a routing table in place, the driver can manage a pool of connections to all of the different machines in the cluster. What the user sees, is usually just the creation of sessions, and running queries from those sessions. What’s actually inside of the driver looks more like a pool of connections to a set of machines (A, B, and so on). Sessions simply borrow and ride on top of connections as needed to execute queries.
These connections are handled separately so that if (for example) Server A goes away and all of those connections end up dying, you at the session level don’t necessarily need to know about that. A session can still borrow a different connection and your application can keep chugging. An ongoing statement execution would fail and automatically retried by the driver.
Users execute queries by using sessions. Sessions are cheap objects to create (unlike connections, which are expensive to set up). Sessions also provide a logical construct for chaining work together in a way that is causally consistent, meaning that you can do a series of transactions where subsequent transactions are always reading the writes made by earlier transactions in the same session.
Query Routing
Now let’s say your driver sends a query to the database. I’ll use JavaScript as an example, but the same concepts apply in any language.
The first thing we do is pull a session from the driver. We then use that session to execute the query.
const session = driver.session();
session.readTransaction(tx =>
tx.run('MATCH (n) RETURN count(n) as value'))
.then(results => {
console.log(results.records[0].get('value'));
})
.finally(() => session.close());How does the driver know where to send this query? In this code example, we have used an explicit transaction or transaction function, meaning that we told the driver we’re doing a readTransaction. We were given a tx Transaction object, and with that we ran our query. Because it has the routing table, and it knows you’re doing a read, it will probably end up sending this query to one of the nodes with the READ role. If we did a write transaction, it would be sent to any node in the routing table that has a role of WRITE. (Which is generally the cluster leader)
If the routing driver has more than one choice of where to send the query, it uses a “least connected” strategy, which helps avoid accidentally routing to a server which is presently too busy to answer in a timely way.
Auto-Commit Transactions
What if you don’t use explicit transactions? You can skip the transaction function and instead you could write:
const session = driver.session();
session.run(
tx => tx.run('CALL library.myProcedure()'))
.then(results => {
console.log(results.records[0].get('value'));
})
.finally(() => session.close());In this case, the driver cannot tell if you are doing a read or a write and will send your transaction to the leader. This way, the transaction should succeed, even if you haven’t told it whether it needs to do a read or a write.
Important Tip
Neo4j drivers do not parse your Cypher to determine whether you’re reading or writing. Always use explicit transactions to tell it whether it’s a read or a write.
This, in turn, helps it route the query effectively, and get the best utilization out of your cluster. If you never used explicit transactions, you might be sending all of your query load to the leader, beating it up while leaving the other machines in your cluster idle.
Conclusion
In a clustered deployment, Neo4j utilizes smart bolt+routing clients to dynamically discover and monitor cluster topology, and route your queries to machines in the cluster using a “least connected” strategy. All of this is done transparently for you, so at the user level all you’ll see is creating sessions, running queries, and consuming the results.
Getting maximum benefit out of this setup requires some understanding of how this operates, and the most important parts to keep in mind are the following:
- Proper configuration of your advertised address
- Use of explicit transactions in driver code
- Where possible, DNS setup to create a single DNS record that all clients can use to talk to any node in the cluster
For more about Neo4j routing drivers, consult the Neo4j Drivers Manual.
