How to backup Neo4j Running in Kubernetes

UPDATE June 2020: This article has been updated and adapted to work with the new Neo4j Helm repo, and with Neo4j 4.0, and 4.1

Once you’ve deployed either a single instance of Neo4j or a cluster in kubernetes, maintenance and administration needs are going to require that you take regular backups. This article will step through how you can do that.

Most of the examples and code in this article are taken from Neo4j’s Helm repository. The backup container code and documentation you can find in the tools directory of that repo.

The approach should be suitably generic to work with most any Kubernetes distribution, but in the examples, we’ll be using GKE.

Approach

To back up a running cluster, first — it doesn’t really matter which machine you connect to. It can be the leader, a follower, or a read replica. As long as it’s got the data, it can be a backup target.

The way we’ll go about this is to use a specialized Docker container that has neo4j installed (thus giving us the neo4j-admin tool). This will be the backup container. Its job is to run the backup tool, compress the result, and upload the backup set to cloud storage (in our case, Google’s Cloud Storage).

The Backup Helm Chart

The tools directory provides a mini-helm chart, with a specialized container and a shell script. To install the backup chart (and take a backup) a few parameters are needed:

• neo4jAddr — the address of your Neo4j instance. This can be any internal to Kubernetes DNS name that will get a connection
• bucket — the name of the google storage bucket where you want to save the backup, such as gs://my-backups.
• databases — a comma-separate string of databases to backup to, for example neo4j,system
• Optional parameters include pageCache, heapSize, and others. These let you control how much memory is allocated to the backup, and the name of the resulting file.

In order to upload your backups to google cloud storage, you also need to create a kubernetes secret with your service account key.

Installing the Backup Chart

It works like this:

helm install my-backup-deployment . \
   --set neo4jaddr=my-neo4j.default.svc.cluster.local:6362 \
   --set bucket=gs://my-bucket/ \
   --set database="neo4j\,system" \
   --set secretName=neo4j-service-key

What’s happening is:

• We’re deploying a single pod that never restarts
• That pod is linked to a certain container image which you’ll want to change if you build this container separately.
• The helm set variables get passed as environment variables to the container.
• In the volumes section, we will mount the kubernetes secret named neo4j-service-key to the /auth path. This lets the container authenticate against google storage.

It’s just another pod, so all of your normal logging and monitoring approaches apply. If all goes well, a backupset magically appears in google storage.

What’s Next?

• You’ll probably want to schedule execution of a container like this on a regular basis (for example at least weekly)
• As your data grows in size, the most likely thing you’ll need to tweak is your HEAP_SIZE and PAGE_CACHE to make sure you have enough RAM to take the backup. Keep this in mind too when planning the YAML for the resource, your kubernetes cluster has to be able to request and allocate enough RAM for the inside of the container to allocate your request.

Points of Adaptation

Let’s say you need to backup to S3 — in this case the entire structure can be used, but you need to change the script to pass the credentials in differently, and change the container to install the AWS CLI tools. But the overall approach is the same.

Other things you might want to consider is whether or not to zip/archive your backups. As raw directories, they’re larger, but it saves CPU time and gets your backup to cloud storage sooner if you just recursively copy the directory. In this implementation we’ve opted for compression, but some go without, as the trade isn’t worth it for them.

References

• Operations Manual: Backing up Neo4j

Questions? Head on over to the Neo4j Community Site and ask them in the Cloud topic! If you figure a way to adapt this approach to a new cloud storage type or deployment setup, please let us know.