Getting Certificates for Neo4j with LetsEncrypt

Published in

Neo4j Developer Blog

8 min readSep 17, 2018

Using Neo4j’s cloud VMs, a common question is how to set up valid SSL to protect data in transit. This article will cover how to do it with LetsEncrypt, a popular free certificate authority.

The instructions below will work with most any public cloud-hosted instance of Neo4j, so let’s get started. Updated June 2020: includes instructions for Neo4j 4+. Updated February 2022: includes notes and warning on Neo4j debug.logtraces you might see.

Come along and ride on a fantastic voyage

Why are we doing this?

You need valid SSL certificates in order for the browser and various client applications to trust that your site is what it says that it is. If you’ve created a Neo4j instance in a public cloud and you’ve seen browser warnings about “this site is untrusted” or “add a special exception” — valid certificates solve this.

The surefire way to know that you don’t have a valid certificate

Additionally, some common connection errors for Neo4j and cypher-shell derive from the lack of a valid certificate. So here we go.

Prerequisites

Your machine must have a valid DNS address in order to have a valid SSL certificate. Certificates typically aren’t granted for bare IP addresses because it’s a lot harder to prove that you own/control a bare IP address.
Important: If you’re doing this for a causal cluster, you must do it for each machine in the cluster, not just one.
This tutorial doesn’t cover how to get a valid DNS address, as the process is different for each cloud, and each network setup. Before using these instructions, ensure that you have a DNS name for your machine, and verify that it resolves correctly to the IP of your machine before proceeding.

When I do this, I register domains with Google Domains, (let’s say mydomain.com) and I use Google Cloud DNS to map my neo4j machines (say, machine1.cluster.mydomain.com) to the IP address of the VM on Google Cloud. But there are many different possible ways to accomplish this part that you can find online.

Install LetsEncrypt

On each machine, we’re going to install certbot, a tool which generates certificates. To do that, we’ll pull some commands from the certbot install guidelines.

sudo apt-get update
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install -y certbot

What is Let’s Encrypt?

Let’s Encrypt is a certificate authority that provides free certificates to help encourage people to use encryption across the Web. Encryption makes the web safer and more secure for everyone. If you need more background on certificate authorities, have a look here.

Generate a certificate

Certbot will generate the certificate for us. In order for it to work, and to verify that you have the domain you claim, we’ll need to ensure via the firewall that port 80 is open (just temporarily) so the tool can do its job. Port 80 is not normally open for Neo4j cloud instances, so make sure to set that up as a separate step.

Here is how we generate the certificate - sudo certbot certonly. As shown in the following, we’re specifying option 1 (we want to create a temporary webserver on port 80 to verify ourselves) and providing the domain name we’re generating a cert for (node2.cluster.graph.center)

$ sudo certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.logHow would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Spin up a temporary webserver (standalone)
2: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'to cancel): node2.cluster.graph.center
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for node2.cluster.graph.centerWaiting for verification...
Cleaning up challengesIMPORTANT NOTES: 
- Congratulations! Your certificate and chain have been saved at:   /etc/letsencrypt/live/node2.cluster.graph.center/fullchain.pem   Your key file has been saved at:   /etc/letsencrypt/live/node2.cluster.graph.center/privkey.pem   
Your cert will expire on 2018-12-16. To obtain a new or tweaked   version of this certificate in the future, simply run certbot   again. To non-interactively renew *all* of your certificates, run   "certbot renew" 
- If you like Certbot, please consider supporting our work by:   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate   Donating to EFF:                    https://eff.org/donate-le$

The success message at the bottom says we now have certificates located in the /etc/letsencrypt/live/* subdirectories.

Configuring Neo4j

LetsEncrypt maintains these certificates in a directory called “live”. These are actually symlinks to files in another directory. This layer of indirection allows the certbot program to update the certificates periodically as needed, so that you don’t have to change cert locations.

So when we place these into the neo4j structure, we’re going to be careful to create symlinks and not copy the files. By doing so, we can take advantage of the refresh abilities of Let’s Encrypt we might want later on.

The first thing we have to do is adjust the very tight permissions on the certificates directory, by changing group ownership to neo4jand to make them readable by Neo4j.

# Change group of all letsencrypt files to neo4j
sudo chgrp -R neo4j /etc/letsencrypt/* # Make sure all directories and files are group readable.
sudo chmod -R g+rx /etc/letsencrypt/*

Next, we set up symlinks and the directory structure neo4j expects.

WARNING — Follow the right instructions for your Neo4j version! Everything up until this point will work with both Neo4j 3.5 and Neo4j 4+ At this point, make sure to follow *only one set* of the following two blocks for your Neo4j version, because in version 4+, directory structure and config changed!

Neo4j 4+ Instructions

The 4.0 series has some key differences in how to configure SSL from 3.5. There is a directory of certificates per “connector” (bolt, HTTPS, cluster), and the config options have changed a bit.

cd /var/lib/neo4j/certificates# Move old default stuff into a backup directory.
sudo mkdir bak
for certsource in bolt cluster https ; do
   sudo mv $certsource bak/
donesudo mkdir bolt
sudo mkdir cluster
sudo mkdir httpsexport MY_DOMAIN=graph.somehost.comfor certsource in bolt cluster https ; do
   sudo ln -s /etc/letsencrypt/live/$MY_DOMAIN/fullchain.pem $certsource/neo4j.cert
   sudo ln -s /etc/letsencrypt/live/$MY_DOMAIN/privkey.pem $certsource/neo4j.key
   sudo mkdir $certsource/trusted
   sudo ln -s /etc/letsencrypt/live/$MY_DOMAIN/fullchain.pem $certsource/trusted/neo4j.cert ;
done# Finally make sure everything is readable to the database
sudo chgrp -R neo4j *
sudo chmod -R g+rx *

Finally, we make adjustments to our neo4j configuration file. (Keeping in mind that if you’re using cloud VM images, this is /etc/neo4j/neo4j.template, and in other environments it’s /etc/neo4j/neo4j.conf)

dbms.default_listen_address=0.0.0.0
dbms.default_advertised_address=your.hostname.com# BOLT Connector
dbms.connector.bolt.tls_level=REQUIRED
dbms.ssl.policy.bolt.enabled=true
dbms.ssl.policy.bolt.private_key=/var/lib/neo4j/certificates/bolt/neo4j.key
dbms.ssl.policy.bolt.public_certificate=/var/lib/neo4j/certificates/bolt/neo4j.cert
dbms.ssl.policy.bolt.client_auth=NONE# HTTPS connector
dbms.connector.https.enabled=true
dbms.ssl.policy.https.enabled=true
dbms.ssl.policy.https.client_auth=NONE
dbms.ssl.policy.https.private_key=/var/lib/neo4j/certificates/https/neo4j.key
dbms.ssl.policy.https.public_certificate=/var/lib/neo4j/certificates/https/neo4j.cert# Directories
dbms.ssl.policy.bolt.base_directory=/var/lib/neo4j/certificates/bolt
dbms.ssl.policy.https.base_directory=/var/lib/neo4j/certificates/https

Neo4j 3.5 Instructions

cd /var/lib/neo4j/certificates
sudo mkdir revoked trusted bak# Move old generated certificates into a backup directory
sudo mv neo4j.* bakexport MY_DOMAIN=graph.somehost.com# Configure cert neo4j will use
sudo ln -s /etc/letsencrypt/live/$MY_DOMAIN/fullchain.pem neo4j.cert# Configure private key neo4j will use
sudo ln -s /etc/letsencrypt/live/$MY_DOMAIN/privkey.pem neo4j.key# Indicate that this cert is trusted for neo4j
sudo ln -s /etc/letsencrypt/live/$MY_DOMAIN/fullchain.pem trusted/neo4j.cert

dbms.connectors.default_listen_address=0.0.0.0
dbms.connectors.default_advertised_address=your.hostname.com
bolt.ssl_policy=default
dbms.ssl.policy.default.base_directory=/var/lib/neo4j/certificates
dbms.ssl.policy.default.allow_key_generation=false
dbms.ssl.policy.default.private_key=/var/lib/neo4j/certificates/neo4j.key
dbms.ssl.policy.default.public_certificate=/var/lib/neo4j/certificates/neo4j.cert
dbms.ssl.policy.default.revoked_dir=/var/lib/neo4j/certificates/revoked
dbms.ssl.policy.default.client_auth=NONE

What This Config Does

Let’s work through explanations of what these do and why you need them. You can find a complete reference in the Neo4j SSL Framework documentation. (Make sure to select the right documentation for your version!)

The default advertised address for the node is set to the public DNS you configured. This is important so that clients can connect to that DNS name, and also so that when the Neo4j Browser application does redirects, it does so to a trusted DNS entry with a signed cert. The Neo4j Browser application tends to issue redirects to the default_advertised_address. As a result browsers will get confused if this is misconfigured, because it will look like an SSL site trying to redirect to some other, untrusted site (for example, a bare IP address).
We configure the bolt transport and/or bolt “connector” (terminology here differs between Neo4j 3.5 and 4.0) to use the “default” policy for SSL and certain settings.
The directory where everything is stored
The private key
The public certificate
Finally, client_auth=NONE means that the client does not have to similarly auth with a valid cert (i.e. it is OK if the client doesn’t have a cert)

Note: client_auth=NONE is important if you want to connect using clients like Chrome and Neo4j browser, because Chrome on most people’s machines does not have a valid cert, and hence cannot pass the client_auth check that Neo4j is capable of enforcing.

Restart neo4j

After all of this configuration, we need to restart neo4j

sudo systemctl restart neo4j

What if I have a Cluster?

If you’re using a Neo4j Enterprise causal cluster the instructions are exactly the same, except you have to do this once per machine in the cluster, and you need to verify that each machine has a *different address or hostname* that it advertises, so that routing client queries will work correctly.

The extra nit for clusters, is that you’ll notice in the Neo4j 4.0 configuration section, you will also need to set up the connector for “cluster” (rather than just bolt and https) to ensure that intra-cluster communication is secured using the same certificate. That’s all.

I’m Still Getting an Error

A very important thing about SSL certificates is that they prove a server’s identity by a domain name. In the above guide, suppose you request a certificate for mygraph.mycompany.com. That certificate proves that machine is secure when addressed by that name.

But sometimes, users might SSH into that box and then do:

cypher-shell -a neo4j+s://localhost:7687

Take note of the hostname! We’re asking for a secure connection to localhost. This is going to fail validation, because when you connect, you’ll get proof that the server is mygraph.mycompany.com, and cypher-shell will fail the connect because it isn’t localhost.

Further, you might see something like this in Neo4j’s debug.log

022-02-15 13:22:27.498+0000 ERROR [o.n.b.t.TransportSelectionHandler] Fatal error occurred when initialising pipeline: [id: 0x00ceb907, L:/127.0.0.1:7687 ! R:/127.0.0.1:37498]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
(...)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

This is the error on the server side showing that certificate validation failed. It’s easy to fix; address the server by its verified name, not by a bare IP address or different hostname, this permits the validation to succeed.

Renewing LetsEncrypt certificates on a schedule

By running certbot with a renew flag, you can update certificates to push out the expiry date. Doing this for example, monthly via a cron job keeps the certificate fresh.

You can check that this will work by running:

sudo certbot renew --dry-run

And check that the output has no errors.

The nice part about our symlink structure above is that our links are always pointing to the certs in the “live” directory, which themselves are symlinks to whatever the latest certbot has created. For that reason, we’ve automated keeping our certificates fresh with certbot!

Remember though that for the challenge/response, we do need to keep port 80 open so that the periodic renewal can function.

Final Thoughts and Optional Configuration

If you’re using certificates and SSL, you should strongly consider disabling HTTP access on port 7474 to your Neo4j instance. Why offer unencrypted traffic when you’ve configured nice secure encrypted traffic?
Many users may wish to configure Neo4j’s ports to use port 443 for HTTPS rather than the usual port 7473. This just makes it more convenient for users to hit your instance by going to https://mycool.host.name/