How to Automate Neo4j Deploys on AWS

Neo4j already provides some documentation on its site for how to do deployments of Neo4j to common clouds, including AWS. But in this article, I’ll provide sample shell scripts that can do this automatically for you.

These are useful when you want to integrate Neo4j into your CI/CD pipeline and be able to create/destroy instances temporarily, and also just to spin up a sample instance. But really, if you can automate Neo4j deployment, then any other piece of software can make a Neo4j instance whenever it needs, which is extremely handy.

If you have any feedback or questions on these approaches, drop by the Neo4j Community site on this thread and share!

Neo4j and AWS CloudFormation

Requirements

Before we begin, you’ll need the aws command line interface program, which you can download and install with directions here. The AWS CLI is the main way you can automate all things AWS.

It will also be necessary to generate an access token, and make sure that the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are defined for you.

CloudFormation

Neo4j provides CloudFormation templates for Neo4j Enterprise standalone (that is, a single database instance), Neo4j Causal Cluster (highly available clusters) and Neo4j Community. So first thing’s first, pick which one you would like to deploy. We’ll cover all three in this article.

CloudFormation is really just a recipe for Amazon Web Services that tells AWS how to deploy a whole set of interrelated resources. By deploying all of this as a stack we can keep all of our resources together, and delete just one thing when we’re done.

In general, Neo4j CloudFormation templates have a few properties worth knowing about before we begin:

• They deploy one more more EC2 VMs in a region you specify
• If you’re deploying more than one VM, those VMs are spread across multiple availability zones within the region, so that if one AZ goes down your entire database doesn’t go down.
• They deploy a new VPC and put Neo4j inside of it. In this way you can control network access by tuning your VPC and security rules.

Approach

How to deploy is going to be very simple: we just create a new CloudFormation stack, pointing to the right stack template URL to tell AWS what to deploy. We then will give each of our stacks various parameters to control how much hardware we’re using and so on.

We’ll need to specify several common parameters, which you’ll see in the scripts below. Here’s an explanation of what they are.

• SSH Key: When your VMs are created, this is the name of your SSH key on AWS that you’ll be able to use to SSH into the instances as the user “ubuntu”
• Network Whitelist: This is set to 0.0.0.0/0 by default, which means that any IP on the internet can contact your instance. If you want to lock it down to just your company’s IP block, this is where you’d specify that.
• Instance: This is the AWS instance type you want to launch, which controls how much hardware you’re giving the database.
• Region: Where in the world you want to deploy Neo4j. The template generally supports at least these regions, but may not work in any AWS region due to availability: us-east-1, us-east-2, us-west-1, us-west-2, eu-west-1, eu-central-1, ap-southeast-1, ap-northeast-1, ap-south-1, sa-east-1.

Let’s get started. Simply run any of these scripts, and it will result in a CloudFormation stack being deployed.

Deploying Neo4j Enterprise Standalone

This will create a single instance of Neo4j without high-availability failover capabilities, but it’s a very fast way to get started.

#!/bin/bash
VERSION=3.5.3
export SINGLE_TEMPLATE=http://neo4j-cloudformation.s3.amazonaws.com/neo4j-enterprise-standalone-stack-$VERSION.json
export STACKNAME=neo4j-enterprise-$(echo $VERSION | sed s/[^A-Za-z0-9]/-/g)
export INSTANCE=r4.large
export REGION=us-east-1
export SSHKEY=my-ssh-keyname
aws cloudformation create-stack \
   --stack-name $STACKNAME \
   --region $REGION \
   --template-url $SINGLE_TEMPLATE \
   --parameters ParameterKey=InstanceType,ParameterValue=$INSTANCE \
     ParameterKey=NetworkWhitelist,ParameterValue=0.0.0.0/0 \
     ParameterKey=Password,ParameterValue=s00pers3cret \
     ParameterKey=SSHKeyName,ParameterValue=$SSHKEY \
     ParameterKey=VolumeSizeGB,ParameterValue=37 \
     ParameterKey=VolumeType,ParameterValue=gp2 \
     --capabilities CAPABILITY_NAMED_IAM

Deploying Neo4j Enterprise Causal Cluster

When you need a clustered deploy, you want to use the Causal Cluster template. Note that the major difference here from the last one is the ClusterNodes parameter, where you can pick any number from 3 on up to indicate how many core nodes you want in your cluster.

#!/bin/bash
VERSION=3.5.3
export CLUSTER_TEMPLATE=http://neo4j-cloudformation.s3.amazonaws.com/neo4j-enterprise-stack-$VERSION.json
export STACKNAME=neo4j-enterprise-$(echo $VERSION | sed s/[^A-Za-z0-9]/-/g)
export INSTANCE=r4.large
export REGION=us-east-1
export SSHKEY=my-ssh-keyname
export CORES=3
export READ_REPLICAS=1
aws cloudformation create-stack \
   --stack-name $STACKNAME \
   --region $REGION \
   --template-url $CLUSTER_TEMPLATE \
   --parameters \
ParameterKey=ReadReplicas,ParameterValue=$READ_REPLICAS \
ParameterKey=ClusterNodes,ParameterValue=$CORES \        ParameterKey=InstanceType,ParameterValue=$INSTANCE \
     ParameterKey=NetworkWhitelist,ParameterValue=0.0.0.0/0 \
     ParameterKey=Password,ParameterValue=s00pers3cret \
     ParameterKey=SSHKeyName,ParameterValue=$SSHKEY \
     ParameterKey=VolumeSizeGB,ParameterValue=37 \
     ParameterKey=VolumeType,ParameterValue=gp2 \
     --capabilities CAPABILITY_NAMED_IAM

Deploying Neo4j Community

Using Neo4j community is barely any different than using Neo4j Enterprise (in terms of the deploy process) other than the different CloudFormation stack URL.

#!/bin/bash
VERSION=3.5.3
export COMMUNITY_TEMPLATE=http://neo4j-cloudformation.s3.amazonaws.com/neo4j-community-standalone-stack-$VERSION.json
export STACKNAME=neo4j-comm-$(echo $VERSION | sed s/[^A-Za-z0-9]/-/g)
export INSTANCE=r4.large
export REGION=us-east-1
export SSHKEY=my-ssh-keyname
aws cloudformation create-stack \
   --stack-name $STACKNAME \
   --region $REGION \
   --template-url $COMMUNITY_TEMPLATE \
   --parameters ParameterKey=InstanceType,ParameterValue=$INSTANCE \
     ParameterKey=NetworkWhitelist,ParameterValue=0.0.0.0/0 \
     ParameterKey=Password,ParameterValue=s00pers3cret \
     ParameterKey=SSHKeyName,ParameterValue=$SSHKEY \
     ParameterKey=VolumeSizeGB,ParameterValue=37 \
     ParameterKey=VolumeType,ParameterValue=gp2 \
     --capabilities CAPABILITY_NAMED_IAM

Checking to see if your instance is up

In each case, the commands above simply submit a CloudFormation stack to be deployed, they don’t wait until the stack is available. If you want to do this (so that you can trigger some next step in a software pipeline) you’ll want to wait for the CloudFormation stack to finish deploying, like so:

aws cloudformation wait stack-create-complete --region $REGION --stack-name "$STACKNAME"

This call will block until you’re finished deploying. Finally, you’ll want to get the stack outputs, like this:

aws cloudformation describe-stacks --region $REGION --stack-name "$STACKNAME"

In general this will output a lot of ugly JSON. To cut straight to the outputs of the stack, use the jq tool. By piping the output through:

jq -r '.Stacks[0].Outputs[]'

You’ll get a nice set of outputs and be able to spot the IP address and password of your new instance. By the time the CloudFormation template is finished deploying, the service will be live and ready to go!

Cleaning up and Removing Your Stack

When you’re done with your CloudFormation stack, you can delete it similarly. I use a script like this, and simply pass it the stack name that I created above as an argument to the script.

Keep in mind you must specify the region correctly.

#!/bin/bash
echo "Deleting stack $1"
aws cloudformation delete-stack --stack-name "$1" --region us-east-1

That’s it!