How to Automate Neo4j Deploys on Azure

Neo4j already provides some documentation on its site for how to do deployments of Neo4j to common clouds, including on Azure. But in this article, I’ll provide sample shell scripts that can do this automatically for you.

These are useful when you want to integrate Neo4j into your CI/CD pipeline and be able to create/destroy instances temporarily, and also just to spin up a sample instance. But really, if you can automate Neo4j deployment, then any other piece of software can make a Neo4j instance whenever it needs, which is extremely handy.

If you have any feedback or questions on these approaches, drop by the Neo4j Community site on this thread and share!

Neo4j on Azure using Azure Resource Manager (ARM) templates

Requirements

Before we begin, you’ll need the az command line interface program, which you can download and install with directions here. The azCLI is the main way you can automate all things with Azure. You’ll also need the jq tool for working with JSON responses, and other common linux utilities.

It will also be necessary to authenticate your az CLI, to make sure it can interact with your resource groups, and that it is configured to use the right subscription by default. You can change the subscription the tool uses by consulting the documentation.

Azure Resource Manager

Neo4j provides Azure Resource Manager (ARM) templates for Neo4j Causal Cluster (highly available clusters), and VM images for Neo4j Enterprise stand-alone. So first thing’s first, pick which one you would like to deploy. We’ll cover both in this article.

ARM templates are really just a recipe that tells Azure how to deploy a whole set of interrelated resources. By deploying all of this as a stack we can keep all of our resources together, and delete just one thing when we’re done. In this method, we will be creating a new resource group for everything that we need for our Neo4j setup, so that we can manage the entire instance by just the resource group.

Approach

How to deploy a cluster is very simple: we just submit a new ARM Deployment job, pointing to the right template URL and providing some parameters in a JSON file. To keep things simple and contained, we’ll create an Azure resource group for our deploy to group everything we need.

We’ll need to specify several common parameters, which you’ll see in the scripts below. Here’s an explanation of what they are.

• VM Size: This is the Azure VM type you want to launch, which controls how much hardware you’ll be using.
• Disk Size/Type: you can use these parameters to control whether Neo4j uses standard spinning magnetic platters or SSD disks as well as how many GB of storage you want to allocate.
• Location: Where in the world you want to deploy Neo4j. The template supports any Azure location, but in our example we’ll use East US.
• Authentication details: specifically, the administrative username and password for access to the VMs. It’s possible to set up the administrative login via either SSH key or simple username/password. We’ll do it with a password because there are no extra pre-requisites to set up.

You can also choose other considerations like static/dynamic IP addresses, and naming in the script.

Let’s get started. Simply run any of these scripts, and it will result in a running Neo4j instance you can use immediately!

Deploying Neo4j Enterprise Causal Cluster

Because this is a clustered deploy, take note that we’re also using a parameter for “Cores” and “Read Replicas” to control how many nodes are in our cluster. This is in addition to the parameters we described above.

The way ARM works, we’ll create a simple JSON file with all of the parameters we want submitted to the deployment, then create the deployment pointing to that JSON file. The ARM template is like a small function: it takes some inputs we’ll provide here, and produces a set of infrastructure as an output.

#!/bin/bash
export CORE_NODES=3
export READ_REPLICAS=0
export NEO4J_PASSWORD=s00pers3cR3T:
export ADMIN_AUTH_TYPE=password
export USERNAME=graph-hacker
export ADMIN_PASSWORD=s00pers3cR3T:
export VM_SIZE=Standard_B2ms
export DISK_TYPE=StandardSSD_LRS
export DISK_SIZE=256
export IP_ALLOCATION=Dynamic
export SEED=$(head -c 3 /dev/urandom | base64 | sed 's/[^a-zA-Z0-9]/X/g')
export RESOURCE_GROUP="neo4j-RG-${SEED}"
export CLUSTERNAME="neo4j-${SEED}"
export DEPLOYMENT=neo4j-bmdeploy
export LOCATION="East US"
# The ARM template to deploy
export TEMPLATE_BASE=http://neo4j-arm.s3.amazonaws.com/3.5.5/causal-cluster/
export TEMPLATE_URL=${TEMPLATE_BASE}mainTemplate.json
echo $(cat <<JSON
{
"ClusterName": { "value": "${CLUSTERNAME}" },
"CoreNodes": { "value": ${CORE_NODES} },
"ReadReplicas": { "value": ${READ_REPLICAS} },
"VmSize": { "value": "${VM_SIZE}" },
"DataDiskType": { "value": "${DISK_TYPE}" },
"DataDiskSizeGB": { "value": ${DISK_SIZE} },
"AdminUserName": { "value": "${USERNAME}" },
"AdminAuthType": { "value": "${ADMIN_AUTH_TYPE}" },
"AdminCredential": { "value": "${ADMIN_PASSWORD}" },
"PublicIPAllocationMethod": { "value": "${IP_ALLOCATION}" },
"Neo4jPassword": { "value": "${NEO4J_PASSWORD}" },
"_artifactsLocation": { "value": "${TEMPLATE_BASE}" }
}
JSON
) > "${RESOURCE_GROUP}.json"
echo "Creating resource group named ${RESOURCE_GROUP}"
if ! az group create --name "${RESOURCE_GROUP}" --location "${LOCATION}"; then
   echo STACK_NAME=$RESOURCE_GROUP
   echo "Failed to create necessary resource group ${RESOURCE_GROUP}"
   exit 1
fi
echo "Creating deployment"
az group deployment create \
  --template-uri "$TEMPLATE_URL" \
  --parameters @./${RESOURCE_GROUP}.json \
  --resource-group "${RESOURCE_GROUP}" \
  --name "${DEPLOYMENT}"
if [ $? -ne 0 ] ; then
  echo STACK_NAME=$RESOURCE_GROUP
  echo "Stack deploy failed"
  exit 1
fi
# JSON Path to server response where IP address is.
ADDR_FIELD=".[].virtualMachine.network.publicIpAddresses[0].ipAddress"
IP_ADDRESS=$(az vm list-ip-addresses --resource-group "${RESOURCE_GROUP}" | jq -r "$ADDR_FIELD" | head -n 1)
echo STACK_NAME=$RESOURCE_GROUP
echo NEO4J_URI=bolt+routing://$IP_ADDRESS:7687

At the end of this, a new resource group is created with all of the assets inside of it, and you get a URI of a bolt endpoint you can use. Alternatively, you could go to https://$IP_ADDRESS:7473/ to access Neo4j Browser for your new clustered instance.

Example of a deployed resource group

Deploying Neo4j Enterprise Stand Alone

This will create a single instance of Neo4j without high-availability failover capabilities, but it’s a very fast way to get started. For this deploy, we don’t use ARM but just create a simple VM and configure its firewall/security rules.

Neo4j provides the VM through an Azure marketplace offer. To refer to the right VM image, you need to know the publisher (that’s Neo4j), the “offer” (Neo4j 3.5 series) and the SKU (which is the particular version of Neo4j you’ll use).

Because we’re not using ARM for this one, this also provides an example of polling and waiting until the VM service comes up, and then changing the Neo4j default password when it does. At the top, you can choose a different password for the neo4j user as for the system administrator.

Make sure to customize the SUBSCRIPTION variable to make this work.

#!/bin/bash
export LOCATION=eastus
export SUBSCRIPTION=My-Subscription-Name
export RG=neo4j-standalone-RG
export NAME=neo4j-standalone
export ADMIN_USERNAME=graph-hacker
export ADMIN_PASSWORD=ch00se:A@PASSw0rd
export NEO4J_PASSWORD=ch00se:A@PASSw0rd
export NETWORK_SECURITY_GROUP=neo4j-nsg
# Options: https://azure.microsoft.com/en-us/pricing/details/virtual-machines/
export VM_SIZE=Standard_D2_v3
# Can change this to static if desired
export ADDRESS_ALLOCATION=dynamic
# Configuration bits of what you're launching
# Publisher:Offer:Sku:Version
export PUBLISHER=neo4j
export OFFER=neo4j-enterprise-3_5
export SKU=neo4j_3_5_5_apoc
export VERSION=latest
export IMAGE=$PUBLISHER:$OFFER:$SKU:$VERSION
echo "Creating resource group named $RG"
az group create --location $LOCATION \
   --name $RG \
   --subscription $SUBSCRIPTION
echo "Creating Network Security Group named $NETWORK_SECURITY_GROUP"
az network nsg create \
   --resource-group $RG \
   --location $LOCATION \
   --name $NETWORK_SECURITY_GROUP
echo "Assigning NSG rules to allow inbound traffic on Neo4j ports..."
prio=1000
for port in 7473 7474 7687; do
  az network nsg rule create \
    --resource-group $RG \
    --nsg-name "$NETWORK_SECURITY_GROUP" \
    --name neo4j-allow-$port \
    --protocol tcp \
    --priority $prio \
    --destination-port-range $port
  prio=$(($prio+1))
done
echo "Creating Neo4j VM named $NAME"
az vm create --name $NAME \
  --resource-group $RG \
  --image $IMAGE \
  --vnet-name $NAME-vnet \
  --subnet $NAME-subnet \
  --admin-username "$ADMIN_USERNAME" \
  --admin-password "$ADMIN_PASSWORD" \
  --public-ip-address-allocation $ADDRESS_ALLOCATION \
  --size $VM_SIZE
if [ $? -ne 0 ] ; then
  echo "VM creation failed"
  exit 1
fi
echo "Updating NIC to have our NSG"
# Uses default assigned NIC name
az network nic update \
  --resource-group "$RG" \
  --name "${NAME}VMNic" \
  --network-security-group "$NETWORK_SECURITY_GROUP"
# Get the IP address of our instance
IP_ADDRESS=$(az vm list-ip-addresses -g "$RG" -n "$NAME" | jq -r '.[0].virtualMachine.network.publicIpAddresses[0].ipAddress')
export NEO4J_URI=bolt://$IP_ADDRESS
# Change password
echo "Checking if Neo4j is up and changing password...."
while true; do
   if curl -s -I http://$IP_ADDRESS:7474 | grep "200 OK"; then
     echo "Neo4j is up; changing default password" 2>&1
     curl -v -H "Content-Type: application/json" \
       -XPOST -d '{"password":"'$NEO4J_PASSWORD'"}' \
       -u neo4j:neo4j \
       http://$IP_ADDRESS:7474/user/neo4j/password 2>&1
     echo "Password reset, signaling success" 2>&1
     break
   fi
   echo "Waiting for neo4j to come up" 2>&1
   sleep 1
done
echo NEO4J_URI=$NEO4J_URI
exit 0

Cleanup

To clean up and delete these deployment when you’re done, simply delete the entire resource group. Here’s a simple shell script which will get the job done.

In both cases (cluster and standalone) this works nicely because we put all resources together into a single resource group. So by dropping the group, there’s no individual resource cleanup, the entire setup goes away.

#!/bin/bash
if [ -z $1 ] ; then
  echo "Usage: call me with deployment name"
  exit 1
fi
STACK_NAME=$1
if [ -f "$STACK_NAME.json" ] ; then
   rm -f "$STACK_NAME.json"
fi
az group delete -n "$STACK_NAME" --no-wait --yes
exit $?

This will schedule the entire group to be deleted and the script will exit immediately. Actually deleting all the resources may take some minutes, and if you log into your Azure console you’ll see the progress in the notifications area.