Working With Neo4j Date And Spatial Types In A React.js App
Building A Dashboard App With Neo4j, Mapbox, React, and Nivo Charts
Overview
Neo4j 3.4 includes support for new spatial and temporal functionality, including Point and Date types, and indexes that enable querying using those types. I thought it would be fun to build a simple app that uses both the spatial and temporal features.
In this post we walk through the steps to build a React.js dashboard type application that allows a user to search for businesses by location that have reviews within a certain date range and display some charts based on aggregations of these reviews. We’ll explore how to use neo4j-import with the new spatial and temporal types, how to use the new types in Cypher queries and with the Neo4j drivers, and how that all fits into a React app.
Data
For this example I wanted a dataset that included both space and time components, so I decided to use the Yelp Open Dataset. This dataset released by Yelp includes a subset of the data powering the online reviews site, including over 5 million reviews of 174 thousand businesses in 11 metropolitan areas throughout the US and Europe. You can download the data here.
The Graph Data Model
There have been a few examples showing how to use the Yelp Open Dataset in Neo4j. From how to model and import the dataset, including the super fast neo4j-import tool, how to query the dataset using Cypher to find recommended businesses, and how to apply graph algorithms to the dataset.
Import
We first need to import this dataset into Neo4j. The Yelp data is provided in streaming JSON format (one JSON object per line). We have several options for how we could import this into Neo4j:
- Use
apoc.load.jsonto import the JSON files - Read each JSON line and pass as parameters to a Cypher statement
- Convert to CSV and use LOAD CSV
- Convert to CSV and use neo4j-import for fast bulk loading import
I opted for the last approach (using neo4j-import). The dataset is large enough that it will be faster to use the bulk import functionality instead of LOAD CSV or the other options. And my colleague Mark Needham had already written most of what I needed. I just extended his script to support the Point and Date types.
Neo4j-import makes use of header files that define the properties and types for the import. So we’ll need to update the headers files for Business and Review nodes. First Business :
id:ID(Business),name,address,city,state,location:Point(WGS-84)and Review :
id:ID(Review),text,stars:int,date:DateWe don’t need to change the CSV file for reviews, since Neo4j is able to parse date strings in the format already used (YYYY-MM-DD), but we will need to add out Point type in the Business csv, we write the data to the csv as a map (or dictionary) that contains latitude and longitude:
"FYWN1wneV18bWNgQjJ2GNg","Dental by Design","4855 E Warner Rd, Ste B9","Ahwatukee","AZ","{latitude: 33.3306902, longitude: -111.9785992}"We then use the neo4j-admin import command, passing in the CSVs to import the data into Neo4j.
It’s important to note that neo4j-admin import does not create indexes for us, so we’ll need to explicitly create any indexes we want to use for initial data lookups. In this case we will create an index on the location property of our Business nodes and the date property of our Review nodes:
CREATE INDEX ON :Business(location);
CREATE INDEX ON :Review(date);You can find the full code for the import here.
Hosting Neo4j In The Cloud
Since we’re building a web app we need to host our Neo4j database somewhere. Also, since we’ll probably want to serve our webpage over a secure HTTPS connection we’ll need to generate some trusted certificates for Neo4j that our web browser will accept. By default Neo4j will use a self-signed certificate, but that’s not good enough for most configurations.
Fortunately, we can use the certbot tool from Let’s Encrypt to easily generate Certificate Authority signed certificates for Neo4j:
# Use certbot to generate certificates
certbot certonly# Copy certs to Neo4j directory
cp /path_to_certs/fullchain.pem /var/lib/neo4j/certificates/neo4j.certcp /path_to_certs/privkey.pem /var/lib/neo4j/certificates/neo4j.key
I initially used this process to secure my Neo4j connection on a VPS instance (see this page for the myriad options for deploying Neo4j), but ultimately I used Neo4j Cloud, which takes care of all the hassle of obtaining certificates :-) You can sign up for early access to Neo4j Cloud here.
Queries
Now that we’ve created our database, we can write some Cypher queries to work with the data in Neo4j. We’ll focus on looking up businesses within some distance of a specific point and finding reviews within a specified date range.
Spatial Queries
The first bit of functionality we want to support is searching for businesses by location; when the user selects a point on the map we need to search for businesses within a user-defined distance (say 1000 meters). Here’s how we can do that taking advantage of the spatial index on our Point type that we just created:
MATCH (b:Business)
WHERE distance(
b.location,
point({latitude:33.329 , longitude:-111.978})
) < 1000
RETURN COUNT(b) AS num_businesses
-------------------------------------------------------num_businesses
117
We make use of the distance function to filter for businesses within 1000 meters of a point that we specify by latitude and longitude.
If we prepend our query with PROFILE we can see the execution plan to verify that we are indeed using the spatial index.
This query takes about 7ms on my laptop to find businesses within 1km of a point in the Phoenix area.
Date Queries
The next query we want to write will search for reviews within a certain date range, say between March 23, 2015 and April 20, 2015. Here’s how we can write that query:
MATCH (r:Review)
WHERE date("2015-03-24") < r.date < date("2015-04-20")
RETURN COUNT(r) AS num_reviews
-----------------------------------------------------num_reviews
63527
We have a few options for constructing dates in Cypher. We could pass each date component (year, month, day) as integers or, as we’ve done here, pass a string to be parsed to the date function. Again, we can PROFILE our query to ensure it is using the temporal index. This query takes ~12ms on my laptop, certainly better than if we had to scan over all 5 million Review nodes.
We only touched on a small piece of the new temporal functionality in Neo4j, there are also other types, likeDateTime and LocalDateTime that take timezone into account, and durations for working with time periods. You can learn more about these features in the Neo4j documentation. My colleague Adam Cowley has put together a couple of great posts showing more detail on the Neo4j temporal types and using them with JavaScript.
Putting It Together
Now that we’ve seen how to search through space and time, we can combine the queries above to search for businesses by location that have reviews within a certain time range:
MATCH (b:Business)<-[:REVIEWS]-(r:Review)
WHERE distance(
b.location,
point({latitude:33.329 , longitude:-111.978})
) < 1000
AND date("2015-03-24") < r.date < date("2015-04-20")
RETURN COUNT(b) AS num_businesses_with_reviews
---------------------------------------------------------num_businesses_with_reviews
69
This query is just giving us the count of businesses in our radius that have any reviews within our date range. To populate our UI we actually need a bit more information. You can see the full query in section below “Querying Neo4j From Our React App”.
If we inspect the PROFILE of this query we’ll see that the query planner chooses to use the temporal index on :Review(date). This makes sense since it should be the most selective index as this index contain 5 million entries (reviews), while the spatial index only contains less than 200 thousand (businesses).
Sometimes we want to force the use of one index over another using an index hint, for example if we wanted to use the spatial index instead of the temporal index. Currently index hints aren’t supported for spatial indexes, but this will be added in the next Neo4j release.
React App
Now that we have our database and queries, we’re ready to start building our web app.
Create React App
Create React App is the easiest way to start a React project. It’s a tool for creating React application skeletons without having to configure build tools like Babel and Webpack. Creating a React app with create-react-app is as easy as:
npx create-react-app spacetime-reviewsComponents
A React-based UI is made up of components that encapsulate logic for how to render a piece of the interface given some data (props). Here’s an overview of the components we’ll create for this application:
App Component
This is our main component which will handle sending queries to the database using the Neo4j JavaScript driver, storing the results in (and maintaining other application) state. All other components in our application will be children of the App component.
Map Component
I’ve previously used Leaflet.js and Mapbox for a few different projects, like this Panama Papers address geocode example and this US Congressional district map, but not in a React app. So the first thing I looked for was a React component wrapper for Mapbox GL JS. I found react-mapbox-gl, published by Uber’s data science team, which seemed like what I needed.
After starting to work with react-mapbox-gl however I felt constrained by working with only the props that the library makes available. Fortunately I found this blog post from Tristen Brown that helps explain how to use Mapbox GL JS alongside React. The basic idea is to use the Mapbox GL JS library inside our Map component, encapsulating the use of the library within this component. This felt less constraining to me, but I’m sure I could have gotten the app to work with react-mapbox-gl since Kepler.gl is built using it.
We also can use React’s two-way data binding approach for triggering a call to fetch fresh data from Neo4j when the user selects a new location on the map.
Here’s the event handler for our draggable marker in the Map component. The user drags it around the map and then releases it to select the center of a new location to search:
In addition to updating the circle showing our search area, we grab the latitude and longitude from the map (as well as the zoom level), and call this.props.mapSearchPointChange(viewport). This function is actually defined in the App component and is passed to our Map component through props as a kind of callback. This is how React’s two way data binding pattern works, as callback functions passed to child components as props. Here’s the implementation of mapSearchPointChange in the App component:
mapSearchPointChange simply updates state in the App component, which will trigger a re-render and thus a call to Neo4j to update data for the new location selected.
ReviewSummary and CategorySummaryComponents
ReviewSummary and CategorySummary are purely presentational components, responsible only for drawing charts based on the data received as props. They make use of the Nivo chart component library.
As data is retrieved from Neo4j in the App component it is passed to these two presentational components to render the charts showing the histogram of review counts by stars and a pie chart of business categories.
I found it useful to use the AutoSizer component from the React Virtualized library to allow the chart components to resize, taking the full height and width of their containers.
Querying Neo4j From Our React App
To fetch data from Neo4j we’ll query the database directly from our client application. This might not be an ideal architecture for a real world application, but will work fine for our app. A more realistic architecture might be to create a GraphQL API that queries our Neo4j database.
To actually fetch data from Neo4j we create a function fetchBusiness that contains the Cypher query we want to execute, and updates state in the App component with the result of the query. We can call this function from the appropriate component lifecycle functions, such as componentWillUpdate and componentDidMount . Note that we import the Neo4j Date temporal type and instantiate a Date object to pass as a parameter to the query.
Deploy With Netlify
To deploy our React app we just need to be able to serve static content, so we we have lots of options. I used Netlify, which makes it easy to build and deploy our app.
We can integrate a Netlify project with Github and create a git commit hook that triggers a build on each commit. We can also specify our environment variables for our Mapbox token and Neo4j credentials with Netlify.
You can find the code for this project on Github and try it live here.
