Meteor + Any Database

Meteor is closely integrated with Mongo, which gives developers exceptional out-of-the-box functionality that you don’t find with other frameworks. The built-in reactivity and latency compensation are nothing short of magical.

Data reactivity in Meteor is built upon Cursor.observeChanges. Sadly, most databases do not support this degree of reactivity. In fact, Meteor had to built its own JavaScript version of Mongo (minimongo) in order to support Cursor.observeChanges with Mongo.

For databases that do not support Cursor.observeChanges, we can still make them reactive by rerunning queries and diff’ing the results. Luckily, Meteor provides us with an efficient implementation to diff two collections of documents and call the appropriate observer methods.

Using this system, all one needs to do to make a datasource reactive is set a dependency on which a change would trigger the query to rerun. You can set these dependency keys in each publication, and trigger them to rerun after a database write.

This is how the new package ccorcos:any-db provides reactivity for any database or datasource. It currently works with Mongo, Neo4j, RethinkDB, and REST APIs, but it can in theory work with any database. I’ve been told that Chet, the package author, met with Ben Green — author of numtel:pg and numtel:mysql — in Los Angeles on July 6th during Ben’s Santa-Cruz-to-Baja-and-back bike trip to discuss the possibility of collaborating to get SQL working with this package.

This package breaks Meteor’s “Database Everywhere” principle and settles with a Subscription-Cursor object on the client. That said, this pattern comes with several benefits:

  1. You do not need to specify your queries both on the client and on the server, which means you no longer need to specify the same query in both Meteor.publish and Template.helpers.
  2. This package separates the results of individual subscriptions without sending repetitive data. Sometimes with more complicated queries, such as aggregations, it is not possible to separate the results with a client-side query after all the documents are mixed up in a minimongo collection.

This package also implements latency compensation — another important principle of Meteor. This is accomplished using observer methods.

Here’s how you can use it in your next project. We’re going to create a chatroom with Neo4j, in which we will specify subscription dependencies so that it is reactive.

This app will have chatrooms. Each of these chatrooms will have messages that are unique to that room. In the UI, you can select a room on the left, and it will display all the messages associated with that room.

Step 1

Add the package from the command line:

meteor add ccorcos:any-db

Step 2

Download Neo4j (see Additional at the end of the article for more options), then start it up and download the package to integrate it with Meteor. If you do not have brew, you should. Or if you’re on Linux, sudo apt-get is also fine. We’re also going to use Ramda:

brew install neo4j
neo4j start
meteor add ccorcos:neo4j
meteor add ramda:ramda

Step 3

Now it’s time set up a publication. The publication below will create the new Neo4j database object, then publish chatrooms, which lists all the active chatrooms in descending order:

if Meteor.isServer 
# Connect to Neo4j
@Neo4j = new Neo4jDB()

DB.publish
name: 'chatrooms'

# Creates the query for all chatrooms
query: ->
Neo4j.query """
MATCH (room:ROOM)
RETURN room
ORDER BY room.createdAt DESC
"""
    # Sets any dependencies for the chatroom subscription
depends: -> ['chatrooms']

Then we need to publish the messages within the chatrooms, also within Meteor.isServer:

DB.publish
name: 'msgs'

# Creates the query for all messages within the current chatroom
query: (roomId) ->
Neo4j.query """
MATCH (room:ROOM {_id:"#{roomId}"})-->(msg:MSG)
RETURN msg
ORDER BY msg.createdAt DESC
"""

# Sets a dependency only on the chatroom with the current roomId
depends: (roomId) -> ["chatroom:#{roomId}"]

Step 4

Now we subscribe to those publications. The code below will subscribe to the chatrooms and watch for the roomId to change. If it does change, it will display the messages associated with that roomId. This is slightly different than what we’re used to with Meteor, as the subscription must be started and stopped. This can be accomplished within an @autorun or within a Template.onRendered and Template.onDestroyed.

We also need to set the default current roomId.

if Meteor.isClient  
Session.setDefault 'roomId', null
Session.setDefault 'msgs', []
  # Creates a subscription object that will contain 
# the subscription for both rooms and messages

@subs = {}
subs.rooms = DB.createSubscription('chatrooms')
  Template.main.onRendered ->
# When the template is rendered, start the rooms subscription
@autorun -> subs.rooms.start()
# Watch for the roomId to change
@autorun ->
roomId = Session.get('roomId')
if roomId
# Start a subscription for the msgs of that room
subs.msgs = DB.createSubscription('msgs', roomId)
subs.msgs.start()
# Watch for changes to the messages
Tracker.autorun ->
Session.set('msgs', subs.msgs.fetch())

Step 5

Then we set up Meteor.methods to write to the database. The methods below will create a new room and a new message.

Meteor.methods
newRoom: (id) ->
check(id, String)
room =
_id: id
createdAt: Date.now()
if Meteor.isServer
# Creates a new room
Neo4j.query "CREATE (:ROOM #{Neo4j.stringify(room)})"
# Triggers re-query for all chatrooms dependencies
DB.triggerDeps('chatrooms')
else
# Much of what you see below is for latency compensation
fields = R.pipe(
R.assoc('unverified', true),
R.omit(['_id'])
)(room)
subs.rooms.addedBefore(id, fields, subs.rooms.docs[0]?._id or null
subs.rooms.addUndo id, -> subs.rooms.removed(id)
Session.set('roomId', id)
  newMsg: (roomId, id, text) ->
check(id, String)
check(text, String)
msg =
_id: id
text: text
createdAt: Date.now()
if Meteor.isServer
Neo4j.query """
MATCH (room:ROOM {_id:"#{roomId}"})
CREATE (room)-[:OWNS]->(:MSG #{Neo4j.stringify(msg)})
"""
DB.triggerDeps("chatroom:#{roomId}")
else
fields = R.pipe(
R.assoc('unverified', true)
R.omit(['_id'])
)(msg)
subs.msgs.addedBefore(id, fields, subs.msgs.docs[0]?._id or null)
subs.msgs.addUndo id, -> subs.msgs.removed(id)

Step 6

Now that we have our publication, subscription, and a method for writing a new message, we need to set up helpers and events to make this a functioning demo. The code below creates helpers that returns a cursor to messages and rooms from our subscription, and allows the user to call the newMsg and newRoom methods. Add the following code within an if Meteor.isClient:

Template.main.helpers 
rooms: -> subs.rooms
msgs: -> Session.get 'msgs'
isCurrentRoom: (roomId) -> Session.equals('roomId', roomId)
currentRoom: (roomId) -> Session.get('roomId')
Template.main.events 
# When you click on a room within the rooms list,
# you set the current room to the one you selected.

'click .room': -> Session.set('roomId', @_id)
# Creates a new room
'click .newRoom': (e,t) ->
id = Random.hexString(24)
Meteor.call 'newRoom', id, (err,res) ->
if err then subs.rooms.handleUndo(id)
# Creates a new message
'click .newMsg': (e,t) ->
input = t.find('input').value
id = Random.hexString(24)
Meteor.call 'newMsg', Session.get('roomId'), id, input, (err,res) ->
if err then subs.msgs.handleUndo(id)
input = ''

Step 7

Then, in your html file, add the following code for a functional app that shows you latency compensation in action (as demonstrated in the gif above):

<body>
{{>main}}
</body>
<template name="main">
<div class="wrapper">
<div class="left">
<div class="row">
<button class="newRoom">New Room</button>
</div>
{{#each rooms}}
{{#if isCurrentRoom _id}}
<div class="row room selected">
<!-- Each item has an "unverified" field that is true
if it is rendered due to latency compensation and
false if the item has been verified by the server -->
{{_id}}{{#if unverified}}*{{/if}}
</div>
{{else}}
<div class="row room">
{{_id}}{{#if unverified}}*{{/if}}
</div>
{{/if}}
{{/each}}
</div>
<div class="right">
{{#if currentRoom}}
<div class="row">
<input type="text">
<button class="newMsg">NEW MSG</button>
</div>
{{#each msgs}}
<div class="row msg">
{{text}}{{#if unverified}}*{{/if}}
</div>
{{/each}}
{{/if}}
</div>
</div>
</template>

Step 8

And while we’re at it, go ahead and copy-paste the following CSS:

.wrapper {
display: flex;
}
.left {
flex: 0 0 250px;
text-align: center;
}
.right {
flex: 1 1 0;
}
.room.selected {
background-color: blue;
color: white;
border-radius: 5px;
}
.row {
padding:5px;
}

And that’s it!

I’ve included a live example with a few alterations here. The code can be found here. The alterations are: 1) added the ability to reset the database with the built-in Neo4j.reset() method from the ccorcos:neo4j package, 2) moved the database to GrapheneDB for deployment (for more information on how to do that, go here), and 3) added an event listener so you can press “enter” so you don’t have to click the button every time.

Additional

If you want to isolate your databases for each project instead of creating a global database, see notes from this package.

You also shouldn’t publish your GrapheneDB keys to a public repo like I have in my example. The best way to protect yourself is to use Meteor.settings and keep your settings.json file private. If you are unfamiliar with how to do this, here is an easy tutorial that gives you the basics.


Sam Corcos is the lead developer and co-founder of Sightline Maps, the most intuitive platform for 3D printing topographical maps, as well as LearnPhoenix.io, an advanced tutorial site for building scaleable production apps with Phoenix and React.

Show your support

Clapping shows how much you appreciated Sam Corcos’s story.