Mastering Meteor Live Data Updates

How to activate oplog-trailing for instantaneous view updates in Meteor

Are you struggling with slow view updates after switching the MongoDB? Your Meteor views used to update instantaneously on data changes in the MongoDB, but now seem to be polling data in batches?

That was exactly what happened to us at docoyo, when we deployed our Meteor app to the Ubuntu staging machine. We are building a tracking and tracing solution which requires a lot of near-real-time updates on live dashboards. Once we switched from the local MongoDB to a dedicated MongoDB on the staging machine, the whole app changed its behavior. The app seemed to be reloading data in intervals of about 10 seconds, instead of updating individual pieces of data immediately. — For a live dashboard, this would have been completely unacceptable.

So, here is the good news: What you are experiencing is very likely not a performance issue, but rather a misconfigured environment.


Unfortunately, the Meteor docs do not tell you a lot about the fact that publications on MongoDB collections work in two different ways:

1) There is oplog-trailing, which looks at incoming changes and immediately publishes them to the client. This is the mode of operation needed to experience the magical and astonishing live update behavior that you desire.

2) There is also poll-and-diff, which regularly performs query on the collections (by default in intervals of 10 seconds) and publish relevant changes to the client in batches. This is the mode of operation that you probably are experiencing right now.

To get live updates, we must use option 1.

The secret behind the live updates in Meteor is that Meteor relies on Mongo’s so called oplog, which is pretty similar to transaction logs of relational databases. In the oplog, all document changes are kept temporarily in a list. The primary purpose of the oplog is replication. MongoDB uses it to transfer data from the primary (master) instance to secondary (replica) instances. Meteor exploits this mechanism and continuously scans the oplog for relevant changes which need to be published to the client.

The oplog, however, is only available if you turn Mongo’s replication feature on. We installed MongoDB using the original setup routine by MongoDB, in which replication is turned off. In contrast, if you create a fresh Meteor project on your machine, the MongoDB running on localhost:3001 is already configured to run as replica set with one and only one instance. You can double-check this if you connect to the local MongoDB on port 3001 and list the collections within the admin database. You will find a collection named “local” which is holding the oplog entries. Now log in to your target MongoDB instance and it will probably miss the “local” collection.


So, the first thing we need to do is turning the standalone MongoDB instance into the primary instance of a replica set.

Before you start, do not forget to stop the running MongoDB service.

Change the MongoDB configuration which you typically find at /etc/mongod.conf

/etc/mongod.conf

The important part is to add the last two lines. They tell MongoDB to be aware of the fact that it is going to be part of a replica set. The replica set does not exist yet, so we need to create it in the next step.

After you have restarted your MongoDB server, log into a MongoDB shell and execute the following command:

This creates the replica set and makes your host the primary role. MongoDB will start creating an oplog from now on. To use oplog-trailing in Meteor, you do not need to add any secondary hosts to the replica set.

Now, create a new MongoDB user who is allowed to read the oplog:

That’s all you need to do.

The only thing left is to restart the Meteor app with oplog trailing activated. You can do this by specifying the URL where Meteor can find the oplog using an environment variable:

Note that you still need to specify the MONGO_URL, too. As you can see in the example above, you would typically want to set up a read-only user for oplog access and a read-write-user as the regular user running the Meteor app.

After starting the application this way, you should gain back the desired live update behavior.

PS. When working with oplog trailing, you should be careful not to overwhelm the server with heavy-weight subscriptions. I came across this well written article from the Meteor blog and found it helpful.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.