Data flow from the database to the UI: Three layers of Meteor

Often I get the following question from my friends:

When I’m writing a Meteor app, where do I put my database queries? It looks like there are three places: publications, subscriptions, and template helpers. Why do I have to filter my data in three different places?

This post is an attempt at a structured answer to where, when, and why you make database queries in Meteor.

Three layers.

Prologue: What publications and subscriptions are, and why we need them

Back in the day when all web apps rendered their HTML on the server and made a round trip for each action, getting some data out of the database and putting it in your view was simple — you made a database query, saved the result in a variable, then printed it out in your HTML template. In today’s world, that doesn’t cut it anymore because people expect apps to respond instantly. Especially in mobile environments, an HTTP roundtrip to fetch data can take a long time. You can’t rely on the server generating all of the HTML — you need to be able to optimistically update the user interface instantly on the client with the data you already have.

In this new world, you need to keep a copy of some of your data on the client to enable instant response times. You need a new set of primitives for sending and requesting data. You also need a smart front-end data store that knows how to integrate changes streaming from the server. In Meteor, you use Meteor.publish and Meteor.subscribe to orchestrate data flow, and Minimongo is the frontend data store that lets you make live-updating database queries on the client as if you had the whole database. Here’s a summary:

1. Publications are functions on the server that define a stream of documents
2. Subscriptions are calls on the client that tell the server we want to subscribe to a publication and get updates about its data
3. Minimongo stores all of the documents that subscriptions retrieve, and can run queries on all data that this client is currently subscribed to

A diagram of how data gets from the server to the client in Meteor.

Working with these new primitives requires some adjustment. I’d like to tell you about the workflow I’ve arrived at for designing my Meteor publications, subscriptions, and client-side Minimongo queries.

Layer 1: Think about security inside publications

The first place a document in Meteor passes through on its way to the front-end is a publication. Inside your publications, think — which documents from my database should the current user be able to see? Since any client can subscribe to any publication at any time, this is the place to filter out data that should only be seen by certain users.

Meteor.publish(“todoLists”, function () {
  // In this publication, we will be careful to only
  // return the todo lists that the current user can access,
  // using this.userId
 
  // A user can read a todo list if:
  // 1. The list is public, or
  // 2. The user is a member of this list
  return TodoLists.find({
    $or: [
      { public: true },
      { members: this.userId }
    ]
  });
});

Publication granularity — how much data do you need at once?

In addition to security concerns, we should also think about how much data each publication should return. Let’s think about the example of a todo list app. It makes sense to have all todo lists as one publication because a user might not have very many. However, we should probably only publish the tasks for one list at a time since loading all of the tasks at once might take a lot of network bandwidth or memory in the user’s browser. The rule of thumb here is that each component of your UI should be able to get the data it needs with just a few subscriptions, but it shouldn’t get more data than necessary.

Layer 2: Think about performance in subscriptions

It would be convenient to have all of the data on the client all of the time, but unfortunately loading too much data is bad for performance. Subscribing to data you don’t need uses network resources unnecessarily and takes up precious memory. Therefore, when you’re making subscriptions, think — what is the smallest amount of data I need to display this part of the UI properly? Subscribe to the right publications with the correct arguments to get only the data you need.

There’s a helpful pattern that can help you accomplish this: do your subscriptions inside templates. Then you can think about what each template needs to display itself. Including this template on any page will automatically make the right subscriptions, and they will be cleaned up when they are no longer needed.

Template.todoList.onCreated(function () {
  var self = this;
  // When this todo list template is used, get
  // the tasks we need.
  self.autorun(function () {
    self.subscribe(“tasksInList”, Template.currentData()._id);
  });
});

Once you write some subscriptions, you might find that you need to go back and define some more publications — maybe you published too much data in one publication, or maybe not enough. It might seem like an extra step, but it is a good idea to have the server explicitly define the sets of data that can be accessed so that you only have to audit your data security in one place.

Bonus: Loading indicators

There is a new feature in Meteor 1.0.4 that makes using subscriptions inside templates even more convenient — a built in helper for showing a loading indicator.

<template name=”todoList”>
  {{#if Template.subscriptionsReady}}
    … display the tasks …
  {{else}}
    Loading…
  {{/if}}
</template>

Since we used self.subscribe in the previous code snippet, we can now use the special built-in helper Template.subscriptionsReady inside our template code to know that our data is still loading. This is much better than having a generic loading template for the whole page, because the loading section is localized to the part of the page that actually has new data.

Layer 3: Think about the UI inside helpers

Helpers are functions that inject data into Meteor’s HTML templates. When you’re putting data into a template from a helper, select exactly the data you want to display. Don’t rely on being subscribed only to a certain set of documents — write the most specific MongoDB query that returns exactly what you need. Also, this is the right place to do display-specific things like sorting.

Template.todoList.helpers({
  tasks: function () {
    return Tasks.find({
      todoListId: Template.currentData()._id
    }, { sort: { createdAt: -1 } });
  }
});

Overview

The combination of publications, subscriptions, and Minimongo are my favorite part of Meteor. They abstract the sometimes difficult design decisions about getting your data to the right place in a flexible yet simple way.

Going back to the question at the beginning, it’s true that sometimes you have to write the same database query twice — once in your publication and then again inside your helper. But those two queries serve different purposes: the one inside the publication is for security, and the one inside the helper is for the UI.

Using these tools, you can take a systematic approach to designing the data flow in your application and avoid the tangle of Websocket events, REST endpoints, and event listeners that you would otherwise need to achieve the same effect. Minimongo is what makes it all efficient — it lets us simply reuse data we have already retrieved without unloading it and loading it again from the server.

If you already use Meteor, hopefully this post has demystified how you can use Meteor.publish and Meteor.subscribe in a principled way. If you haven’t tried Meteor yet and this sounds interesting, give it a shot!