illustration by Adrien Griveau

How to make Meteor web apps communicate together

a comparison with the REST API method

As a Hacker in Residence at eFounders (which is a startup studio, a startup that builds startups), I develop internal tools to help our startups grow faster and better. One of them is an app that enables our marketing team to build curation newsletters very easily: users can curate web pages using a bookmarklet or a Slack command line, format the newsletter — or create templates — and send it (we use Mailjet to send the emails). Each curation list comes with its own pre-built website, where people can browse all the curated content.

You can see an example here: SaaS Club (newsletter about SaaS) or here: BuildTogether (newsletter about Startup Studios).

It was important for us to create a clear separation between our backend tool servers (only accessible by us, which means a low traffic) and the public curation frontend servers (accessible by all user, which means potentially a lot of traffic).

That way, we can have two different server architectures at the same time. One for our internal tools and one for curation frontend. These 2 clusters can evolve differently and even if we have too much traffic on the curation frontend. If the frontend is down because of a peak in traffic, our internal tools are still working.

Why the “Classic” (non-meteor way), a REST API, couldn’t work

In a “classic” stack (ror, mean …), we would have created a REST API on our backend server, so that the frontend server could access this API to get the content and store it. Then we would have created another REST API on the frontend server so that the clients can access this content.

Since it’s a REST API, the frontend server would have had to regularly call the backend server to know if there’s new content, modified content, removed content.

Why and how we chose the Meteor way (DDP)

Now, let see how we did that with Meteor.

Since we have a real-time platform, Meteor cannot rely on REST API.

It mainly uses WebSockets (or long polling if WebSockets are not available) to create a real-time communication channel and then uses DDP (Distributed Data Protocol) to transfer data in both ways. DDP is like “REST for WebSockets”. Like REST, it is a simple, pragmatic approach to providing an API. Yet, it is WebSocket-based, unlike REST, allowing it to be used to deliver live updates as data changes.

How is it done?

On the internal curation tool (backend server)

Curation contents are stored on a MongoDB collection called CurContents.

CurContents = new Mongo.Collection('curContents');

We have to expose a subset of data contained inside this collection so that the frontend server can get them. In Meteor, we use the publish / subscribe system to do that:

Meteor.publish('curContents', function (curListId) {
  check(curListId, String);
  return CurContents.find({ curListId: curListId, curCampaignId: { $exists: true } });
});

It creates a new publication called curContents that will check the curListId parameter sent by the subscriber and then return all documents related to this list to the subscriber.

curCampaignId: { $exists: true } tells Meteor to only give the contents already sent by email in a campaign. We don’t want to expose the contents that is already in our tool but not released in a newsletter.

That’s all we have to do on the backend.

On the curation website (frontend server)

In a classic Meteor app, it’s the client who subscribes to the server publications.

But in our case, we don’t want all clients to connect to our backend so it’ll be the dedicated curation website (frontend server) who will subscribe to the backend.

So on the server part of the frontend server:

• We create a connection to backend server:

var cnx = DDP.connect('ip_of_our_backend_server');

• Now we have a connection, we create a collection that will contains all the documents sent by the backend:

CnxCurContents = new Mongo.Collection('curContents', { connection: cnx });

note: The ‘cnx’ is passed to the collection so it knows where to get/send the data.

note2: We called the collection ‘CnxCurContents’ to make it easier to understand but we could have called it the same name as on the backend; ‘CurContents’.

• Since Collections are persistent (stored on mongodb), we clear all the data at the frontend server launch time:

CnxCurContents.remove({});

• Let subscribe to the backend publication to get the data:

cnx.subscribe('curContents', Meteor.settings.curListId);

note: We get the ‘curListId’ from the settings.json file and give it to the subscription so the publish will know which listId we want data from.

With this 4 lines of code, the frontend now has a fully synced collection with the backend server. Each time a content is added (and published in a campaign), modified or removed, the backend will send it instantly to this frontend.

Sync between frontend server and clients

The final part is to send the data to a client connected to the frontend. We just have to create a new “publish”, this time on the frontend server, and a “subscribe” on the client.

Imagine we send all the data to the client (don’t do that for real, it’s only for the example!): in the real application, you only publish what the client really needs (i.e.: 20 last contents).

• on the frontend, we create a publication:

Meteor.publish('curContents', function () {
  return CnxCurContents.find(),
});

Note: we publish the collection ‘CnxCurContents’ that is filled by the subscription to the backend server.

• on the client, we create a subscribe:

ClientCurContents = new Mongo.Collection('curContents');
Meteor.subscribe('curContents');

Note: we created a collection on the client called ‘ClientCurContents’ but we could have called it also ‘CurContents’. This collection on the client is stored inside a minimongo. It’s mini mongodb where we can call mongo queries on it but it’s not persistent, just on the browser ram.

And voila, the client now has all the data and can access, filter, sort them with mongo queries, directly on the client. For example, to get the 5 most recent contents:

ClientCurContents.find({}, { createdAt: -1, limit: 5 });

Conclusion

As we’ve seen, Meteor doesn’t use REST API (even if it’s really easy to create your own REST API) because it doesn’t support the real-time nor the websockets needed by modern applications. It defines a new simple open protocol called DDP. Today, more and more wrappers exist in different languages to handle DDP.

That said, the fact that Meteor uses websockets as the way to communication between client and server can be problematic. Websocket-based applications are still not easy to scale and it can be a big challenge if your app has to handle a lot of simultaneous users.

Hope it’s useful. If you’re a Meteor-friendly curious developer, I might have a job for you (in Paris !) — ping me on Twitter.

This article is part of the publication Unexpected Token powered by eFounders — startup studio. All eFounders’ startups are built by extraordinary CTOs. If you feel like it could be you, apply here.