Should we allow custom ddp messages ?

The ddp protocol lives in the heart of the Meteor framework, its clear and straightforward architecture is great and one of the strengths of this framework.

On the other hand, sometimes the ddp’s closed nature, limits/encumbers optimizations which are required in order to mitigate scaling issues (re-use subscription between many clients - 500+, sending heavy load of messages 1000+ updates/sec).

This short blog post includes an overview of a base architecture used in an app I’m developing, my need for custom ddp messages and how it is being implemented into the ddp-client package (one of meteor base packages).

The solution combines a couple of optimizations. Some of them aren’t related to the ddp, but it will be interesting to get the community's thoughts on these solutions.

A short summary of the optimization:

The optimization is called “batch publication”. Instead of just re-using the multiplexer for same live query in a publication (nice out-of-the-box meteor feature), for each unique publication query we compute once the messages needed to be sent over the socket, and send the stringify message as a batch to all listening clients. We ignore the merge-box so we can grow the number of clients listening to the subscription with almost static memory footprint.

This solution has many pros (much better performance) and some cons as well (no merge box). Like almost any optimization, it depends on your app requirements.

You can read more details about the optimization and its pros and cons in this blog post — How to send 2000+ updates/sec per client, using meteor, and stay alive.

This article proposes an option to extend the ddp protocol, so users can specify and extend the variety of allowed messages sent over the meteor connection socket. The proposed solution (PR) #6117 is not breaking anything in the protocol nor creating problems for already existing ddp clients (ObjectiveDDP for example). It just allows you to create support for custom messages, so each user who extends his ddp connection with custom messages, will also be able to extend his other ddp clients in the same manner.

This solution supports two optional ways to extend the ddp messages:

1. Giving an optional parameter in the connection creation -

var options = { customMessages : { updateBatch: function() {} }, …);
DDP.connect(options);

2. Extending Meteor.connection or any other connection -

_.extend(Meteor.connection._customMessages, { 
 updateBatch: function() {}, 
 subscriptionRedirect: function(){}
});

Besides this, all that needs to be added to the ddp client code, is a check whether the message is a custom message. This check can be done last after all the original ddp messages.

For better use I think we should add to the PR an option to extend all newly created connections:

DDP.extendCustomMessages(options)

Now each new connection that will be created after calling this function, will know about the custom ddp messages.

I know that there are “hackish” ways to implement this solution without changing the ddp-client package (for example by sending messages through a ddp.added message with different parameters, and changing the client store for this subscription), but I believe that an option to extend the ddp with your own custom messages (BatchUpdate, SubscriptionRedirect for example) is required, and a much cleaner solution.

The solution of “batch publication” heavily depends on the option to extend the ddp protocol, but there are many other interesting solutions that could evolve from this option. Another new ddp message that is being used in a project I’m working on is ‘subscriptionRedirect’. This message tells the user that he should get this subscription from server handling the query that already has the data cached (subscription level routing). I think that an option to extend the ddp protocol will drive a lot of new creative solutions that could help us solve the burning issues regarding the scaling of meteor production apps.