Workaround for Meteor DDP limitation and client part of Meteor.subscribe() nuance if you want to subscribe for more nested fields of already received documents

Sounds like a rare case, doesn’t it? But it is not. See #903, #998 & #3361. It’s only a matter of time until you face it.

Better to know about that when you modeling documents structure for a new app or faced with this limitation on already established document models like we at alpheratz.co and need a workaround.

I think the most possible scenario when you could face it — it’s a reusing subscriptions. In the 2014 thanks for Arunoda from Meteorhacks Meteor community got meteorhacks:subs-manager package to make apps more efficient by making subscriptions caching play nice with Iron Router. And we at Alpheratz decided to cache subs especially as one of the most important keys to get smooth views transitions in the future.

But something went wrong

Example of the issue

Here the whole document structure for the album from Albums MongoDB collection:

When you go to the published version of the album you subscribe only for the .published nested fields. Since you visited album once — data was cached.

Imagine you are the author of the album and want to edit it. You go to the album edit route https://www.alpheratz.co/my/a/YWNXpEzdfo7TEAPxZ/edit (won’t work if you are not logged in and not an album author). Subscrition on that route subscribes for the whole album document.

note that we didn’t unsubscribe from album data received earlier since we cached that sub with subs-manager

Since you already have all album fields except .unpublished nested fields you expect that Meteor efficiently send only them to you.

Unfortunately it won’t happen. You will only have what you already had — title and slug with .published fields.

You won’t face with it if you previously unsubscribed from album data and subscribe to the whole album doc again. It’s not our case — we building efficient app which sends as minimal data as possible.

Why you can’t subscribe for more nested fields of already received documents?

At first we subscribed for published version of album data and Meteor sent to following DDP message to the client:

Then we went to album edit and sent subscription request for the whole album doc especially expecting .unpublished fields.

On the server Meteor MergeBox decides how to smartly send as little as possible new data to the client. And MergeBox would be happy to send back to you only that missing .unpublished fields in DDP message like this:

But it won’t happen.

DDP protocol has limitations.

“right now DDP only exposes top level fields of objects, which does keep the protocol much less complex.” — David Glasser, MDG

DDP message sent to us was like this:

Yeah, it’s over publishing but only by title and slug top-level fields.

So, OK OK. We actually subscribed for new nested fields data but why it didn’t appear in client collection?

Beacause of another Meteor nuance:

“If more than one subscription sends conflicting values for a field (same collection name, document ID, and field name), then the value on the client will be one of the published values, chosen arbitrarily.” — Meteor.subscribe last paragraph in Meteor API docs.

Arbitrarily.

We need a workaround.

Workaround

If you planning to build new app or ready to refactor current doc models

You can model documents structure in other ways.

Like this:

Now when you go to album editor album doc extending will happen by the top-level .unpublished fields.

Or like this:

Idea is to expose nested fields to the top-level.


I think both ways are wrong even if you modeling docs structure from scratch for a new app. Examples above is very simple. But I imagine how we will decide to add more fields related to the album title such as pre and sub title, styling and etc. Overall album document fields number may grow in the future. Doc will look like a mess without logic. Keeping logically separate data under own field is much cleaner even if parent field name is too vague.

And of course it’s a pain to restructure already established docs structure for in-production app.

Best solution should work nice with current docs structure and be ready to be thrown away when DDP protocol evolve.

If you want to stay on your current docs model and be flexible with your app and Meteor future changes

Our solution is to use supportive client-only collections to publish nested fields into them.

Supportive because their role is just to be fixing collection that contains only docs with only nested fields that cause issues.

It helps to avoid over publishing because of DDP limitations and client side same doc arbitrarily fields merging.


Let’s add client-only collection CC_Albums_unpublishedFields

«CC» — Client Collection

«Albums» — supportive for Albums collection

«unpublishedFields» — contains only album docs with .unpublished fields

Now we need to publish .unpublished fields in that collection when client subscribes for album_edit.

This line saves the day:

Also note to call this.ready() if you only do _publishCursor inside publication without returning cursor:


Here is the all our client-only collections and complete publication for album edit for the whole picture:


After that there was a question — «Should we refactor collection updates methods on .unpublsihed fields to be performed against client-only collection like CC_Albums_unpublishedFields.update()?»

Fortunately it’s unnecessary.

Updates on .unpublished fields happens against Albums collection as usual despite the fact that Albums client side collection didn’t receive them.

We update:

This .unpublished fields won’t appear in client Albums collection since we didn’t publish them into that collection.

But fortunately publication into client-only collection

remains reactive

and after updating against Albums collection we reactively get .unpublished field changes in CC_Albums_unpublishedFields on the client.

We made refactor only when we need to make some operations with .unpublished fields data, like Publishing album:


And both collections lived happily ever after

The End.

Tools to track data that comes to the client

These tools were very helpful during all «Reusing subscriptions» quest.

ddp-analyzer — thanks Arunoda

MongoInspector — thanks Max Savin

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.