Tracker 2.0: The Evolution of Tracker & Solutions to its Core Problems

This started out as a forum post. But because of “popular demand,” I decided to move all my Meteor forum posts above a certain length to Medium articles.

THE GOAL: to achieve both the predictability that functional programming is known for and the readability that object oriented approaches are better at.

So to start, I’d like to call out — just kidding — fellow Meteor forum buddy “Adam Brodzinski”. The following is a quote from him that I would say accurately reflects a lot of people’s frustrations with Tracker (which, on a side note, most have inaccurately attributed towards Blaze):

Adam Brodzinski:

Using Tracker in React has been the root of all of my React performance problems and much prefer the events used in Redux. It takes a choppy laggy UI with Tracker and makes it silky smooth using connect in Redux.

Here’s perhaps an even more common sentiment by Chet Corcos:

Tracker is amazing and really cool, but it inherently causes side-effects which is why Meteor feels so magical. But you can easily find yourself getting entangled with Tracker.autorun. I prefer the explicitness of pure functions.

My response is that both sentiments are leading to the unfair and premature “benching” of Tracker, given that Tracker can easily be improved to fix these issues, and given that you can manually make those fixes today. I will be presenting ideas for a “Tracker 2.0” below.

So, these 2 assessments have become so common thanks to the insurgence of Redux and functional programming. But I think we’ve all been too easily impressed by the might of the functional approach. Not that I’m against functional programming (I love it), but it doesn’t mean that we can’t patch the problems of implicit tools we’ll likely be using for a while to come. It doesn’t mean we can’t learn from the functional approach and translate its best aspects to the object oriented imperative version — even if in the not too distant future our entire stateful side-effects-rampant imperative OOP world is replaced by pure functions operating over full stack streams (a concept I was turned on to by Dan Abramov who in turn was turned on to via: http://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/ — he “Thanks” this link on the Redux homepage: http://redux.js.org/index.html) .

…Anyway, functional purists basically say the never-ending battle of patching side-effects isn’t worth it, but until the readability of today’s OOP solutions is fully replaced by the more predictable stream-based functional approaches, I think it’s important we make our OOP solutions as good as possible.

Teleport to present time in the Meteor world: Redux and React is now keeping us on our toes. Where we previously had an unwillingness or lack of creativity to come up with solutions, we are now forced to take Tracker to the next level. Everything that’s going on now in our ecosystem is essentially a necessary evil.

First off, Tracker’s implicitness is beautiful and indeed makes for a great “sideways data loading” pattern, as Mobservable goes on to describe in depth for React: https://mweststrate.github.io/mobservable/index.html . If we could achieve the same level of predictability as Redux, we’d have a far better option than Redux. If you haven’t checked out my TrackerReact package for Meteor + React, it achieves much of the same thing: https://github.com/ultimatejs/tracker-react

Let’s get back on track…What’s preventing that predictability with Tracker I’ve discovered to be precisely this: its autoruns are triggered more than it should be. The reasons:

1. developers don’t use the `fields` option to their `find()` calls.
2. developers combine multiple, but unrelated, data sources in a single autorun
3. autoruns re-run when changed dependencies have no effect on the main output
4. autoruns re-run when data dependent on doesn’t even change!!!

If an autorun runs even just one more time than it should, we’ve broken the purity necessary to properly compose reactive dependencies. This is akin to the purity no side-effects stuff in functional programming.

It’s a lesser problem whether you are using `Session` or scoped `ReactiveDicts`, etc. That’s easy to solve by any consistent file/placement organization pattern you choose. Using single state tree such as @luisherranz ‘s ReactiveState and particularly its `modify` method will go a long a way in the organizational aspects. The Meteor Guide likely prescribes lots of solutions for organizing your code and globals, e.g. don’t use `Session`, but that really isn’t even the problem. To really solve the problem requires fundamentally changing how Tracker operates so it re-runs less.

TRACKER 2.0: SOME IDEAS TO MAKE TRACKER RUN LESS:

• “property level cursors” to automatically populate our `fields` options. So that means the objects in the array returned from `find()` have “cursor-like” behavior when you access its properties. For example, when you access, say, `post.title`, `title` is automatically added behind the scenes to `fields` like this: `Posts.find({}, {fields: {title: 1}})`. What I’m thinking is that in js code you have to do this: `post.title.get()` but in Spacebars `post.title` suffices and Spacebars automatically transpiles it to its correct form under the hood. In short, we need a way to statically tell the transpiler what fields we actually want. Nobody is spending the time specifying exactly the fields they need — in fact there is 1-to-1 parity between doing that additional work and the boilerplate of all the props you must pass down in React; and we all know that too is work that the underlying abstraction could figure out on its own for us just like Blaze always has. We gotta figure out how to achieve the implicitness without all the caveats.
• options to box in what a given autorun really cares about. For example, say you have a global autorun, and it is dependent on a URL parameter, but this particular autorun should only run on a certain route. You should be able to pass a config of options to `Tracker.autorun(func, {route: ‘/foo’})`. As you can imagine, we could dream up all sorts of rules to box in when our autorun is even considered to be re-run. Here’s another example:

Tracker.autorun(func, {
    subscriptionsReady: [‘posts’, ‘comments’],
    isTrue: () => Session.get(‘foo’)
    userIs: ‘admin’
    templateIsVisible: ‘AllPosts’
});

To me that fits into the category of “so obvious and right under our nose — I can’t believe we haven’t done that!”

The fact of the matter is you do a lot of that via `if/else` logic in your autorun, but without standard “questions” the keys in the of options map provide, developers aren’t guided into the “pit of success” of explicit autorun behavior.

How to address that in Helpers? Well here’s one solution (i don’t particularly like its readability, though functional kool-aid drinkers will point out Facebook is doing these exact things with “higher order fill_in_the_blanks”):

Template.AllPosts.helpers({
    foo: Tracker.run(function(arg1, argEtc) {
    //return whatever
    }, options);
});

• re-runs should occur only when dependent outputs have changed — that must be fixed. Currently, just by calling `dep.changed()` any dependent autoruns will re-run, even if the data hasn’t changed. That right there will kill the whole predictability of everything — any library can cause your autoruns to re-run just because it calls `dep.changed`, not because the data you’re actually dependent on has actually changed! The dispatcher needs a record of the old state dependent on in order to compare to determine if the dependency actually changed. I’m not fully sure if Mobservable is that intelligent, but I’ll check. In addition, what’s also happening here is people are writing an autorun dependent, say, on URL params, the hash, and query params, when all they really need is, for example URL params. This problem is similar to not specifying `fields` options. In short, an API was provided by a library to access all this URL data reactively, for example, without giving you fine-grained access to just the key you actually used. However, your autorun function ends up re-running in response to changes to all keys of all objects examined in the `params()` method. I believe this is something `FlowRouter` got more correct than `IronRouter`. Anyway, this sorta stuff characterizes the problem as a whole — we need finer grained reactivity!

If we can achieve finer-grained autorun-style reactivity, then we have a foundation to do a lot of the things that Redux does. We can build hot module replacement and time traveling. Time traveling simply wouldn’t work now — not cuz we are using mutable state — but because every action replayed has too many opportunities for side-effects if functions are being re-run more than once. Here’s what we would have to do:

1. snapshot Mongo state at every commit and keep a record of every call to `insert`, `update`, etc, for replay (likely simply using Oplog itself). Same with any calls to `set` on `ReactiveDict` or `ReactiveState`. That’s right, we don’t need “immutable state”, we just do a damn good job at making sure our state by reference is up to date. It will require a lot more memory to keep these snapshots of all of Mongo, but keep in mind this is only for development. The functional approach used by Redux only has to keep a log of the latest state mutations — we achieve the same by snapshotting all of Mini mongo.
2. once we can reproduce state, components and other UI code can be replaced, and “rehydrated” with our state just as in Redux.

We can also go a step beyond Redux and really truly take a hold of our autoruns — here’s how:

• a Chrome devtools extension that provides a UI to you navigate a graph of reactive dependencies throughout your app (similar to the Elements tab in Inspector, and the component tree in the new React devtools, or very close to the following for RxJS: http://jaredforsyth.com/rxvision). What if we could see everywhere that `new Tracker.Dependency().depend()` was call called, and we could time travel through a log of every time `new Tracker.Dependency().changed()` was called. So that’s a visualization of statically what’s in memory in terms of dependencies, and then a sequential visualization (akin to a stack trace) of one various branches trigger each other to re-run. React needs this too — so you can see which branches are forced to update. Redux could benefit from this visualization as well. I don’t buy that in Redux just because you manually wrote all the connections between actions, stores and components that it doesn’t have a whole new set of problems all its own — that’s a lot of code to navigate back and forth between!

..THE PART I RANT A LITTLE BEFORE I FINISH:

But either way, Tracker is the one that suffers from these connections being traversed more than it needs to, when that isn’t the case at all in Redux — so that means we stand to benefit from a visualization way more than Redux. I think just visualizing it would pin-point more such redundant re-run patterns than I’ve mentioned today. That’s why i think the visualization perhaps is the most important thing that must happen first, before we embark on some grand re-write of Tracker, i.e. Tracker 2.0.

The way we achieve the visualization is via Meteor package that overrides `ReactiveDict`, `dep.changed/depend` and core functions in Minimongo (`insert`, `update`, etc). We need a record of every time they are called and the trace of where they are called. We can easily produce the trace within functions via things like `arguments.callee.caller`. As a devtools extension, we could link from a visualization of dependent reactive functions to their actual function in the sources tab. We can produce counts of when they are re-run; we can highlight which functions are re-run when they are re-run just like the React devtools. ..And of course, we would eventually use a similar log of collection methods called as the basis for Time Traveling.

I have a funny feeling that once we have an “autorun visualization tree”, and once we use it to build implicit and explicit tools (as described above) to box in Tracker, then we aren’t going to be having this discussion anymore about how unpredictable Tracker is. We can make it predictable.

So that’s all for today, but there’s a final related concept I’d like you to think about which I too am exploring and will likely discuss in a future article:

Redux is for “UI STATE” and Relay is for “DOMAIN STATE”. In Meteor, Minimongo is also for “domain state” but we have no good “ui state” pattern. Conversely, people using Redux for “domain state” are doing it wrong and will likely change to Relay as it evolves (that’s my prediction). Minimongo — unpredictable side-effects and all — still makes more sense for “domain state” than Redux does. Don’t be fooled. ..To be continued…