What “Reactive GraphQL” means for Meteor…
This is a response to the following announcement by Meteor:
You will need to read it before the following.
Overall my perspective is that this is excellent news, though for anyone following GraphQL, the obvious path to take. In short, the remaining questions are about the implementation:
- the implementation of subscriptions on the server
- but also the implementation of the interface for the view layer, specifically Relay or our own solution
Let’s start with the second bullet.
My assumptions: the idea will be that it must be able to interface with Relay (why forego Relay?), and I since the stated goal is to build each layer so it can be “immediately” used, the view layer interface will remain an unrestrained opportunity to close in on as we rise up the stack.
My take is: since where developers will likely interface with this system the most is the view layer, its interface into the view layer should be deeply thought through from the start, even if it initially appears it won’t have much bearing on how the server side subscription stuff is built (it might). Is it just Relay or a simpler unrelated system or a custom implementation that allows you to “dropdown” to Relay while also providing a simpler interface on top of it (similar to how they are addressing GraphQL)? I’d like to see us make something a lot better than Relay. Without this answered, us application developers really will have no clue about what we are getting except that we will specify GraphQL queries/mutations somewhere???? Today’s announcement leaves application developers with a lot of new questions.
Ways to improve on Relay:
- Multiple datasources. We should have a client-only store that we can perform queries against in the exact way we can server-side queries as you can do in Om Next. While Om Next lets you specify multiple data-sources, perhaps the more straightforward implementation for us is: a server side data source + a client side data source, not an abstraction that allows for infinite datasources. The goal being that such a system would also replace the need for Redux.
- Router State. I think it should also replace the need for a Router since a Router is just a custom way to specify a sub-set of state (there’s several Redux Router implementations out right now by the way). In essence, there is a lot of evolution going on in terms of client side state right now, aimed at finding the best all-encompassing idiomatic way to maintain it — the new GraphQL subscriptions system has to hit the spot on the client and be able to compete if not be better than all these solutions. We should aim to unify all forms of state under a consistent interface. My prediction is this entire year will be geared towards the open source client side javascript community solving this — providing that ideal client side interface between state and your components. Both Redux and Relay have set the stage, but there is a long way to go before we get it right. I do think there is a point where will get it right (i.e. it won’t go on forever). It will be absolutely crucial that Meteor is part of that story and doesn’t write their own — possibly closed/siloed — story.
As for the first point that the implementation on the server is biggest question, only time will tell. But from what I can tell the overall plan is to use GraphQL types to determine which which concrete query types (i.e. taking into consideration their parameters) need to be invalidated based on mutations received. And do so purely on the webserver, so as not to be dependent on database capabilities. That is the exact general plan I’ve been suggesting, and I’m glad Sashko Stubailo n co have proven this to be a viable path.
That all said, there are still tons and tons of questions about exactly how fine-grained you can get using solely GraphQL types. Meteor’s experimental PostgreSQL solution for example refreshed all subscriptions any time the corresponding table changed! It needs to be way more fine-grained than that. What about joins in SQL databases — the problem likely gets exponentially complex there. It would seem their idea is to provide the best reactive support they can generally provide, while also providing a way to fully customize it. So does that mean the plan is to provide refreshes on any changes to a table + customization if we feel like doing it ourselves? Basically, I hope we can do better than refreshing subscriptions any time a table changes. Only exact work on the implementation will determine that. But, if we get the low level customization API on point, it will also likely mean lots of community-contributed packages for common reactive use-cases. I think overall we have a bright future here.
In addition, it seems if we are to do this right, it will have to be something that in production mode fires up a subscription-focused server (or ideally a fault-tolerant cluster of them) dedicated just to performing this task of monitoring/broadcasting the matrix of queries and mutations. The reason this is necessary is so that each webserver in your main webserver cluster doesn’t perform this as duplicate ever-amounting work (which is the problem with the current oplog implementation). That will be the first time Meteor supports different types of webservers and communicating between them — probably via something like Redis — out the box. Thus far, this sorta stuff seems to have been a level of complexity and sysops work that Meteor has not wanted to undertake. That means this is a lot of work. I think the upside is that it will make for a great product for Galaxy to sell. I think we’d be lucky to see that on Galaxy before the end of the year that just started — it will likely take longer. That means this will be one big experiment for a relatively long time. But building big apps usually take at least 6 months, so if you start in 6 months, perhaps by the time you’re done MDG will have an appliance to launch your production app on. Overall, I think this is very major stuff, and it’s better late than never. Perhaps with MDG’s new devotion toward embracing the community we can build this faster than ever.
My final thought about implementation is: it would seem GraphQL and its type system somehow greatly aids this effort. That’s the hope at least. Only time will tell if it really makes it that much easier, but hopefully it does. And actually I’ll finish this topic with a question:
Notwithstanding it will open the doors to many DBs, can this compete with the performance gains provided by directly using database features such as Oplog tailing or database triggers? What role will they play if any? Do we simply intercept all mutations before they hit the DB as our own sort of log tailing trigger? Maybe it has a chance of being more performant.
In conclusion, I’m excited that the development of this will be more open — i hope it starts out as its own Github repo. I think getting developer participation could perhaps be more important than the actual product itself. I have an idea to do that:
throughout development, make “Reactive GraphQL” a package that you can easily add to Meteor 1.2, 1.3 etc.
That would be big for getting help testing it and contribution along the way. That’s as opposed to requiring a separate github branch. Being that it’s totally separate from the current pubsub LiveQuery approach, this in fact may be the first time something like this is a possibility for MDG. It’s a seemingly small wall to specify a Meteor version or branch, but the friction of creating a new app based on a different Meteor version or branch is forgone too often than not. More than likely the majority of meteor users don’t even know how to do that. But they do know how to add a package. I think it could be big.
I think we really need to pioneer the new path for MDG’s open source development efforts. To do that, we need to showcase at every step of the way how open they are to contribution. I think a big part of that is simply developing in the open, and making it as absolutely easy as possible to test out what they’re working on, and from a very early stage (i.e. like next week). All contribution will stem from simply being able to see it in action — visibility is why the client side view layer gets so much open source activity and growth. In addition, if there’s ever a chance to live-code some of the client side interface, that should be offered immediately. In general what I’m saying is that, for example, Meteor 1.3 beta has been tested by a fraction of how many people it could be tested by vs if testing it was was a case of doing:
meteor add reactive-graphql
Long Live Meteor! I like the name “Reactive GraphQL.” If we do this right — and by “we” I mean “we” not just MDG — we will continue to be able to compete on the greater NPM stage. This should also be able to live outside of Meteor. That’s provocative enough of a concept that I’ll just end with that:
