Some thoughts about the next version of sUTL

I’ve been looking at bits and pieces of information on Operational Transformation. It was the big innovation with Google Wave, and there are all sorts of fun little libraries and bits and pieces around that play with it.

Operational Transformation is the idea that if you can describe edits to a complex data structure in just the right way, you can merge edits from multiple people in a safe way, without locking or any real coordination.

Say that when you edit a complex document (think a big text document, although it can be more complex than that), you describe it as a list of small transforms… jump forward this far, insert this text, jump a bit further, delete that text, and so on.

If you do this just so, then you can take edits from two sources, say A and B, and apply them to the document in either order (A then B or B then A) and get the same result, even though the person that made edit A didn’t know about B and vice versa. So you can then send B to the person who did A, and A to the person that did B, they can update their view of the world to A+B, and move on.

It does this through coordination by a server, taking the edits A and B and producing special transforms of them to make all this work. Much more here: http://www.codecommit.com/blog/java/understanding-and-applying-operational-transformation

You can do this on text, but you can also do it on other structures. Unsurprisingly if you’ve read any of my previous posts, I’m interested on doing this with JSON data (ie: MAS).

There are already really interesting efforts in this space. One I’m digesting at the moment is ShareJS. Here’s their set of canonical transforms: https://github.com/ottypes/json0

Firstly, they’ve done something much better in ShareJS for referencing into a MAS document than JSONPath (which is what sUTL uses). They simply do this:

{‘p’: [‘key’, 2, ‘anotherkey’, …] }

ie: a path is a list of strings and integers, where strings are dictionary keys and integers are list indexes.

JSONPath is a lot more complex than this, but I've found that sUTL doesn’t need this complexity. One extra feature of JSONPath is that you can use a double dot “..key” to mean “recursively look for key down through the tree”, but that is unusual, and could be replaced with a transform just for that job.

Secondly, and much more interestingly, sUTL could incorporate an operational transform library. Given a canonical set of (operational) transforms like in the ShareJS document above, sUTL could provide basic (sUTL) transforms to coordinate multiple editors and keep a central document up to date.

This could obviously be used for multiple people editing a JSON document, ok. But there are more interesting uses.

I’ve been thinking about user interfaces, and how damned annoying they are to build. In the current world, there are a myriad of great ways to build client side javascript based UIs for the web, and they all require elaborate tooling and the kind of attention only a dedicated front end developer can give. Bleh. And then you get to mobile apps and etc, all done with completely different paradigms, all again incredibly convoluted.

But what is a user app really at heart?

It comprises:

  • something like a DOM; ie: a recursive structure of UI components
  • some styling
  • a bunch of state
  • some code
  • an ability for code to talk to external things (eg: do web requests)
  • an ability to code to respond to events (eg: button clicks, web responses)

I’m thinking that a DOM could be specified in MAS, styles could be specified in MAS, state is just data and certainly could be specified in MAS, and with sUTL code can be specified in MAS.

So we come to a user facing application being specified as a sUTL transform, like so:

{
"manifest": { ... top level app config ... },
"ui": { ... },
"style": { ... },
"state": { ... }
}

where you can put sUTL in anywhere (because technically the whole thing is sUTL).

I’d need to define a whole basic UI library. For example:

{
"type": "button",
"text": "Click Me",
"onclick": <... some sUTL ...>
}

What would “onclick” look like? Well, the sUTL transform would need access to anything potentially, so the source can be the entire document. What would the output be?

Maybe a list of transforms on the entire source document, as in ShareJS?

Say the button click wants to add a label to the UI, the result of the onclick sUTL might be

[
{
"p":["ui", ..., <idx>],
"li": {
"type": "label",
"text": "Hello World"
}
}
]

So, the entire UI becomes this one huge self transforming JSON/MAS structure. Pretty cool!