Action Logs for Firebase

When building a system, you should have as many abstraction layers as you need, and no more. I love Firebase. But it feels like something is missing. I think that missing thing may be an action log.

But first, some background…

With common frameworks like, say, Rails, there’s an application layer that sits between the client and the data storage layer. This is where all of the policies and behaviors should be encoded. For example, whether an admin user can delete the content by a non-admin user, and what to do if the opposite is attempted. It’s also where you’d invoke side-effects for actions, like indexing text and sending mobile notifications when content is created. The result is an API. The clear boundary enables client and server to develop, be tested, and evolve independently.

Firebase merges the storage layer and application layers. While for some applications this is a revelation — there’s no server code needed to build a realtime chat app — this, in fact, is too few layers. There is a declarative rule system, which is clean way to do some of the things you’d want to do in your application layer like validation and access control. However, there are also obvious things like sending notifications to mobile devices and full text indexing that you cannot do. This is understandable… but there are also non-obvious things that cannot be solved client-side like counters and secondary copies of denormalized data.

In general, what’s needed is a way to handle side-effects server-side.

Counters

The Firebase team brilliantly advocates temporally-sorted collections instead of arrays. This is perfect for something like a chat room: clients can add messages atomically, and the server does not need to hold any locks. All clients could be talking to one server, or each client could be round-robining to 1000 servers. It doesn’t matter. Awesome. The thing that’s missing, however, is a way to keep count of the number of messages.

If you want to keep track of how many messages are in that chat room, you’re suggested to implement a counter client-side. This doesn’t work, however. The transaction boundary is just on the counter — it does not include the creation/deletion of items — so they can (and do) get out-of-sync. Also, rogue clients could lie (you can work-around this with access rules). Worse, in my opinion, is the implications for the overall architecture. Say you want to add a second counter — you cannot add such a feature without deploying an updated app to all of your users. And good luck keeping that counter accurate while the update is rolling out.

Denormalized Data

Another brilliant firebase idiosyncrasy is data denormalization. Much like temporally-sorted collections, this is a design choice that promotes scalability. I love it. But, it’s at odds with the lack of an application layer. Again, the client should not be responsible for creating secondary copies of data. In addition to the obvious scalability and data integrity problems with implementing something like twitter this way, it also means you cannot add new features without updating all of your clients. Not good, particularly for mobile.

The Action Log

A possible solution to all of these problems — from notifications, to text indexing, counters, secondary indices, etc— is to have a server that uses the REST streaming API to implement side-effects for client actions. This allows the client to be simpler, as it does not need to understand the consequences of its actions. You can add features, rebuild counters and indices, etc, without touching client code.

I first set about building a streaming server the wrong way — by listening to the whole datastore. Based on this great suggestion, I added an action log instead. Initially I didn’t like this idea because it means the client has to write everything twice. Worse, it really needs to do this transactionally and it cannot. However, I think it’s the best you can do with the current firebase feature set.

Action log entries contains just these few properties:

• The path to the resource being modified
• A newValue field, that’s populated when data is added or modified
• An oldValue field, that’s populated when data is modified or deleted
• Timestamp and user id

Every time the client does something, it writes an entry into this log that says what it did. A server component listens to this log, and performs whatever actions are needed, and then deletes the entry when it’s done. For example, if you were writing a twitter clone, the client would append a tweet to the user’s own timeline, and the server would fetch their followers and update each of their timelines. The author would still see his tweet in real-time (and even offline). Also, it’s ok that it might take a few more seconds for the change to propagate to other users.

More importantly, the client does not need to know the implementation of this feature, so you can change it. Imagine that the twitter clone started to scale, and a small number of popular users gathered large numbers of followers. You could handle these users differently — instead of making thousands of copies of each of their tweets, you could have their followers fetch the streams of popular people in parallel with their home stream, and merge the results.

In practice, I’ve found that this log works quite well. I’ve built some infrastructure in rails to map the resource path to a controller, so you can create a FooHandler, and any resource with a path that begins with /foo/ invokes a handle(ref, old_value, new_value) method on that handler. I’ve also added a declarative way for managing secondary copies of data, so you can state that any resource like /foo/$user_key/$item_key should be replicated as /derived/foo-by-item/$item_key/$user_key. This declarative approach makes it easy to add new derived data, because it can operate in batch over existing data or on-the-fly as actions are processed. It also means we can set up an access rule that makes the /derived/ prefix read-only for clients, so no-one gets confused about which direction the data is moving.

The action log is also very convenient when there is a problem: you can easily inspect its contents and see what the server is unable to process. When you fix it, the server has all of the unfinished work waiting for it in the queue, and you can watch it clear it out. Since these side-effects are asynchronous, the client impact is minimized (e.g., a late notification).

The one real shortcoming with my approach is that the log itself is not written transactionally. This isn’t as bad as it could be — one can write batch jobs to ensure consistency, which is not the worst thing in the world. Nevertheless, it would be awesome if the Firebase team added support for action logs so clients could be even simpler. I have some ideas about details, maybe I’ll write them up here if anyone’s interested (I’m reachable on twitter at @spf2).