What our Clojure Web Framework looked like

I listened to What a Clojure Web Framework might look like by Eric Normand, and what he said reminded me about how it was to develop Sulo LIVE. In this article, I’ve tried to describe how the framework we created around om.next could be extended in terms of plug-ins like Eric mentioned. I recommend listening to Eric’s piece first, but if you don’t, you’ll hopefully get some insight in to what it’s like to build a system with om.next.

Motivation

Eric Normand talks about a web framework with:

“plug-in as the unit of sharing, as the unit of composition.”

Eric also mentioned that the big thing about Ruby on Rails is that:

“If you just think of it as a CRUD app and then you can figure out how to break stuff up in to MVC, you’re 90 percent on the way there.”

In Rails, as long as you can break your program in to MVC, it’ll take care of you.

The quotes I listed made me think about what it was like to be a developer in our framework. As long as one could break stuff up in “Views with Queries”, and define their “Reads and Mutates” the rest is “taken care of”. Also, if one wanted to extend the system, like adding a payment service or adding user authentication to the system, there were known hooks to do this in.

We didn’t get to plug-in as unit of sharing and compostion, but I think it’s a cool idea that we should explore further.

The way features were implemented in this framework was very similar for both the client and the server. I’ll begin by explaining how we accidentally got there.

Symmetry

The mental model ended up being the same for both the server and the client mostly because om.next worked that way, but also beause we valued symmetry when making design decisions. We wanted things to be similar across the system because we thought that would make us faster by not having to learn new ideas at every point in our system. Symmetry heavily guided our choices and we started out with:

Clojure(Script) — Everywhere

I think using the same programming language for client (Web, Mobile, Native) and server development is a good idea. Whether it’s JavaScript, Clojure(Script) or something else. Our theory was that wherever in the system we write code, we’d slightly improve at every other part of the system. Here are some other benefits, some specific to Clojure(Script):

  • Being able to use the same (or similar) tools for the same problems on both client and server.
  • Whenever client or server code was changed it was reloaded in the environment it was running, making for non stop interactive development.
  • Immutable data structures by default is a good idea and we get to have it everywhere. Including the client and server databases.
  • Being able to create a version of the client that could run in the server environment, creating opportunities for shared tooling and new sorts of testing.

There’s more to be said about this but let’s move on.

Datomic & DataScript — Common database API for client and server

We were hoping that using similar API’s for all our data access and modification would help us move faster and possibly share a lot of code. I’m honestly not sure how good this decision is. Some of the pros:

  • It was very nice to be able to have query everywhere
  • We could use DataScript instead of Datomic when testing sometimes, just because DataScript requires less setup.
  • Most of the things we learned about query (datalog) applied to both Datomic and DataScript.
  • It made for a flexible format to be sent between server and client, as DataScript can merge database transactions instead of a tree of data.

Some cons:

  • Client cares about view data + data needed to perform query.
  • Does not work with om.next out of the box.

om.next — Client↔Server relationships

Even though it was hard for us to get om.next to work with DataScript, we kept trying until we got it working. The reward was that om.next’s architecture made as much sense on the client as on the server.

If there’s only one take away you get from this article, let it be this:

With om.next, the frontend is broken apart to a client↔server relationship between views and data access. The views (a client) creates a query to be served by the om.next parser on the frontend (a server). The parts of the query the frontend parser (a.k.a client parser) can’t answer, it will ask the server parser on the backend.

If this doesn’t make sense yet, hopefully it will when you’ve finished reading.

Let me try to explain how om.next works on a high level. If you already know about om.next, you can skip to the next section.

You define a view together with a query describing all the data the view needs to render. This query will be passed through an om.next parser, which will return the data described by the query. Take the data returned by the parser, pass it to the view, and it’ll render correctly. In pseudo code:

(render View (parser (get-query View)))

The queries compose, such that if we have views A and B and A renders B, then A’s query will contain B’s query.

These queries can be sent remotely, where a server parser will respond with data that can be merged into the client’s app state.

(go (merge (deref app-state) (<! (send! (parser (get-query View) :remote)))))

A parser has two functions it calls when parsing a query, read and mutate. These functions will be called when the parser comes across certain items within the query. Reads are defined as a keyword and mutates are defined as a symbol. Reads will return data from the database, mutates will modify the database.

Read query example: You want to create a view for a contact list. It has components rendering a contact, with their first name. An om.next query for this could look like this:

[{:read/contact-list [:person/first-name]}]

Where the keyword :read/contact-list is defined as one of the reads in the parser and this query requires [:person/first-name] to be present for every entry in the contact list.

Mutate query example: You want to add a contact to your contact list. The om.next query could look like this:

[(mutate/add-contact {:name “Alice”})]

Where the symbol ‘mutate/add-contact is defined as one of your mutates in the parser. The namespaces for these example reads and mutates are not important.

A query can contain any number of reads and/or mutates.

You can almost think of the reads and mutates as API endpoints.

That’s all I’m going to say about om.next and parsing for now. I hope this is enough to help you get through the article.

stuartsierra/component & om.next/shared — Components

For calling out to our external services or to other stateful components of your system, we used stuartsierra’s component library. For us, it made it easy to reload our code as we could restart the system every time there was a code change. To be able to keep our state between reloads, we used also opted in to using weavejester’s suspendable library, to be able to supsend and resume components when needed. Like the web server and database component.

The other big thing we got out of using Component was that it made it easy for us to switch out different implementations of components. We mostly used different implementations for testing, development and for production. It was nice to be able to switch these implementations at runtime.

It’s not only on the server where you have stateful components that need to be stopped, paused and resumed, or components that need to be faked during development time. This is true for the client as well. For example:

  • We’d have a component for providing photo urls.
  • A Firebase component.
  • A chat component which used a WebSocket connection to our server.

We organized these client components just like the server components, such that they could be controlled when we reload the code and so that the implementation could easily switched out. The client components were instantiated when they were first used, which made it possible for cljs modules to place them in the smallest possible module.

Routes

We used bidi. It worked👌for us.

Valuing symmetry

In the end, the system is not the same for client and server programming. The edges are different and they should be pluggable. When things that can be modelled the same, such as reading and writing to a database, having components with lifecycles and routes, it’s nice when they are the same (or at least are used in the same way).

Extending our system

Using the framework one could easily add a new view, a new route or a new read or write to an existing source. But what about whenever one needed to do something with our system that one couldn’t already do, such as add Auth or add a new Source or new functionality to our read/mutate system?

There are a few known hooks in our system. Most of these exist both on the server and the client. They are:

  • :enter-request, :leave-request
  • :enter-parser, :leave-parser
  • :enter-read, :leave-read
  • :enter-mutate, :leave-mutate
  • :add-component
  • :add-route

With these hooks I can imagine implementing most features for our web app. Plug-ins to the framework could be written in terms of these hooks. The plug-ins might then also want to expose new hooks that other plugins could use, so this hook infrastructure would have to be dynamic.

Now that we know the hooks, let’s see how one would add Auth to our system.

Adding authentication

When implementing auth we wanted to check that the request had access to the route and we made a decision to be able to declare different auth roles for our routes and each read and write. We hooked in logic to:

  • :enter-request, for re-directing if the request didn’t have access to the route.
  • :enter-read, for getting the auth role required for the read and checking for access.
  • :enter-mutate, same thing as for the reads.
  • :enter-parser, to place some common data used for both :enter-read and :enter-mutate.

This could have been declared as a plugin to the system, defined as a map or as a protocol. Something like:
{:server
 {:enter-request (fn [request])
 :enter-read (fn [env key params])
 :enter-mutate (fn [enter key params])
 :enter-parser (fn [env query])}
 :client {}}

Looking back on our code, I think we’d been happier had we implemented functionallity interms of plug-ins like this.

Another example for using these plug-ins (one that Eric mentioned) was that we found ourselves using a common pattern where we wanted to verify something against the user entry in the database or update data for the user who made the request. So we created another extension for :enter-parser, where we got the user from our database and placed it in the parser’s environment such that the read and mutate could have access to them.

Order

Another thing that comes to mind is that these plug-ins could have dependencies between other plug-ins. Let’s say plug-in A needs to run its :enter-parser function before plug-in B. Luckily we’ve got stuartsierra’s dependency library (and weavejester’s fork) to making sure plug-in’s hooks would get run in a correct order.

A slightly different programming model

From the server’s point of view, there’s not much difference between using and om.next parser to dispatch to reads and mutates, compared to ring web development where the routes dispatch to endpoints. A single read (the implementation of :read/contact-list) is very similar to that of a GET endpoint and a mutate is similar to a POST, PUT or DELETE endoint. Interceptors or middlewares are used to extend the functionallity, so what’s difference?

By using the concept of a query to describe data resources and by structuring our frontend as a client↔server relationship, we can program the same way on the frontend as we’re used to on the backend. These queries can contain multiple reads and mutations, so HTTP requests are sort of batched up together.

As a client programmer, you stop thinking in terms of requests, you just assume they’ll happen at some point. Eventually, we found ourselves not thinking in terms of async calls to the server. We just described the different states our views could be in and the code for sending requests and handling responses, was for the most part untouched.

When you give up control of being able to send and act on each different request, you also lose some of the things that used to be easy. Since requests to read or mutate are no longer one to one, a single response may contain multiple error messages. Being able to render the status of a mutation from a view that created it, was no longer easy, but it was doable and it could probably be implemented as a full stack plug-in.

When the client has the same mental model for programming as the server, there’s nothing stopping a frontend developer of such a system to start programming on the server. Creating reads, mutates on the front and backend, add new data sources or other stateful components on front and backend, adding routes, all in the same language, in a real-time environment, ever changing as the programmer saves her code.

Different flavors of om.next

Just want to mention that om.next libraries can be very different. Every part of om.next is configurable and depending on whether you use the defaults, create your own (as we did) or use the very impressive framework Fulcro’s configuration, you’ll have different developer experiences.

Summary

I wrote this article because I have some experience with om.next I think it has potential of being a good foundation for creating Eric’s notion of a “ Boring Framework”.

Breaking apart the frontend in to a local client↔server in the way om.next does by having a client and server parser, is a very interesting concept to me. Possibly because it is a useful symmetry.

I hope you learned something today. Thanks Eric for the inspiration!

Have a nice day 👋

PS.

There are some problems with using DataScript on the client together with om.next which I’ve written about in a work in progress om.next DataScript parser library. If you’re interested in this problem, please let me know.