Oh god, I built another JavaScript framework
How and why I was driven into the warm embrace of my very own web stack
How did this happen? What have I done?
I wrestle with these questions on a daily basis, and to be honest I can see where murderers are coming from when they say things just got out of hand. It started as an experiment, almost a dare — I had no idea it would go this far.
I started working on this thing almost a year ago, if you can even believe it. It was back then that I got to thinking about all the things that made me crazy about web development, and I set out to fix everything in one fell swoop.
At first it was like, “Great! Maybe one day this crazy thing would be something cool.” And then I had some amount of free time, in between projects and such, and I just kept making it more real.
Fast forward a year, on and off, and my crazy thing is, well, pretty awesome.
I’m using it every day, I’m building every new project with it, and I keep making it more awesome. Usually bad ideas get worse when they become real, but this became real and got better — so that means maybe it wasn’t such a bad idea after all, right?
Allow me to tell you about it in more detail, and maybe you’ll jump off the deep end with me.
How it all started
Back in mid-2014, I was working on the first version of Cluster’s web app.
The logged-in web app was built with RequireJS, Mustache, jQuery and a bunch of other stuff on top of Django. It was a Single Page App, and I had some elaborate RequireJS configuration that loaded up all the client-side JavaScript modules.
This app did a ton of things that web apps should do:
- I served optimized, versioned JavaScript and CSS bundles in production from Cloudfront over S3;
- I rendered all sorts of real metadata in the page <head> for Facebook and Twitter and Google;
- I prerendered some above-the-fold content on the server, and the rest in the client, so users didn’t see a totally empty page on a cold pageload;
- I passed data loaded on the server into the client as JSON, so subsequent client-side rendering could take place immediately;
- and even more I can’t remember.
I was trying to do everything right, and I did get it all working just so.
With a lot of small hacks and a nontrivial amount of configuration, this Rube Goldberg machine was working.
But all was not right with the world. A disquiet was building.
Over time, I noticed something funny happening:
Eventually, this webpage got so complicated and unwieldy that
I started avoiding making changes to it.
The frontend got so complicated that I was actively avoiding making improvements to the product, because editing it was no fun. Everything was too touchy, and it was too difficult to get the whole thing in my head all at once on any given day.
People started sending me nastygram support emails about the website falling behind the mobile app for new features. It was bad news.
I used to like web development, so what happened?
Best I can tell, the configuration became overwhelming, and the partial server- and client-side rendering became such a huge context switch that everything, taken together, just kind of … sucked.
This is when I got to thinking: What would it take to make frontend web development like this not suck?
Could I make something to make building web frontends simple and fun again?
Once I got the nagging feeling that I might be able to just fix it, I couldn’t help myself. It’s how I’d imagine someone feels when he tries heroin for the first time: I felt bad, but so good.
What “not sucky” looks like
Once I started going down this path, I tried to come up with what the ideal web app bootstrapping framework might actually look like.
I took my codebase and literally started moving files around, trying to come up with a sane way of organizing everything that didn’t feel like a bunch of duct tape and bubble gum.
Here are the guidelines I came up with during this process:
No configuration
I had a client-side configuration hangover, so I wanted no configuration. I don’t want to have to configure where to find modules, or how to serve them in the client, or how to bundle them in production. Production mode should be bundled automatically, and it should work without thinking.
I didn’t even want to configure my URL routes — how about just organizing the code into directories, like a traditional webserver? When do we actually need fully arbitrary regular expression-based routes?
Automatically including associated styles
I also wanted modules that included CSS. I never liked how you’d make a JavaScript module over here that made some cool widget, and when you included the JavaScript for that thing you also needed to remember to include the CSS. That part sucks.
But I also didn’t want to have to specify where to find the CSS every time, so I needed some way associating CSS with a JavaScript module automatically.
Automatic resource compilation
Then there’s this whole thing with taking Handlebars or React’s .jsx files or CoffeeScript or whatever and turning this stuff into raw JavaScript to run in real life. (There’s also LESS, SASS, etc. on the CSS side of things.)
I didn’t want to configure the build system for one of these ever again — RequireJS plugins are incredibly confusing to build, and once you get them working in development mode, it’s a completely different beast in production. This stuff should be easier.
Server-side bootstrapping
Lastly, I didn’t want to have to write the gnarly page to output all the <script> tags and the <link> tags and to wire up all the client-side JavaScript.
I also didn’t want to have to duplicate efforts to load client-side data on the server, validate URLs on the server, and manually serialize data I loaded there back into the client.
So, when it came right down to it, I wanted my client-side framework to also run on the server. I wanted it to generate that initial bootstrapping webpage by itself.
It was isomorphic all along
Turns out solving this stuff would require a bunch of server-side work, so this isn’t just a client-side JavaScript framework anymore.
For example, it needs to know how to bundle and optimize client-side files in production, and separately how to include a bunch of individual, uncacheable JavaScript files in development mode, and to just handle both of these seamlessly.
This thing also needs to deal with including CSS files and bundles in a similar way, and automatically include the ones that correspond to the JavaScript modules we load.
Furthermore, if we want to get rid of the configuration-glue bootstrapping web page, you’ll need to relinquish control over server-side rendering altogether.
As a result, this framework ended up being kind of monolithic, rather than being a pluggable library — without configuration, it needs lots of structure and convention.
I ended up with a client-side framework, and then a server-side runtime for the framework that handles the HTTP request and subsequently gets everything all up and running in the browser.
Hello world
Meet skit — an opinionated client- and server-side JavaScript framework for building websites that act as pure API clients.
This is the crazy thing I ended up building. The shame! I’m just glad I finally get to tell somebody about it.
A skit app’s codebase is a single codebase, sharing the same exact JavaScript modules, templates, etc. from server to client-side. The code gets executed by the skit runtime in Node.js, and then gets re-instantiated in the browser transparently so the page can pick up where it left off once it gets loaded there.
The way skit works is by abstracting the page lifecycle across the page-load barrier. You write a single controller for a page that handles loading the data on the server, rendering the data into HTML, and then hooking up a live DOM in the browser, all in the same file.
The data-loading and rendering steps happen on the server side initially, then the controller is re-instantiated in the client with the same-ish state automatically by the boostrapping web page — but you no longer have to worry about how this works.
It sounds crazy or maybe horrible, but in practice it’s actually quite nice.
Every URL path in skit corresponds to a single skit Controller subclass, and a controller looks like this:
module.exports = Controller.create({
__preload__: function(done) {
// This runs on the server side at first, but can be
// run later in the client just the same.
MyAPIClient.getThing(function(thing) {
if (!thing) {
notFound();
} else {
this.thing = thing;
}
done();
}, this);
}, __body__: function() {
return template(this.thing);
}, __ready__: function() {
// This runs only in the browser once the DOM is ready.
events.bind(dom.get(‘.thing’), ‘click’, function() {
this.doSomething();
this.rerender();
}, this);
}
});
Skit runs the preload and body calls on the server side, then calls ready in the client. You can re-load and re-render the entire page in the browser if you want.
We don’t do anything too magical the server side: I don’t try to fake being a browser, for example. window and document are undefined there, so don’t try accessing them before ready is called.
Also, your template language or DOM-generation tool needs to spit out HTML as a string, and it needs to be able to wire itself back up in the browser. I use fairly old-school event wiring here so it’s pretty straightforward, but the new hotness should also work as long as said hotness supports server-side prerendering.
(React, for example, has this functionality out of the box — so I was able to build a proof of concept React project in skit very easily.)
The magical part of skit, the part that makes all this actually work, is a small set of libraries that work on the server and the client with the same API. These libraries are: net for sending XHRs, cookies for reading and writing cookies, and navigation for navigating to new URLs or generating 404s.
That’s pretty much it.
How’s this different from X?
There are plenty of ways to accomplish this now, of course — things like browserify and webpack can compile your Node.js codebase down to something that can run in the browser as well. But you still need to figure out how to compile all your client-side code and get it into the browser. It’s a lot of work to get the pieces in place, conceptually.
This all strikes me as way too complicated, but people have done it: Here’s an example of an amazing solution using existing tools.
The closest thing I’ve found to a wholesale server- and client-side rendering solution is an Airbnb library called Rendr, which enables running Backbone apps on the server, too. Then you have Brisket by Bloomberg doing something similar. Cool stuff, but the number of steps to implement something like this is overwhelming.
On the other side of the universe are things like Meteor, which want to control the whole stack, all the way down to your data layer. This level of integration is never something I’ve been comfortable with, but at least the bootstrapping web page from hell seems to be less hellish here.
Anyway, I’m not claiming this problem doesn’t have solutions. I just think it’s still way too complicated, and we need better tools for this kind of thing.
In closing
So that’s how it happened.
I didn’t mean to write another JavaScript framework, but once I started down the road, I couldn’t help myself.
I would love for you to give it a try, find out why it’s terrible, and then tell me all about it.
If this sort of thing is interesting to you, we’re growing our web team in San Francisco! We are using skit, Django, go and more to build https://launchkit.io/ — shoot us an email at jobs@getcluster.com. (Looking for FT in SF only for now.)
Thanks for reading! If you enjoyed this article, I would really appreciate you hitting the recommend button below. Connect with me on Twitter @taylorhughes with any comments or thoughts.
