Node.js is a Salad Bar

Thoughts on Boilerplate, Frameworks and Usability

Dan Abramov began his day as many open source maintainers do: fielding unrelated tech support questions on the issue tracker for his project.

Why “combineReducers” can not work at my project today suddenly, which could work well yesterday?

After some painful back-and-forth, the Redux team determined that the problem was caused by:

1. faulty package.json configuration
2. conflict with “redux” being used as both a package name and a filepath imported via Webpack’s resolve.root

…with both of these problems being a result of copy-pasting a boilerplate configuration. Closing the issue, Dan writes:

Copy-pasting configs from boilerplate projects always leads to hard-to-debug issues like this. It’s easy to miss somebody’s configuration decisions when you’re not the one making them. Don’t use boilerplate projects unless you understand each and every technology it uses!

I understand Dan’s frustration. But could we look at this from a different perspective? Don Norman’s The Design of Everyday Things teaches that there’s no such thing as “user error” — humans always make mistakes, and the failure to deal with these is on the product, not the user. How would we approach these user errors if we looked at them as design failures?

Problem: package.json configuration

When we design user-facing software, we ensure that a user can never get their data into an inconsistent state. But package.json is a magnet for inconsistent state; it is incredibly easy to break it and not know why. It’s particularly sensitive to copy/paste errors because of how commas work in json — even if one copies a working configuration exactly, one can still end up with a dangling or missing comma.

Many build tools enable or expect configuration in package.json, exacerbating this issue. While users can do most of their dependency management safely via npm install, there’s no corresponding safe way to edit ESlint or Babel config; we’re expected to do it by hand. And even when command-line tools are available, they’re limited in flexibility and aren’t discoverable for novice users.

Solution: Structured Data UI

Git has benefited from graphical tools; why not npm?

npm-gui

Paweł Stefański’s npm-gui is one such tool, which manages dependencies and tasks with a graphical editor. Future versions of this tool could handle package metadata, updating outdated dependencies and integrations with the npm registry, just as Git GUIs integrate with GitHub.

Problem: Webpack path resolution conflicts

CommonJS’s require has two ways of resolving paths:

• as packages in the node_modules directory, e.g. require(“redux”) loads node_modules/redux/lib/index.js
• as files using the filesystem, e.g. require(“./foo.js”) loads foo.js from the same directory as the file requiring it

This works well for libraries, where dependency graphs are simple parent-child trees, but leads to code like require(“../../../lib/foo”) in complex applications.

Webpack’s has a few hacky workarounds for this: resolve.root and resolve.modulesDirectories, which allow one to require from a projects own source directories as it would from node_modules; adding src to modulesDirectories lets one use require(“foo”) to load src/foo.js.

However, this puts the app’s source files in the same namespace as the imported modules; that means that, if you have a directory like src/redux, require(“redux”) will load that instead of the “redux” in node_modules. Even if one avoids this particular problem, files that use Webpack’s module resolution no longer work as regular node scripts; any code that uses that module resolution, such as tests, must also be run through Webpack.

Solution: Node-friendly Application Managers

Node module resolution is designed for libraries, not applications. Webpack’s module resolution hacks try to fight against this, but an elegant solution would work within this system. Just as Legit and git-up implement high-level workflows using low-level git commands, we could use tools to implement application workflows over npm’s library-oriented commands. Instead of overriding require’s semantics, this tool could help us manage our applications as collections of modules, handling the generation of package.json boilerplate and npm linking them together. Lerna could be a good starting point for this tool.

Metaproblem: Boilerplate Surprise

It’s telling that neither of the two problems above have anything to do with Redux. Both issues — package.json parse errors and Webpack paths — are fundamentally problems with Node and how browser tools interact with it. Both suggested solutions are tools that abstract over Node without fighting against it. If these problems share a fundamental cause, can they share a fundamental solution?

The notion of “React Boilerplate” reminds me of when I was learning Ruby on Rails back in 2011. I had a little bit of formal CS training, and some experience with web development going back to the GeoCities era, but I jumped into Rails understanding basically nothing about:

• Rails
• Ruby
• Git
• Unix

For all I knew, git push heroku master might as well have been a magic spell. But Rails embraced magic, allowing me to be surprisingly productive while using magical thinking — with a couple of incantations I could scaffold an app together, leaving me the simpler tasks of designing the views. And Rails did everything out of the box: server, ORM, testing — Rails didn’t require the user to put the pieces together.

The following generation of web development can be seen as a reaction to Rails. Some frameworks, like Ember, have adopted Rails’ “omakase” approach. However, much of the modern JS ecosystem, particularly on the server, rejects this in favor of independent, loosely connected modules with no magic binding them. If Rails is omakase, Node.js is a salad bar.

But even with a strongly modular approach, common patterns emerge. React, Redux, Babel and Webpack frequently end up connected together in various ways, but instead of combining them together into a framework with a CLI and high-level configuration, we’ve got boilerplate to copy and modify. But boilerplate is a worst-of-both-worlds solution: it combines the difficult parts of “magic” — a bunch of components that one doesn’t understand, all wired up together — without the benefits of a unifying abstraction.

Solution: Tools to Build-Your-Own-Framework

One approach to this problem is to build a framework around React and try to standardize around it. This is the approach driving Shasta, nwb and the next version of react-boilerplate. However, as useful as any of these may be, I suspect that no single dominant scaffolding system will emerge; the JavaScript ecosystem moves too quickly for that. I have my doubts that we’ll even be using React a year from now.

The real platform we’re building on is much lower level. React, Redux, Babel and Webpack are all incidental to our common thread: we are using Node modules for browser development. That’s it! The unifying feature is decentralization. There is and can be no One True React Framework; everyone builds their own framework. But if we’ve decided that we’re each going to build our own frameworks, we need to understand that we’re also responsible for the tooling, documentation and support that go with that. Can we modularize a framework’s culture?

Yeoman, a tool for generating code for web apps, came pretty close to doing the tooling part, but it emerged right before this style of app took off — it leveraged node as a build tool, but it didn’t fully embrace it as the way the app itself would work. Yeoman workflows were organized around Grunt, Bower and script concatenation right as those were becoming irrelevant.

The next generation of these tools will pick up where Yeoman left off and help application developers build tools that manage the complexity of their own apps, but in a form compatible with the Node philosophy. These tools would have share CLI and programmatic protocols and generate standard node modules, without requiring across-the-board buy-in for a particular build system.

Solution: Modern Browser Development Guide

No matter how intuitive our tools may be, we still need direct instruction. When I was learning Rails, I had Michael Hartl’s Ruby on Rails Tutorial. This book was useful for me because it took nothing for granted; it explained the whole workflow of Rails development — design, testing, deploying and using Git. It covered a lot of specific technologies — some of which are dated now — but the fundamentals have become the basis of my knowledge.

A tutorial like this for contemporary frontend development would guide novice developers in the basics of browsers & Node as well as Git & Unix. The tutorial would also cover React, Redux, back ends and build tools, but treat them as helpful modules that work inside the node/browser ecosystem; when those particular libraries go out of style only those chapters need to change. In other words, the tutorial embraces modularity as Node does.

Modular is the new Vanilla

Learning contemporary frontend development is a daunting prospect, even for experienced programmers in other languages, because there’s so much implied context. Every new tool is predicated on already understanding the history of problem that it’s trying to solve and the predecessors from which it differs. Babel and especially Webpack expect that the user is already familiar with Node modules and conventions, even though many frontend developers are coming from script tags and jQuery.

The modular approach is becoming the standard approach: ES6 Modules, regardless of the implementation details, will enshrine Node’s module culture into JavaScript as a whole. Developing for modules will be as “vanilla” as it gets. How can we do this without leaving novice users behind?

When we build the next generation of tools to take advantage of this modular world, and when we build our bespoke frameworks with them, we need to keep the novice programmer in mind. How can we bring them on board? How can we design systems that don’t require having read five years of discussions on GitHub to understand their design goals? Designing for modularity can mean designing for independence of ideas as much as independence of code.

Novices need frameworks. But a framework is more than code: a framework is a culture, with community, idioms and shared mores, built atop a foundation of code. Dan Abramov is right — boilerplates are hazards for developers who don’t understand them — but this is a failure of the boilerplate, not the novice user.

We might not be able to provide novice users with One True Framework, but we need to do better than boilerplate, both for their sake and for our own. If we can’t find a higher level abstraction to work with our own frameworks, how are we sure that we understand it ourselves?

JavaScript is big enough that it can support many approaches, from tiniest module to mightiest framework; it solves a diverse enough range of problems that it must.

Thanks to Vijith Assar, Nolan Lawson, hurricane jane, and Matthew McVickar for review.