Roots Next

a proposal

The following will be a breakdown of a concept that we have developed for introducing the next version of roots. This is very much an open proposal, and something I want everyone to be involved in, so please feel free to offer feedback on anything and everything!


The Problems

Why are we thinking about a new version? There are two primary motivating factors behind the decision. First, roots is too slow to be used with large websites, and that is a big problem. Second, roots’ maintenance needs are vast, and we currently don’t have a team that can fully support them to the point where roots can be stable enough to be adopted heavily as a mainstream tool. Let’s get into these a little deeper.

Roots is Slow

Not only is roots slow for fairly small sites, but it also gets slower the larger the site. It’s not because unoptimized performance or poor architecture. Roots’ code is actually quite fast, clean, and modern. It’s because of its dependencies, and users’ abilities to use a variety of different tools with it. Roots’ speed suffers because of two primary problems.

First, some compiled languages are very slow, and are not threaded at all. Stylus is one example. So when stylus runs on a large file, it can take up to 10 seconds (and we have clocked these times for some of our larger projects). It also runs on the main thread, meaning that while a stylus file is compiling, all other operations are blocked.

Many people think of node as a non-blocking language, but this is not true at all. Node’s async core methods like reading and writing files, making http requests, etc. are simply made such that in the background they offload the task to a separate “process” (you can think of this as a thread), which returns when it has finished. Stylus’ compilation does not do this, therefore it is a CPU-intensive parsing operation that runs in the main process and blocks it. This is the case with every other language that roots supports as well, as far as I am aware.

You may think that a potential solution to these issues would be simply to have roots delegate the entire compile for each file into a separate process. However, many compilers are passed options that can be functions (we frequently pass axis to stylus, which is a function, and various helper functions to jade). Functions cannot be passed between processes, only strings. Therefore, this cannot happen, and as long as we continue to allow the use of compiled languages that accept functions as options, we cannot have them execute in separate processes.

Second, roots does not utilize incremental builds. This means that every time there is a change to a single file in a roots project, it recompiles the entire project. Why don’t we just compile the file that has changed, you may think? Well, what happens if you change a jade layout file, or partial? Not only does that file need to be recompiled, every other file that incorporates it needs to as well. And in order for us to know that, jade needs to provide us with a dependency tree, that is, a map of which files depend on which other files, much like how node_modules works. And in order for roots as a whole to support this, every language that can be used with roots also needs to provide a dependency tree, and we need a way to normalize them all too.

Most languages that people use with roots do not provide a dependency tree. When roots v3 was released, not a single one did, in fact, so the architecture to support this type of thing was not built out for obvious reasons. Now, there are a few more that do, and this is great. In fact, jade and stylus both now produce dependency trees, as do browserify and webpack.

The dependency tree issue right now is one that could be solved. There are still issues — not every language supports dependency trees, they all output them in different formats, etc. But solving it would mean an absolutely enormous speed boost for roots, without question. So this is a primary goal of the next version.

Roots is Hard to Maintain

Any system as complex as roots will inevitably be difficult to grok and contribute to. In its history there has been maybe one or two other people who have been able to make consistent contributions to roots core other than myself. When v3 was written, one of the primary aims was to have a more clear and understandable codebase, and it was successful. But alas roots is simply a very complicated piece of software, so regardless of how clear and well-commented the code is, it’s difficult to understand thoroughly.

People also have a lot of requests for roots functionality. Typically, I am the only one who is able to respond to and build these pieces, and this is not sustainable. I don’t have the time to handle all the requests we get even with the small user base we have right now. And I also do not want to be stuck as a maintenance janitor, listening to requests and building them out for people. That would make me miserable, and then I would just stop and move on to other things in my life.

So another goal for roots is to have some way that makes it easier for people to get what they want, either by relying on third party plugins from a system that is already filled out with hundreds built by the community, or by having a plugin system that is so clear and well-built that anyone is able to easily build a plugin for anything they want. The former has obvious advantages, as it means we would not have to build a large foundational set of plugins, and people could use tools they are already comfortable with rather than starting fresh.

The Solutions

The basic structure I have in mind for the next version of roots is as such:

The core compiler would be webpack. Webpack has a large amount of community support, and is well built, many ways similar to the roots core. It also has a nicely documented and well-used plugin interface. It is built to keep track of dependency trees and only compile what is necessary when watching in development mode. Roots’ “extensions” api would simply become webpack’s, meaning we would no longer need to maintain and support it.

To compile html files, we would create a plugin for webpack picks up any jade files in the project, compiles them, reporting the dependency tree to webpack, then outputs each one as an html file in a public folder. Along the way, the plugin would pick up any images references directly within the jade files and add these to webpack’s pipeline so they could be processed as the user wishes, like compressing for example. Jade has been chosen here as our preference and a tool strongly supported by the community. While users could swap out jade for other languages we would not support these changes and consider them to be a fork of the project.

For css, we would use postcss to compile everything, and not support any other css compilation engines out of the box. With postcss, users can pick and choose their preferred transforms and set up a css development environment that is ideal for them. The interface for doing so is outside our control, so we would see no maintenance for this portion either. We would have to assemble a suite of postcss tools that we prefer, and I would probably release a version of axis that is specifically for postcss. As soon as the postcss syntax that removes brackets and semis is released, I would transition axis from a stylus tool to a postcss tool via a major version bump.

Fun fact — what is now called axis used to be written in sass, and was v1 of roots in its entirety. v2 made axis mandatory, and only in v3 could users remove it if they wanted, which most do not.

Javascript would be processed by babel and webpack. Webpack’s primary strength is processing javascript, so users would have a lot of control and flexibility here. Babel provides transforms that allow users to shape javascript to fit their specific preferences, much like postcss does for css. Our default stack would be ES6 (aka es2015).

We would also need to support a variety of popular roots extensions, such as dynamic content, yaml, records, contentful, etc. We would need to rebuild these extensions as webpack plugins. We also should carefully consider i18n, as it has been a goal for roots for a long time, but is without a doubt possible to achieve through webpack’s plugin interface.

The Interface

So, you may be thinking, does this means roots is becoming just a bloated webpack config file? Well, no. One of the primary strengths of roots throughout time has been its simplicity, achieved through a balance of opinionatedness and flexibility. We would also be wrapping the core webpack compiler in a thin config layer, to make things clean and convenient for users.

The first step here would be creating a new “app.coffee”, which would become an “app.js” written in es6. This would slim down config by covering up the default plugins and config that ships with roots, while also giving the user the ability to change things if they want. Basically, rather than being a blank slate by default that becomes bloated with configuration over time, it would be an opinionated and complex config file that abstracts away most of the decisions and bloat in favor of a cleaner, simpler interface with specific slots where customization is available.

We also would preserve the command line tool and sprout and ship integrations, which have proven to be very useful for us and our users. So “new”, “watch”, “compile”, and “deploy” commands would still work the same.

Ideas for Improvement

Ship is still a very difficult tool for us to maintain, but at the same time has a huge amount of potential. By the stats, it’s still widely used by roots users, with an average of about 15 deploys per day through the “roots deploy” command. At the same time, I feel like our use of netlify has been extraordinarily valuable, and I would feel comfortable advocating netlify as a primary deploy target for roots projects and building in tooling to make this more efficient specifically.

Some sort of tool that would parse netlify’s server config files and mirror them in the local dev server, in my eyes, would be an excellent addition to the platform, and presumably would not be a monumental effort to build.

Webpack’s dev server is an essential part of their workflow, and at least initially, is likely what we would be adopting. However, I think that browsersync’s offering is stronger as a dev server. We could potentially merge these two thoughts by making a new dev server for webpack that we’d use with roots that uses browsersync for refresh and has the ability to slot in config values such as netlify’s, as described above.

I’m very much open to any other ideas for improvement as well, please send in your feedback!

The Advantages

This new approach comes with considerable advantages (why else would we be thinking about it?)

  • Incremental build out of the box, massive speed boost.
  • Strong modern default stack with jade, babel, postcss.
  • We keep the slim config, opinionated yet flexible like users expect from roots.
  • No maintenance for the extension interface.
  • If users want to change the config to things that do no report deps, slow languages, etc. it is on them, we can and will not support this.
  • We get to build a lot of new plugins for webpack, for babel, for postcss, to get us back to the ideal syntax we are after (axis, etc).
  • We are all forced to bone up on es6, because this whole thing is going to be written with es6. this is finally a language that is universally agreed upon by js developers.
  • It will be more stable and we can market this aggressively and it will be adopted by a lot of people.

The Weaknesses

  • We give up total control. We have no longer written the core ourselves, so if we run up against a webpack limitation, it’s bad.
  • It takes time to build. This is not a one-week project by any means.
  • We need to learn a lot of new languages and interfaces, and not everyone might have time for this.
  • It still does not solve the threading problem.
  • We will almost certainly lose some of the conveniences we are accustomed to with the jade, stylus, coffee stack. es6 is more verbose than coffee. there still is no plugin for postcss that eliminates brackets and semis. stuff like this. more convenience is coming, but it will not all be there at first.
  • Webpack supports commonjs and amd in addition to es6. if only es6 modules are supported, tree-shaking can be used to eliminate unused library code and produce smaller bundles, which is a huge deal. rollup is a bundler that supports this right now, webpack version 2 apparently plans to support it but I couldn’t find a potential release date for webpack 2.

How to Leave Feedback

You might already realize that I’m in something of a love affair with medium. So it shouldn’t be a huge surprise I chose this platform to share this proposal with everyone. What I love most about medium besides its clean presentation is how it allows you to leave feedback on a piece.

If you have a note on a specific section, highlight the section, then click on the little button that pops up to leave a line comment. By default only I will be notified and it will be private between the two of us. However, I have the ability to set these comments to public, where all other collaborators can see them. For all conversations here, I will be doing this, unless it’s a quick question-answer exchange or an inappropriate comment.

If you have overarching thoughts on the whole thing, you can leave a response. There’s an area on the bottom where you can do this. It might not work for drafts, if that is the case just drop your commentary in the #roots slack channel and that will work fine.

For discussion, use the #roots slack channel. Do not pollute other channels with this stuff. If people decide to leave the channel that is fine, the channel is meant for these types of discussions, so if they aren’t interested they were in the wrong place.

Plan of Attack

So how do we tackle this in such a way that we’ll be able to begin using and testing it as soon as possible? Here’s my proposal:

  1. Build a webpack plugin that will compile jade files in the same way roots does for a static site, consuming all jade files in a directory and outputting them to public.
  2. Set up a system that does a similar thing for css, through postcss. I am pretty sure we can do this with just postcss and the extract text plugin.
  3. Get good conventions in place for javascript, this should be more straightforward.
  4. Make sure other types of files are passed through, add image optimization as a test.
  5. Mod the jade plugin so that it passes images through this pipeline as well, if needed. Also make sure it is correctly reporting dependencies.
  6. At this point we should be able to start building small sites to test it out.
  7. Do a full sweep and some testing to ensure that we have incremental builds working solidly, and everything is reporting dependencies as it should.
  8. Build out a roots-records-esque plugin, this is the core of the hybrid static approach and is important for our goals.
  9. Build out the other essential plugins like dynamic content, yaml, contentful, wordpress, etc.
  10. Start working on the interface and the way it wraps up the webpack core config
  11. Ensure that multiple environments work smoothly
  12. Polish up the deploy command and ship, make sure we are able to deploy to all targets efficiently, especially netlify.
  13. Make sure source maps are working correctly for html, css, js
  14. Think about building a dev server based on browsersync instead of the default webpack dev server, or test out this one.