Gulp.js and Front End Perfectionists With Deadlines

Crossposted from whatisjasongoldstein.com.

2016 is the year of the Javascript debate. Or, more accurately, the year of the inarticulate tirade written on Medium. Against my better judgement, I’m going to take a pass at criticizing the state of the art.

To be fair, it is the best of times for static asset build tools. It is also the worst of times for static asset build tools. We have better options than we’re ever had before, but they’re also abstruse, and woefully incompatible with other ecosystems.

The Problem

We can’t talk about this intelligently without defining the problem we’re trying to solve. Let’s suppose I have a huge site (people building single-page marketing sites have different needs than I usually do). I want to use SASS, some ES6 features, and some way to have explicit imports and exports in my javascript (I’m liking Browserify for this).

SASS and Browserify already are command line tools. I can do a sass -w static/css/site.scss:static/css/site.min.css and it will do exactly what I want. I can also do a browserify app.js -o app.min.js -t [ babelify — presets [ es2015 react ] ] and get results.

There are two problems with this:

  1. On bigger projects, I probably have more than one bundle for CSS or javascript. The Portfolio App rolls up the code for the CMS site separate from the client site. The Atlantic has different rollups for different regions of the site that don’t really overlap.
  2. Assuming I want to use — watch in order to render new outputs every time I make a change, I need to run a separate process in a separate terminal tab for each file. That’s ugly.

What about front-end perfectionists with deadlines?

For the rest of this essay I’m going to talk about Django, because that’s what I use for pretty much all my sites. If you don’t know Django, you should know the following:

Django is a Python web framework for “Perfectionists with Deadlines.” Part of the way it helps you work fast its (optional) batteries, such as an ORM, template language, user authentication system, etc. If you want to swap any given component out, you can, but they’re all solved problems by default.

Django has no opinions whatsoever about front-end code. However, there are third party libraries that will run these command line tools through the Python process.

Compressor

Django Compressor is a very opinionated app that believes your front-end assets belong in your templates. The code looks like this:

It’s elegant. Notice type=”text/browserify” on that last item? That’s because Compressor has settings that let me tell it to run the Browserify command from above on file with a certain type, like this:

For straightforward projects (and The Portfolio App, which needs to render CSS with variable color names on the fly for client sites), this is great.

But it gives its output files nondeterministic names, and it’s pretty much impossible to make it work with sourcemaps. If either of these are dealbreakers, you’ve probably outgrown it.

During development, Compressor only knows about the top layer of file it can see. Meaning if you use require or @import, and edit the contained files, Compressor has no idea it needs to re-render the file. You can get around this by telling it to build new files every time, but that results in slow refreshes as your site grows.

Pipeline

Prior to Node, the most popular answer to this in Django was Pipeline, which was modeled after a rails project. In Pipeline, you list your CSS and JS bundles in dictionaries, like this:

Pipeline, like Compressor, renders the bundles as part of the request cycle during development, and part of the deploy cycle [sic] in production.

Pipeline is solid, but suffers from the same shallow watching problem as Compressor, and because of the way its built, it’s extremely difficult to add support for Source Maps. The PR for this does work, but isn’t exactly light reading.

Gulp

Of course, Gulp is the de-facto standard for Node these days (I think?), and most of the people I’ve talked to with Django and complex Javascript seem to be using it in parallel to Django.

I finally dug in to test it as a replacement, and here’s the gulpfile for my personal site:

On the one hand, it’s simple, like a javascript presentation of the command line tools, but with named tasks that don’t have to enumerate each script. Because it watches for changes based on globs, imports and requires aren’t a problem at all.

But I don’t think this is the end game either.

For starters, switching from Pipeline to this added 53% more code than it deleted. I tried using a project called Django Gulp, which hooks my gulp tasks into the collectstatic process, but can’t tie watch into Django development server. Instead, I ran the gulp watch task in a separate tab. Which is… fine.

To run my tasks, I had to install a separate node package to wrap each piece of functionality I wanted to use, including SASS (which I have installed already), concat and rename (which are standard unix commands), and sourcemaps (which are builtin features of SASS and Browserify), something called are-we-there-yet and some kind of vinyl that I can’t play music on. There are 281 packages in all. That’s insane.

Think of the Apps!

We’re not quite done. Django’s standard architecture breaks your code into “apps,” written at separate folders that can each have their own static folder. In production, these are gathered into one folder (STATIC_ROOT), which is excluded from git. It’s not unusual to want to roll up static assets from across a wide range of apps.

I got this working by having my gulp file find assets from within the collected folder, and writing into a /dist/ folder that would be committed.

The results are the same, but now SASS @import’s work across folders. (The follow option tells it to follow symlinks ot their files). Since my watch code still monitors the whole repo, the task will build when I make changes to the original.

Still, I can’t shake the feeling that something here is very hacky. I seem to be weaving in and out between Gulp and Django’s staticfiles system: Django symlinks the files in the apps into the STATIC_ROOT, Gulp builds off of those and writes back into an app, which is then symlinked back into the STATIC_ROOT to serve on the site.

I wonder how this scales — what happens when I try to apply it to a large codebase with many different rollups and a large number of apps. It seems like a lot more cognitive overhead than Pipeline or Compressor had.

There’s some other yellow flags. RevSys, who generally knows a thing or two about Django, suggested combining Gulp with Compressor.*

So where were we?

I might be close. I might not be. I wonder if anyone has found a better way to make Gulp with the architecture of Django sites without cutting corners.

But this part I’m confident of: it’s about time for Django to have an opinion on this. It’s an ugly problem, and at this point, virtually all developers need a more powerful build process than staticfiles gives us by default.


To be continued. If I missed something, email me. This isn’t meant as an an excercise in armchair programming — I’m looking for real ideas to solve these problems.