How JSPM, System.js and Gulp simplified our build pipeline

I work at a company called Points, the leader in loyalty commerce transactions and currency management. I joined the team about a year ago to help implement the rewrite of Points.com into what you see today. At that time we were moving from a legacy website to a modern, responsive, single page application.

When I arrived, there had already been some work done but the project was in a very early stage and the front end lacked any real structure. The first thing I noticed was there was a lot of code and not a lot of functionality. Our primary tools were Grunt (task runner and build tool), Angular (MVC) and Sass (CSS precompiler). Though I really like these tools, there is a lot of confusing configuration involved in setting up the build pipeline. It didn’t help that we were using a Gruntfile from a Yeoman generated Angular project modified to our needs.

So, once we refactored the existing pieces of application code to follow some Angular best practices, I took some time to rewrite our Grunt configuration from the ground up. I wanted to be able to account for each task we needed based on two basic contexts: development and production builds.

In development, our web app needed to:

• compile Sass to css
• pre-cache our Angular templates (avoiding a network request for each individual template)
• render an index.html file with our application code using <script> and <link> tags
• start a local server to serve our app
• reload the page any time a file changed

For production we needed to do these additional steps (minus the server and livereload):

• concatenate and uglify our application code (to have a single js and css bundle instead of individual files being loaded over the network)
• minify our css
• revision our application files (avoids users getting cached/stale files after we deploy new code)
• copy over the files into a build directory

In addition to those two scenarios, we also needed to run unit and end-to-end tests for obvious reasons, but essentially to ensure our application continued to work as we continued to build.

Finally, we needed to have a single build toolchain that could work for multiple apps. We have both a consumer facing application as well as an admin app to manage users and other business needs. Both these apps had different browser dependencies (managed by bower) but used the same Gruntfile and build tasks.

What we had

left: filesystem showing structure. middle: Gruntfile showing list of build tasks. right: the generated index.html for development

Now, if you’re a developer, you will look at the image above and cringe! But you will also sympathize with my situation and your forgiving side might even commend me after seeing that it actually worked pretty well. So well, in fact, that we were able to build both the admin and flagship app using this process all the way into production and it still stands today. It isn’t perfect but it does what it needs to do to keep the business moving forward.

I won’t bore you with all the nitty gritty details on how it works but I would like to focus on how exactly we generate our index.html page.

In order to get the output you see on the right side of the image, we use two primary tools: A Grunt template, and wiredep. We feed our index.html template a list of application files and wiredep uses our bower.json file to wire up our 3rd party dependencies.

Then, when we want to bundle for production, we use a tool called usemin that takes the generated index.html and replaces references from non-optimized scripts, stylesheets and other assets to their optimized version. Again, ugly and imperfect but effective. The end result is exactly what we wanted, a single app.js file, a and a single vendor.js file (same for css).

production build index.html file

For all its short comings, this is great for developers because they really don’t need to think about any of this at all. All they need to do is focus on writing code for the angular app and run grunt serve at the command line.

The Big Refactor

After we made our way through the launch and settled down from the excitement, the very first thing I wanted to do was revisit the tooling and processes we used in development. I was proud of what we accomplished but unsatisfied.

Also, a year was a long time and all the while I had been following the advances in web development. Words like ES6, Browserify, React, Babel, and Webpack began popping up on my Twitter feed and emails. I knew ES6 was a big thing and thanks to these new open source projects, it became obvious that we could start using them today if we could just fit them into our build system.

As work slowed down, I used the opportunity to begin researching and experimenting with all the things I read about. We weren’t about to rewrite the app in a different framework (like Ember or React) but I knew I wanted to start using some of the new ES6 features, especially modules, to help bring a better structure to our codebase. As I researched, I was reading more and more about a new tool called jspm and jspm-cli. I won’t turn this story into a sales pitch for the tool (although i urge you to click the link and discover how awesome it is) but I will list the things that really made me excited.

• Use ES6 modules right away without any build task just by using System.js
• Import 3rd party dependencies regardless of their format (commonjs, AMD, UMD etc.)
• Manage third party dependencies using JSPM instead of Bower from multiple registrys including NPM and Github
• No more wiring up each file using the ole’ script tag
• Bundle your app for production including concatenation and uglify in a single task
• Single main.js file used to import all your apps dependencies and bootstrap the application

After a bit of reading I began tinkering. I decided to switch up the task runner to Gulp. Having used it at my previous company, I really liked the more straight forward configuration approach compared to Grunt. Since I was experimenting, I took the opportunity to compare and contrast. Turns out it was indeed simpler to use, read and configure. After a day or so I came up with a seed project of my own that was a simple and straightforward implementation for development and production builds. Getting more comfortable with the workflow with a basic implementation, I set my sights on the big refactor.

All the requirements still applied. We still needed to accommodate multiple apps that could use a single Gulpfile configuration for development and production builds. I also needed to be sure that however this refactor panned out, it was not going to be a nightmare to port over all the existing code. If this was going to work, the majority of the application code needed to be copied over untouched.

After a few days, a lot of Googling, Stackoverflowing, Gitter chatting, and beer drinking; I had a working app.

What we have today

Development build — far-left: filesystem structure, left: gulpfile with simpler build tasks, middle: rendered index.html file for development, right: main.js file where we import all of our apps dependencies.

Without even looking too closely, the thing to notice is that our rendered index.html file, which was previously 200 lines of code is now only 19 lines (and without all those silly html comments). We’re still loading all of our app files over the network but it’s all handled for us (including transpiling the ES6 code into ES5) on the fly, in the browser, by System.js and Babel. (pause for applause). Goodbye ridiculously long list of script tags!

We now have a single main.js file that is our manifest where we import all of our application’s dependencies and registers the Angular controllers, directives, services, etc. on the app. Of course there is more refactoring to be done here by breaking out the app into smaller modules, but this is a huge step in the right direction.

The Differences

Today, the way we add a new piece of code to our application is by doing this:

app.js

angular.module('app', []);

my-controller.js

angular.module('app').controller("myController", [
  "$scope",
  "$injector",
  function ($scope, $injector) { ... }
]);

This works because we ensure app.js loads before my-controller.js. We have code like this spread all over our codebase! Conversely, the new and imroved ES6 way of doing the same thing would be

my-controller.js

export default [
  '$scope', 
  '$injector', 
  function ($scope, $injector) { ... }
];

And then in main.js

import myController from './my-controller';
let app = angular.module('app', []);
app.controller('myController', myController);
export default app;

This same pattern can be followed for all files making it very easy to see exactly what code is being used in the app by looking at a single file.

Another thing to point out with this new pipeline is that we get sourcemaps for free. So when we add breakpoints in our ES6 code we break at that exact line and not the transpiled code making debugging that much easier.

Production Builds

Finally, I’d like to point out how this simplified our production build. Earlier I mentioned how in our current pipeline, we render all the apps dependencies into an index.html file and then run a grunt task called usemin to replace those references with an optimized file. Really what happens is usemin generates configurations for three tasks for us (concat, uglify, and minify) and then replaces the list with the generated file.

All that nonsense is replaced by using systemjs-builder, which traverses our import tree, resolves any circular dependencies, transpiles our ES6 to ES5, and bundles everything up into a single self executing file in the form of app.js. Our build task is 25 lines of code and looks like this

gulp.task('buildJsAndRender', function(){
  var builder = new Builder({
    baseURL: "./"
  });
  builder.loadConfig('config.js').then(function(){
    var source = path.join(appPath, 'main');
    var destination = path.join(buildAssetPath, 'js/app.min.js');
    var compileJs = builder.buildSFX(source, destination, {
      minify: true,
      sourceMaps: false
    });

    compileJs.then(function(){
      //add revision hash
      gulp.src(path.join(buildAssetPath, 'js/app.min.js'))
        .pipe(rev())
        .pipe(gulp.dest(path.join(buildAssetPath, 'js')))
        .on('end', function() {
          //delete unrevisioned version and render index.html
          exec('rm ' + path.join(buildAssetPath, 'js/app.min.js'), function () {
            render();
          });
        });
    });
  });
});

Conclusion

Long story I know, but if you made it this far then you’re like me and truly interested in learning about modern web application development. There is a wealth of technology and open source code out there to try and a million ways to get the job done, this just happens to be one.

In my case, JSPM and Gulp have been invaluable in helping us build a framework for developers to write applications easier and more efficiently. It also de-obfuscates our build steps allowing any developer to come in and contribute even more tasks to the pipeline. That kind of clarity is extremely satisfying.