Do you wanna build a snowball?

Or how creating a framework agnostic front-end development workflow can dramatically increase productivity and happiness

This is sort of a continuation of my “Abstract all the things” post, except here I’m going to dive into greater detail about how we solved all of our problems through abstraction and bootstrapping.

It’s been about 6 months since I started my contract at Noble Group (did I mention we’re hiring?) and this week one of my biggest projects was finally feature complete.

What is a snowball?

Snowball is the codename we use to define the set of tools and templates we use to create our front-end applications; we’re cool like that. Anyway, the idea is that a developer’s workflow, bootstrapping, templates, local development, testing, and deployment is wrapped up in a ready-to-rock-n-roll package that requires no configuration and is totally agnostic to OS, IDE, and user.

Why create a snowball?

It’s all about creating a consistent approach to developing applications, and the “target market” so to speak are the developers; so it makes sense to make things easy for them.

We have a three different engineering offices: USA, UK, and HK; and one of the key concerns is the unpredictable nature of front-end development could lead to greater issues down the road if, for example, these separated teams do things differently.

It’s about driving innovation, maintaining standards, and speeding up development

Some would argue that this approach could harm innovation because we’re requiring developers take a specific route on their development journey, or tying them in with technology that could overkill for what they’re trying to do. And that’s a very fair argument, in fact this has been one of my main worries through this process: am I making this simple enough? Am I reinventing the wheel? Am I repeating myself?

We don’t want developers to repeat themselves (in fact with this Snowball approach it allows us to keep development lean and DRY) and, more importantly, we want developers to share their code through modularisation of components. It’s through this sharing and componentisation that allows us to keep each application as lean as possible and with as little replication as possible. This opinionated approach means that developers can just kick-back relating to configuration and JFDI when it comes to front-end development.

So how do you build a Snowball?

Now that you’ve had the hard sell, let’s look at how a Snowball is created and why. Before any coding starts it’s always a good idea to start with why:

“Why do we need a Snowball?”

We want to enable developers to build applications rapidly, consistently, and in a simple, future-proof, and scalable manner. The belief is that a Snowball will give developers the ability and tools to successfully fulfill the needs of the business.

So how did we go about it? We started with a simple set of requirements:

• Quickly and easily bootstrap a new application
• Enforce a consistent user experience and design
• Ensure that code is well tested and correctly formatted
• Keep the development experience lean and DRY
• Create and reuse existing/new components
• Assist the developer in keeping the application’s dependencies up to date
• Provide an automated start-to-finish process from bootstrapping and templating, to CI and deployment

Seems like a pretty simple set, but as you can imagine each one posed its own challenge, so we made a number of assumptions:

Be written in JavaScript, HTML, and SCSS:
I’m not hating on pre-compilers; JavaScript is a pretty crappy language, but it’s sort of the assembly language of the web. It’s also very important to ensure that your application can be read by everyone, present and in the future, so writing it in JavaScript is much better than a punch in the face.

Comply with JShint, JSCS, and SCSS-lint rules:
This is very important to us: we needed to ensure that all our code was readable by everyone in the organisation; we strive to ensure that you can look at a piece of our code and not know who the individual was who wrote it.

AMD provided by RequireJS:
I believe *MD is very important when it comes to developing scalable JavaScript applications, especially for the front-end. IMHO RequireJS is the most suited because it doesn’t mess with the source while you’re developing locally, it’s got a very impressive compiler, and it’s a very mature library. There certainly are other *MD libraries out there, but for us RequireJS is the most sensible choice.

Bower for front-end dependencies and NPM for building tasks:
Business in the front, party at the back; this also allows us to roll our own internal Bower and NPM registries that we use to host our reusable components.

Grunt for building, Karma for unit testing, and Protractor for e2e testing:
Grunt works great for us because we have a lot of sequential tasks, tasks that share information between each other, and it’s a lot more mature than some of the other build systems out there (again, not hating, it works for us very well in this purpose). Karma (using Jasmine) is a brilliant way to test our application’s code against our target browsers, and Protractor is the icing on the cake allowing us to not only run e2e tests but (soon) platform stability tests (periodically run tests that keep an eye on the health of our platform and applications).

Bootstrap (with Bootswatch themes) and Font Awesome:
Many may argue that we don’t need to use Bootstrap, however I think it’s an excellent kit that fulfills its purpose of allowing developers to knock up a slick, good looking, and functional interface without ever having to write any CSS. What we need to remember is that Snowball is catered primarily for Java developers whose front-end skills may not be as refined as others; and Bootstrap is a very well documented solution to the problem. “And what’s wrong with Glyphicons?” I hear you say? Well, nothing, it’s just that Font Awesome gives us more, and with the two combined it makes for a very nice set.

Work out-of-the-box in any environment:
This is probably one of the most important requirements as we needed to be sure that every application was agnostic to whatever environment it was being deployed to (usually local, dev, UAT, and prod). This is important because applications are built by the CI and if the CI detects a new tag (version) has been pushed it’ll then publish an artifact to our internal repository; the idea of having an artifact per environment is stupid and damn right silly.

Use AngularJS:
“Wait, wat, AngularJS? But you said framework agnostic” — Yep, I did say framework agnostic, but I also said opinionated; the whole premise of a Snowball is zero configuration, just letting the developer JFDI. We chose AngularJS for a number of reasons, but mainly because there are a lot of resources out there and there are a lot of people doing (which in turn should make recruitment easier). I’ve had plenty of discussions with plenty of people about framework vs framework vs framework, etc… but at the end of the day I always say the same thing: don’t choose a framework (or technology) just for the sake of it, choose what’s best suited to your situation.

“This sounds way too opinionated for our liking”

Snowball serves a great purpose for us, and for your team and you it may be different; I keep coming back to this point about being opinionated, there’s nothing wrong with it because the less configuration the developers are faced with then the faster they can create.

Now let’s have a look at how the end-to-end development, testing, and deployment process is developed and brought together.

Bootstrapping

We’re big fans of automation, especially when it comes to tooling. In my Abstract all the things post I talk briefly about how we used Yeoman to bootstrap our Snowballs and IMHO I think it’s one of the better tools out there, especially for the approach we took. All Snowballs are started with one simple command:

$ yo snowball

This triggers our Yeoman generator, a developer is only presented with two choices: application’s name and its git repository URL. By default both are pulled in automatically (if possible) where the application’s name is usually the name of the folder the developer’s currently in:

Snowball generator automatically picks up the existing git config

If no git configuration is found (and thus a git hasn’t be initialised in the folder) it’ll initialise git and set the given URL as the remote origin:

Snowball generator will initialise a git repo if one doesn’t already exist

What’s the advantage of this? Well it’s a smart approach to helping the developer get up and running quickly.

Once those choices are in place Yeoman comes into its own by placing our base Snowball template in place. This template exists as a repository that works exactly like an application, instead of just a load of untestable templates (I wrote about this approach in Abstract all the things). What a developer is left with is just application code:

.
├── assets
│ ├── img
│ ├── js
│ │ ├── controller
│ │ ├── directive
│ │ ├── factory
│ │ ├── service
│ │ └── vo
│ ├── partial
│ │ └── directive
│ └── scss
└── test
 ├── e2e
 │ └── load
 └── unit
   ├── controller
   ├── directive
   ├── factory
   ├── service
   └── vo

I’ve left off the files because they just make the list unreadable, but this is all that’s inside our Gruntfile.js:

module.exports = function (grunt) {
 grunt.loadTasks(‘node_modules/grunt-snowball-tasks/grunt’);
};

Yep, that’s it (I mean just compare this to generator-angular’s effort). All our Grunt tasks and their configuration are stored in an internal NPM module (called grunt-snowball-tasks). So our package.json dependencies are simply:

{
 “name”: “snowball”,
 “description”: “A scalable AngularJS framework for snowball apps”,
 “scripts”: {
   “postinstall”: “bundle && bower install”,
   “test”: “grunt test”
 },
 “repository”: {
   “type”: “git”,
   “url”: “git@somewhere.com/snowball.git”
 },
 “author”: “Ahmed Nuaman (ahmednuaman@thisisnoble.com)”,
 “devDependencies”: {
   “grunt”: “0.4.5",
   “grunt-snowball-tasks”: “http://some.git.server/grunt-snowball-tasks/repository/archive.tar.gz?ref=0.0.20"
 },
 “version”: “0.2.0"
}

Nice right? The beauty of this approach really comes in play later when I talk about how Snowballs deal with auto upgrading.

Now since a Snowball is a front-end application and we use AMD, this configuration needs to be somewhere right? This is all kept in the Bower.json file:

{
 “name”: “snowball”,
 “dependencies”: {
   “snowball-helpers”: “http://some.repo/snowball-helpers/0.0.11/snowball-helpers-0.0.11.tar",
   “snowball-host-variables”: “http://some.git.server/snowball-host-variables/repository/archive.tar.gz?ref=0.0.7"
 },
 “appConfig”: {
   “paths”: {
     “angular-animate”: “../vendor/angular-animate/angular-animate”,
     “angular-cookies”: “../vendor/angular-cookies/angular-cookies”,
     “angular-mocks”: “../vendor/angular-mocks/angular-mocks”,
     “angular-resource”: “../vendor/angular-resource/angular-resource”,
     “angular-route”: “../vendor/angular-route/angular-route”,
     “angular-sanitize”: “../vendor/angular-sanitize/angular-sanitize”,
     “partial”: “../partial”
 },
   “shim”: {},
   “appRequirements”: [
     “angular-animate”,
     “angular-cookies”,
     “angular-resource”,
     “angular-route”,
     “angular-sanitize”
   ],
   “ngDependencies”: [
     “ngAnimate”,
     “ngCookies”,
     “ngResource”,
     “ngRoute”,
     “ngSanitize”
   ],
   “startupRequirements”: [
     “configs”,
     “partials”,
     “routes”,
     “controller/app-controller”
   ]
 },
 “version”: “0.2.0"
}

This file could get quite big, but this contains the configuration required to run the application locally, test it, build it, and run it when it’s deployed. It has the configuration for what we install via Bower, RequireJS paths and shims, r.js paths configuration for building, and the startup requirements and dependencies at runtime.

Let’s have a closer look at some of the internal libraries we use to make developers’ lives easier:

grunt-snowball-tasks

We ship all our Grunt tasks in a single NPM module. The advantage here is a much smaller dependency (package.json) file for the developer, in addition to making it a lot easier to keep application’s build tools up to date.

We have four Grunt tasks that we expose to the developer:

grunt

This is the Grunt default task, it does the following:

• Compass dev:
  This simply runs a Compass task in development mode, allowing source maps and verbose compiled code to help with debugging.
• Freeport:
  A very handy little task that allows us to grab freeports between 8000–8010 (for the local web server) and 35729-35739 (for livereload); this means that a developer can run up to 10 applications locally without port conflicts.
• Express dev:
  “Dev?” — Yep, we have two environments for the local Express web server, the dev target is for local development, and we have a prod target that we use for our e2e tests.
• Open:
  Another handy little task that launches the local application in the developer’s default browser, it’s a lot easier than the developer having to type the long host (for CORS issues) and port every time.
• Watch:
  Finally we watch the developer’s files for them and run tasks when changes have been detected.

grunt test/unit/e2e

This is our shortcut task for running all the tests:

• JShint, JSCS, and SCSS-lint (unit and e2e)
• Karma (unit only) and Protractor (e2e only)

I’m still looking for a nice angular-compliant HTML linter, so if you know one then please drop me a message.

grunt build

I believe it’s always a good idea to allow the developer to do a dry run of building the application (essentially testing and packaging it ready for deployment to our artifact repository). The build task is the first part of the deployment process:

• Get the latest git sha1:
  This is very important for cache busting; the latest git sha1 is appended to the compiled JS and CSS files, this helps ensure that there are no old cached files when the end user launches up the application.
• Clean:
  Essentially remove all existing dist files and folders.
• Test (as described above)
• Imagemin:
  It’s always good practice to optimise your application’s images, after all the artifact’s size will be smaller (taking up less space on the repository) and there’s less for the end user to download.
• Compass prod (as described above, but ready for production now)
• Copy:
  Simply copies the files that won’t be touched by optimisers such as the index.html, *.json files, fonts, svgs, etc…
• Combine media queries:
  Another handy plugin that just tidies up the final outputted CSS by putting all the media queries together. (Plus it’s faster to combine them)
• CSSmin:
  “But you’re using Compass right?” — Right, but unfortunately Compass doesn’t allow us to specify the filename of the final file, whereas CSSmin allows us to specify the filename and pass in other CSS files to combine into one.
• RequireJS:
  One of the greatest features of using RequireJS is its awesome r.js compiler. The task Snowball uses is incredibly simple as all the configuration is kept in the bower.json file.
• ngTemplates:
  A great plugin that pre-caches all the application’s HTML partials into a JS file, this not only makes the application more performant but also deals with cache busting the HTML as well.
• Uglify:
  “But doesn’t RequireJS uglify it?” — It does, however, it then also mangles the little flag we set that ngTemplates overwrites, so it’s turned off and occurs after the templates have been JSified.
• Text replace:
  Remember the git sha1 I mentioned above? Text replace is used to replace the standard references with the new sha1'd files.
• Express prod and Protractor prod:
  By this point in the build process everything has been compressed, uglified, cached, and moved into a dist folder, it therefore makes sense to run sanity tests against that code.
• Compress:
  Finally when everything is nice and happy the dist folder is compressed into a tarball ready to be sent to the artifact repository.

grunt deploy:VERSION

One of the main requirements of Snowball was painless version deployment, that’s the purpose of this task. It takes a version as its only argument (as semver) and checks to that the specified version hasn’t already been deployed. If that’s all good, it’ll then run the build task, and once complete it’ll bump the version in the bower.json and package.json files, create a new tag with the version, and then push the files and tag up to the origin.

By then the CI will receive a build hook, and should the build be successful the CI checks whether the version deployed exists in the artifact repository, if not the dist.tar file is then pushed and it’s then ready for automatic deployment.

Simples right?

snowball-helpers

Of course Snowball does a lot more than just abstract the build process, one of the great things about taking a structured approach to your code is that any problems down the line (library API changes, new requirements, etc…) can be easily catered for with no to little code changes for the developer.

Snowball uses a base library of helpers (delivered via Bower) that allow the developer to forget about configuration and help them deal with any quirks a framework (in our case AngularJS) may throw at them:

Start up:
This is the common definition that every application executes at runtime. Since it’s the same code for every application it makes sense to write it once as part of a library. It’s also a pretty important step in making the application run in any environment, since it loads environment variables from a json file (delivered via Bower) that then overwrite the application’s defined variables. This allows the devops team to change this host variables json file to suit each environment, thus making the applications totally agnostic.

Stopping duplication:
Something that’s a bit annoying about AngularJS is the way dependency management is done; this often leads to duplication:

var app = angular.module('myApp', []);
app.controller('FooController', ['$scope', '$location', function ($scope, $location) {
  $scope.foo = 'foo';
  ...
});

This is painful. Yes it’s great for minification and what not, but leads to a number of problems:

• Getting a reference to the AngularJS application in order to register a module.
• Repeating the injectables for the module.

Instead, using the helpers, Snowball allows a developer to define, say, a controller, like so:

S.controller(‘FooController’, [
  ‘$scope’
], {
  init: function () {
    this.$scope.foo = 'foo';
    ...
  }
});

It’s a lot tidy, duplication has been dealt with, and as an added extra the init functions on class-based are executed at the initialising time of the class (eg when a user lands on a view).

This also solves the problem of missed injectables, where minifying can mangle the argument’s name and cause problems when the application is deployed.

Abstracting out the framework:
This is an important factor IMHO because it gives us the ability to migrate to any framework we desire with minimal effort and impact on the application’s code. One of the brilliant features of Yeoman is the sub-generators, Snowball allows the developer to run a simple command that’ll create collections, controllers, directives, factories, filters, directives, services, VOs, and partials:

yo snowball:controller

Snowball allows the developer to keep their application consistent

This approach allows the developer to just focus on writing a great application and not having to worry about missing out injections, templates, and such.

So what about upgrading?

All the applications developed need to be future proof. They also need to live on, and that’s important because no one ever liked BAU code. So how does a Snowball stay fresh? Well the beauty is all in the abstraction.

All of a Snowball’s internal dependencies are handled by the abstract packages, this means that all a developer needs to do to upgrade a Snowball is simply run:

yo snowball:upgrade

It’s like *magic*

And just like that all the internal dependencies are checked and if there’s a new version then the respective json files are updated with the latest. All a developer then needs to do is just run npm install.

Not bad right?

In conclusion

Snowball allows our developers to really hit the ground running by making all the decisions up front, and making the whole development, debugging, testing, and deployment processes as simple and streamlined as possible.

It may not be for everyone or every project, but if your team are popping out the same sort of applications using the same sort of libraries and tools then it makes sense to stop repeating yourself and love the world of automation!

Peace.