Let’s Play MEAN, Part VI.

Building a starter repo from scratch


Welcome back to this exploratory dive into the MEAN stack.
We’re building a seed repo we can use for all future MEAN projects.
This is Part VI— You can start this adventure over at Part I.

Introducing Gulp

Getting Gulp set up has been one of my favourite parts so far.

Gulp is a task runner that helps us with all of the menial, dumb tasks we need to do. Rails gives us a lot of these tasks for free, and while that might seem like an advantage, the amount of customization Gulp provides is fantastic.

Gulp Vs. Grunt

So, Gulp is sort of the new kid on the block; Grunt has been around for longer and has only recently been usurped by Gulp in popularity (or is it uslurped?? Sorry.)

There’s a bunch of reviews comparing them, and it didn’t take me long to figure out that Gulp was more my style. It features linux-style piping, which saves on disk reading/writing (so it runs faster) and lets you build tasks quicker and easier, in less code.

Gulp is a streaming build system because one task flows into the next, instead of each one being its own separate entity. I think the best way to understand it is to look at examples, so don’t worry about it just yet if it’s confusing you.

So, what’s it used for?

For Javascript, it’s commonly used to uglify/minify (reduce the size of each javascript file), concatenate (reduce the number of server requests), lint (catch errors), and more.

For CSS, it’s commonly used for SASS to CSS conversion, minification and auto-prefixing (adding those annoying vendor prefixes to make CSS browser-compatible).

We’re going to do all this stuff, as well as a few other nifty tricks along the way. We’ll also set it up so it does all these tasks automatically when the files change, and auto-reload our browser so we can see the changes.

Getting Set Up

Gulp is an NPM package, and on its own it doesn’t do all that much. It relies on plugins, with the mindset that each package should have one job, but do it really well.

Because Gulp and Gulp plugins are only used in development, we’ll add it to our devDependencies in our package.json:

// package.json
“devDependencies”: {
“chai”: “^1.10.0",
“gulp”: “^3.8.10",
“gulp-autoprefixer”: “^2.0.0",
“gulp-concat”: “^2.4.2",
“gulp-jshint”: “^1.9.0",
“gulp-livereload”: “^2.1.1",
“gulp-minify-css”: “^0.3.11",
“gulp-notify”: “^2.0.1",
“gulp-rename”: “^1.2.0",
“gulp-sass”: “^0.7.1",
“gulp-sourcemaps”: “^1.2.8",
“gulp-uglify”: “^1.0.1",
“mocha”: “^2.0.1",
“supertest”: “^0.15.0"
}

Run ‘sudo npm install’ and take a minute to do some stretches or pet your cat while NPM does its thing.

Also, you’ll want to run ‘sudo npm install -g gulp’ to install Gulp globally, so you can use the ‘gulp’ CLI command.

Choosing SASS Plugins
As you’ll see below, there are a lot of tasks I want to accomplish with CSS. Often times, there are multiple options for plugins to use.
Gulp-sass and gulp-ruby-sass are competing plugins that do the same thing in different ways. I spent several hours trying to find a configuration that would do everything I wanted, and gulp-sass was the only one that worked.
There have been times where I wonder if this guide is actually helpful to anyone else, but I’ll say right now that the end recipe in this section can save you hours of tedium if you’re looking to do sass-to-CSS task running.

Writing our first Gulp task

We need to create a file named gulpfile.js in our root directory. Let’s get the infrastructure set up:

var gulp         = require(‘gulp’),
sass = require(‘gulp-sass’),
autoprefixer = require(‘gulp-autoprefixer’),
minifycss = require(‘gulp-minify-css’),
jshint = require(‘gulp-jshint’),
uglify = require(‘gulp-uglify’),
rename = require(‘gulp-rename’),
concat = require(‘gulp-concat’),
notify = require(‘gulp-notify’),
livereload = require(‘gulp-livereload’),
sourcemaps = require(‘gulp-sourcemaps’);
gulp.task(‘default’, function() {
gulp.start('styles', 'scripts');
});
gulp.task(‘styles’, function() {

});
gulp.task(‘scripts’, function() {

});

First, we require our boatload of dependencies. Then, we create our default task. This is the task that gets run when you run ‘gulp’ from the command line. We want to use it to launch our two other tasks, styles and scripts.

The best way to learn is to play along, but I do go back and forth a few times.
If you’d prefer, you can just grab the final gulpfile here.

Starting with Styles
For the SASS to CSS task, I wanted to take all my SASS files, prefix them, concatenate them, and spit out two files: style.css for development, and a minified style.min.css for production.

Here’s my first draft for this task. We’ll go through it in detail below, and then we’ll have a think about how to improve it.

gulp.task(‘styles’, function() { 
return gulp.src(‘public/**/*.scss’)
.pipe(sass())
.pipe(autoprefixer(‘last 2 version’, ‘safari 5', ‘ie 8', ‘ie 9', ‘opera 12.1'))
.pipe(concat(‘style.css’))
.pipe(gulp.dest(‘public/assets/css/’))
.pipe(rename({suffix: ‘.min’}))
.pipe(minifycss())
.pipe(gulp.dest(‘public/assets/css/’))
.pipe(notify({ message: ‘Styles task complete’}));
});

First stop: gulp.src
So the first thing we need to do is find all of our SASS files. I’m looking for any SASS files in the /public folder.

Wildcards.
I think its pretty well-known that the asterisk (*) is a wildcard, so *.scss should be pretty straightforward, but the double-asterisk preceding it (**) might not be as intuitive. Essentially it signifies that there may be multiple levels of directories, and we want to . It could be public/style.scss, or public/a/b/c/d/e/f/g/style.scss. It’ll match it either way.

Alright, so we’ve gathered all of our SCSS. That data is loaded into memory and, like a car in a factory, moves down the assembly line.

Second stop: Sass
This is where our scss files get rewritten as straight-up CSS. It takes an options object as an argument, but for now we’ll go with the defaults.

Third stop: Autoprefixer

I don’t know about you, but I’m pretty sick of this sort of baloney:

-webkit-box-shadow: 2px 2px 8px rgba(0,0,0,0.5);
-moz-box-shadow: 2px 2px 8px rgba(0,0,0,0.5);
box-shadow: 2px 2px 8px rgba(0,0,0,0.5);

Delegate this job to autoprefixer, and you never have to think about it again. I’ve decided I want to support the most recent 2 versions, as well as a selection of hand-picked browsers like IE 8.

Fourth & fifth stops: Concat and gulp.dest
We’ve opened all the .scss files that match our glob, and we’re holding them all in memory as we de-sassed and auto-prefixed them. Now we want to write it to disk.

The first step is concatenating it: combining all the files we’ve opened into a single, monolithic CSS file.

Why concatenate?
I’ve spoken to developers who dislike the idea of concatenating all their CSS into a single massive file. If you’re not careful with your CSS organization, it can cause namespacing conflicts, and it seems like wasted bandwidth if the user doesn’t visit every page on your site.
The reason we do this (as far as I understand) is to reduce the number of server requests. The main cause of latency typically isn’t large CSS files, it’s needing to go back and forth to the server to fetch lots of small files. The technical explanation is beyond my current understanding, but I think there are limitations to the number of files that can be made per round-trip.
Plus, it’s just nice to get that content loaded and cached from the get-go. Makes every page load after that feel that much snappier.

We’ve concatenated it into a file named style.css, and then with gulp.dest, we’ve saved it to /public/assets/css/style.css.

Final few stops: rename, minifycss, and notify.
By now, it should be pretty clear how this piping thing works, so let’s speed up a bit with our coverage.

The final step is to create a minified production-ready version. We rename the file to have a .min suffix (style.min.css), and run it through minifycss(). Finally, we write that to the same directory as our development version, throw up a notification message to let us know its done its job, and call it a day.

Better Solution with Sourcemaps.

Now this process is all well and good, but it does seem a little redundant to have two copies of our CSS file when they contain exactly the same styles. What if we could only have the minified version, without losing that human readability we know and love?

That’s the thinking behind sourcemaps. They’re a separate file that “maps” every line the minified version to the location of the original. This means that in the Chrome Inspector, file names and line numbers are accurate. When something goes wrong, error reporting stays true.

Here’s what the task looks like with sourcemapping :

gulp.task(‘styles’, function() {
return gulp.src(‘public/**/*.scss’)
.pipe(sourcemaps.init())
.pipe(sass())
.pipe(autoprefixer(‘last 2 version’, ‘safari 5', ‘ie 8', ‘ie 9', ‘opera 12.1'))
.pipe(minifycss())
.pipe(concat(‘style.min.css’))
.pipe(sourcemaps.write(‘./’))
.pipe(gulp.dest(‘public/assets/css/’))
.pipe(notify({ message: ‘Styles task complete’ }));
});

We need to tell sourcemaps to start ‘recording’ at the beginning, because we need to retain the original location before we start modifying it. So the stream starts with a call to sourcemaps.init(), and when we’re done all of our modifications, we call sourcemaps.write() to create the maps file.

Maps can be done inline, but I prefer to write them to a separate file, for reasons I’ll get to shortly. You can do this by passing in the file location, and I’m just putting them in the same directory with ‘./’.

The Results
Check out the magic. On the left, we have our non-sourcemapped version. On the right, sourcemaps.

Kaboom. That’s your mind, being blown.

Because we’re concatenating and minifying all our CSS, it shows up as ‘style.min.css’ line 3. But I don’t have a style.min.css! Not in my development SCSS files, at least! I do, however, have a main.scss, and line 2 is exactly where that box-sizing property lives.

Sourcemapping by the numbers.
Sourcemaps are an awesome idea, but are they efficient? Is it really an optimization if the sourcemap file is bigger than the un-minified CSS file?
I did the math. I have about 1900 lines of commented, un-minified CSS (mostly font-awesome and normalize). Uncompressed, this file is 32kb. Compressed, it goes down to 25kb. The sourcemap file is a whopping 52kb. This means that minified + sourcemap is more than twice the size of just using the uncompressed file.
For this reason, I store the sourcemap in a separate file, and I don’t check the map file into the repository. This way, in development I have all the cool perks of a sourcemap, but it never gets pushed to the production server.

Next up, Javascript.

Let’s have a think about what we want our Javascript tasks to do.

Similarly, we want to minify (for whatever reason it’s called uglify when you do it to JS) and concatenate. Sourcemaps are even more helpful for JS than they are for CSS, because when there’s an error, it’ll tell us what file and line the error is on.

Let’s throw it into our file and see what it looks like:

gulp.task(‘scripts’, function() {
return gulp.src(‘public/**/*.js’)
.pipe(sourcemaps.init())
.pipe(concat(‘main.min.js’))
.pipe(uglify())
.pipe(sourcemaps.write())
.pipe(gulp.dest(‘public/assets/js’))
.pipe(notify({ message: ‘Scripts task complete’ }));
});

We’re going to concatenate this into a file called main.min.js, and put it in the assets/js folder. Let’s update our index.html to replace all of our custom angular scripts with just that one:

<head>
<meta charset=”UTF-8">
<base href=”/”>

<title>MEAN stack template</title>

<link rel=”stylesheet” href=”assets/css/style.min.css”>

<script src=”assets/libs/angular/angular.js” type=”text/javascript”></script>
<script src=”assets/libs/angular-route/angular-route.js” type=”text/javascript”></script>

<script src=”assets/js/main.min.js”></script>
</head>

Keeping vendor libraries separate.
We’re only going to run our Gulp process on our code. The biggest reason is because in production, we can (and should) use CDNs for big libraries like Angular. Most big sites use CDNs, which means that most of your users will already have these files cached. No reason to make them re-download them.

The problem, though, is we never told Gulp which files to include, we just told it to grab all Javascript in the /public directory, which unfortunately includes our /libs.

There’s a second, and even bigger, problem with just passing in all our javascript: order matters!

Remember back in Episode IV, we pointed out that we need to load our modules before our controllers, and our Things/Dashboard modules before our mainApp module in app.js? Well, when we concatenate it all together, that order gets all fucked up. Take a look at the resulting hot mess:

The first two lines refer to the fact that we’ve loaded Angular 3 times; in our original document, the angular.js we’ve concatenated in, and the angular.min.js we’ve also concatenated in. Those are just warnings; not good, but not breaking.

The main issue comes from the fact that we’re trying to load ThingController in our Routes before we’ve injected the Thing module into our app. This is the first of a cascade of errors relating to this problem.

Gruntfile.js? Huh?
I was perplexed by the third error, coming from a Gruntfile. I never installed Grunt and I’m going 100% gulp! What’s the deal?
Turns out, several node modules like animate.css and JSHint (which we’ll get to shortly) use Grunt behind-the-scenes. I’d be lying if I said I wasn’t a bit bothered by having a redundant task runner lurking in my project, but whatever. I can live with it.

Concatenating Angular Properly
When selecting gulp.src, so far we’ve been passing in strings like “public/**/*.js”. Thankfully, Gulp lets us instead pass in an Array, and it’ll preserve the array’s order, loading them one at a time.

Here’s what the new line looks like:

return gulp.src([
‘public/**/*.module.js’,
‘public/app.routes.js’,
‘public/app.js’,
‘public/components/**/*.js’,
'public/shared/**/*.js'
])

We’re starting with all of our .module.js files (we have two: things.module.js and dashboard.module.js). Then, we need our third module, app.routes.js. Modules loaded, we can include our app.js, and then finally, whatever other component files (controllers, services) and shared files (filters, directives) we have.

Run ‘gulp’ from the terminal, and Angular should be back to normal!

Don’t forget to lint!

Linting is a practice popularized by Javascript guru and oldschool badass Douglas Crockford. Along with writing Javascript: The Good Parts and inventing JSON, it’s a pretty safe bet that if this guy made a tool, you ought to use it.

JSHint is a fork from JSLint, and as far as I can tell, the only difference is that JSHint is a little bit less opinionated. I’ve chosen to go with JSHint because it has way more community support: 150,o00 downloads in the last month compared to 1,300.

JSHint comes with different reporters, and you can define your own. Personally, after spending a few minutes looking into alternatives, I’m happy with the default for now.

Here’s our final scripts task, with the new lines in bold:

gulp.task(‘scripts’, function() {
return gulp.src([
‘public/**/*.module.js’,
‘public/app.routes.js’,
‘public/app.js’,
‘public/components/**/*.js’,
‘public/shared/**/*.js’
])
.pipe(jshint())
.pipe(jshint.reporter(‘default’))

.pipe(sourcemaps.init())
.pipe(concat(‘main.min.js’))
.pipe(uglify())
.pipe(sourcemaps.write(‘./’))
.pipe(gulp.dest(‘public/assets/js’))
.pipe(notify({ message: ‘Scripts task complete’ }));
});

Watch it!

So, let’s be real. It would be a pretty massive drag if we needed to run this task whenever we updated our CSS or modified our JS. What we want is for this to happen automatically, the way Rails does it without us even noticing.

Thankfully, Gulp provides a nice built-in way to do this. Let’s make a new task:

gulp.task(‘watch’, [‘styles’, ‘scripts’], function() {
gulp.watch(‘public/**/*.js’, [‘scripts’]);
gulp.watch(‘public/**/*.scss’, [‘styles’]);
});

gulp.task() takes three arguments. The first is the task name(my creative drought continues as I name it ‘watch’). The second is the tasks that this task depends on (think Angular Dependency Injection). Finally, the task contents.

gulp.watch takes two arguments. The first is the files we want to watch. The second is the task we want to run when those files change. Whenever a JS file gets updated, I run the ‘scripts’ task defined in the same file.

Now, we just need to add the ‘watch’ task to our default task, so it’ll start watching when we type gulp:

gulp.task(‘default’, function() {
gulp.start(‘styles’, ‘scripts’, ‘watch’);
});

And voila! Give it a shot: start gulp, and then add a comment to your app.js or whatever. Watch as the terminal comes alive and runs the appropriate task!

Error Handling

Let’s say you’re writing some SASS, and you forget a semi-colon. When you save the file, Gulp tries to process it into CSS, but it hits an error:

On the one hand, it’s good that it threw an error! On the other hand, there are two problems with this error:

  1. It doesn’t tell us what file or line the error happened on!
  2. It stops the ‘watch’ task, so we’d need to restart it every time there’s an error.

The fix for this particular problem took me hours to solve. I swapped out my sass plugin, tried using gulp-plumber, and rewrote the task a dozen different ways. In the end, the only thing I needed to do was add some error-handling to my styles task:

gulp.task(‘styles’, function() {
return gulp.src(‘public/assets/css/*.scss’)
.pipe(sourcemaps.init())
.pipe(sass())
.on(‘error’, function(err) {
console.error(err.message);
this.emit(‘end’);
})

.pipe(autoprefixer(‘last 2 version’, ‘safari 5', ‘ie 8', ‘ie 9', ‘opera 12.1'))
.pipe(minifycss())
.pipe(concat(‘style.min.css’))
.pipe(sourcemaps.write(‘./’))
.pipe(gulp.dest(‘public/assets/css/’))
.pipe(notify({ message: ‘Styles task complete’ }));
});

When we run our gulp task and remove that same semi-colon, this is what we see:

The exact file name and line is specified with the reason for the error, and Gulp continues to run.

I’m not 100% satisfied with this solution, because it’s too easy to miss the fact that the task hasn’t run. If you don’t have the terminal visible, you’d have no idea. I tried getting gulp-notify to throw up a warning, but no such luck, and I’ve spent enough time today with this.

The same line can be added to our scripts function, but for now I’ll assume that JSHint will do a good enough job catching errors.

LiveReload.

Because Cmd-R is for suckers.

If, like me, you rock a dual-monitor work setup, this next bit will be indispensable. No more alt-tabbing and refreshing just to see that 1px CSS tweak, your browser will now automagically refresh when you save your files.

There are a number of ways to get this set up, but I’ve found the most convenient to be the browser plugins. I use Chrome, and you can run and grab the extension for it here.

We also need to include the Gulp package in our Gulpfile, but if you copied my big list of dependencies above, it should already be in there. If not, just add this line:
var livereload = require(‘gulp-livereload’);

Using LiveReload is incredibly straightforward. Essentially, we want our page to refresh whenever our pre-existing CSS task runs, so let’s just add it as a new pipe. New line is bolded.

gulp.task(‘styles’, function() {
return gulp.src(‘public/assets/css/*.scss’)
.pipe(sourcemaps.init())
// some tasks excluded for brevity.
.pipe(sourcemaps.write(‘./’))
.pipe(gulp.dest(‘public/assets/css/’))
.pipe(notify({ message: ‘Styles task complete’ }));
.pipe(livereload())
});

Because we already have a watch task that runs the styles task whenever we save, no further gulping is required!

When you run Gulp now, a couple new lines are added:

[08:19:49] style.min.css.map was reloaded.
[08:19:49] gulp-notify: [Gulp notification] Styles task complete
[08:19:49] Live reload server listening on: 35729

We can see that the LiveReload process runs, by default, on port 35729. Now, you might think we need to change that to match the port our Node server is running on (I’ve chosen 3000, as a Rails tradition), but that’s where our browser extension comes in.

Yellow glow not included.

In Chrome, the plugin’s icon looks like this. Click it, and that little dot in the middle fills in.

Open your Chrome Inspector, and check out what happens when the extension is enabled:

Check it out: It opens the script located on port 35729. This is the glue between the LiveReload server started by Gulp, and the browser file reading from Node.

Amazingly, that’s all there is to it. Open up your CSS file and do something dramatic, like set the background colour to #F00. Just like that, the page becomes headache-inducing.

To Infinity, and Beyond!

By now, it should be clear that there is no shortage of amazing things Gulp can do. The big one we haven’t touched on is image optimization, but there are hundreds of ways to customize Gulp tasks to save you time and energy; we’re just scratching the surface here.

Additional Reading
The world is your oyster! Check out Gulp’s official plugin directory, and experiment with something new.

Our repository has been built-up and fleshed out throughout the full stack, we’ve got a good testing infrastructure in place, and now we’ve streamlined our development process.

The only thing left is to get this baby online! Next time, we tackle deployment with Node.

Coming soon!

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.