From Grunt to NPM — part 2

Diving into the migration details

Matti Bar-Zeev
Oct 1, 2017 · 8 min read

In Part 1 of this article we’ve talked about the motivations and challenges involved in migrating Capriza’s Designer build from Grunt to NPM.
In this part we will dive a little bit into the details of implementation to get a sense on what it actually means to migrate the build process of a matured front-end process to NPM.
All ready? here we go…

Before we start, it is important to note that this article assumes basic knowledge of how NPM handles scripts. If you feel you need to refresh on that, this following NPM documentation is a good place to start.

Some Core NPM Scripts Concepts

Like in any tool you’re using, NPM also has conventions and concepts to help us achieve our goal. Here are the ones of our main interest:

  • Life-cycle hooks — For each NPM script you can hook a “pre” and “post” script simply by adding a prefix with the same name; say you have a script called “create”, a script names “precreate” will run automatically before it. Read more about it here.

The Package.json File

The place to put our scripts is in the project’s package.json file, under the “scripts” prop. Before you jump and write your first script here is a little tip -

Not only true for NPM but goes for code as well - it is considered a good practice to break the scripts into units with a single responsibility and later combine and reuse them via the tools NPM provides us.

With that in mind lets review what we’ve done.

Running Unit Tests

The first script we’re going to talk about is “test”. The “test” script is a built-in NPM script which typically has a placeholder made for it if you create the package.json using npm init.This is how it will be written:

"test": "echo \"Error: no test specified\" && exit 1"

To implement the test according to our needs we used the karma node package. This is a good place to tell you that one of the cool things about NPM is that it knows where to look for the packages within the projects node_modules directory so it’s pretty straight forward with no long paths. Our test script looks like this:

"test": "karma start"

NPM will take it from there, given that you have a karma.conf.js file on the project root directory, your tests will start running.
We also wanted to generate a coverage report and for that we’ve used Karma as well with an additionalset of parameters:

"coverage": "karma start --single-run --reporters junit,coverage"

Here we’re running a single run that generates 2 types of reports: junit and coverage (both are Karma’s plugins).
All the parameters you see in the script above can be found under Karma’s docs. Nothing hiding in a 3rd party mediator. Since the “coverage” script is not a built-in NPM script, in order to trigger it you need to use the “run-script” or in short just “run”, like so:

npm run coverage

Code Manipulations

Usually you would like to concatenate the project code and maybe uglify it in order to prepare it for production. The Grunt plugin used for these purposes was grunt-usemin which internally utilizes several node packages.
This process is made of two core phases — One is concatenating JS and CSS “blocks” from target HTML files while the other is generating new HTML files which have reference to these newly concatenated files.

This is all great but we wanted more control over the process. We found out that if we extract this process to a custom node script which uses usemin package (among others) we are not bound to support just “js” or “css” block types but rather have custom blocks like “remove” for blocks that shouldn’t be included on the final result.

Our first move was to create a usemin-script.js and start constructing what we exactly want this process to do. Within this script we concatenate the JS code, concatenate the CSS, clean and minify it and finally remove code we do not need in production, like livereload scripts.

But having a node script allowed us to do even more — we can actually control how each part of these manipulations works. For instance we can set exactly what kind for compression the UglifyJS library will do (read more about the options here) or how usemin.getHtml() method will generate our final HTML file (minified or not? remove empty attributes? remove comments?).

We were no longer chained into the configurations the Grunt plugin provided us. If we would like to remove one phase (like the uglification and we shall soon see why) or add another processing into it, it’s pretty straight forward.
Eventually, NPM script only calls our usemin-script.js node script with a parameter for the target HTML file path and takes it from there:

"usemin:designer": "node build/scripts/usemin-script.js main.html"

Something to note about uglification — As mentioned above, as part of the usemin package there is also an option to uglify the code (using UglifyJS). After several comparisons we’ve made we noticed that when using gZip compression, the difference between the uglified compressed code and the gZipped code does not justify the maintenance around source-mappings (for debugging usage), and since we don’t consider uglifying obfuscation a sort of security we chose to avoid uglification.

Pre-Compiling Templates

As mentioned in Part 1 of this article, Capriza’s Designer utilized Handlebars templates to render part of its DOM. Handlebars documentation is pretty straight forward, but the Grunt plugin adds some additional “features” over it, which can lead into misuse of Handlebars and add (even more) complexity to the configuration since the parameters names are not quite the same as with Handlebars. This is a classic case where a Grunt mediator obscures the vendor implementation from us, thus makes it very hard to maintain later on.

Moving to NPM scripts using the Handlebars pre-compiler package directly we turned something which looked like this:

into this simple NPM script (excuse medium for wrapping it):

"templates:designer": "handlebars templates --output templates/output/compiled-templates.js --namespace Handlebars.templates"

I will explain what it does though I think that it speaks for itself — take the template files under the “templates” directory, compile them with the “Handlebars.templates” namespace and dump the result into the compiled-templates.js files.


And while the rest of the work follows the same implementation concepts, there are two more scripts worth further elaboration:

Amazon S3 deployment

Capriza’s Designer versioned artifact can be deployed to Amazon’s S3 to later be fetched and used. In our day-to-day work we use this task to upload a new version so that our automation tests will run against a specific version for regressions.
As we’ve mentioned earlier, one of the great benefits NPM script gives you is the ability to also run NodeJS scripts, thus allowing a greater flexibility in what can be done.

Amazon S3

We have created a custom node script which utilizes the S3 Node package, and the git-user package to extract user data we need for deployment parameters (such as the unique version name we’re giving the newly deployed artifact) and deploy the artifact.
We also use the Progress package to display the upload progress and attach it to the progress event the S3 package has, just to be nicer to the ones using it :)

In the end, when we run npm run deploy our script builds the Designer artifact (on the “predeploy” hook) and then runs the deploy node script which does the rest. Here is how it looks while uploading:

Uploading the Designer artifact to S3

Triggering a Jenkins Job

In general, we don’t just deploy the Designer artifact; we also trigger a Jenkins job which runs our automation tests.
Capriza uses an in-house tool called “Watchdog” (which deserves a post of its own) to run the Designer’s automation tests. The process is comprised of building the Designer artifact, deploying it and then triggering a Jenkins job which runs the tests against the uploaded version.

As you can probably guess, this script is dependent on the “deploy” script — when it’s done, it takes the deployed version and passes it to the Jenkins job as a parameter. The Jenkins job triggering-script is also a NodeJS script which uses the Jenkins node package. God, you can find anything you can think of in NPM!.

But how do we pass the deployed version to the this script?

Jenkins

For this we use Bash’s piping feature like so:
The deploy script writes the deployed version to process.stdout when the upload completes, e.g.: process.stdout.write(deployedVer). The jenkins-script listens to the stdin to get that info, like so:

process.stdin.on('data', (data) => {     deployedVersion = data;});

And the way we register it under package.json is like so (notice we’re using the pipe operator):

"watchdog": "npm run deploy | node build/scripts/jenkins-script.js"

Pretty neat, right? When we run npm run watchdog we get it all “magically” done. Now you can see how all the building blocks come together. That’s some powerful stuff we’re enjoying right there.


And so…

Our journey to replace Grunt with NPM has completed. We’ve managed to migrate all our Grunt’s responsibilities to NPM, removing all of Grunts dependencies from our code and simplify the way we work and maintain our Designer product.
While this is not the last step in this path, taking it helped us to get more agile and flexible to changes and progress.

Capriza Engineering

Capriza engineering blog focused on technology, interesting challenges and related topics

Matti Bar-Zeev

Written by

Mind reflections & individual thought patterns of one code musician ▪ Believes in craftsmanship ▪ Culture over Technology ▪ Always learning...

Capriza Engineering

Capriza engineering blog focused on technology, interesting challenges and related topics

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade