NPM Task Running Techniques
Task running — it’s not sexy, but it makes the world go ‘round. — Madonna
Updated 11 Oct 2017: Thanks to Alireza Motevallian and Toru Nagashima for their feedback on handling never-ending processes. I’ve updated the last section accordingly.
Background
As a software engineer, I use software to automate boring tasks. If I can’t find an existing software tool to do what I need, I can write my own. In the NodeJS ecosystem, this is as simple as writing a JavaScript file and then running it from a terminal window: node my-file.js
.
As I’ve automated more tasks, I’ve found I need to organise tasks into sequences to accomplish my overall-task. A certain kind of software called a task runner allows me to do this efficiently. A few years ago my chosen task runner was GruntJS. But for the last few years I’ve used an NPM-like task runner because it is simpler than Grunt and easier to maintain.
npm-run-all
In case you have not come across it, allow me to introduce npm-run-all
— a powerful task-runner for NodeJS. I use it only within my package.json
file to compose smaller tasks (implemented via “scripts” inside package.json
) into larger ones. (See the CLI documentation for more information). This avoids creating yet-another-config file and leverages the existing capability of npm
to run tasks.
Running tasks in series
In the following example, calling script A
(via npm run A
) would cause npm-run-all
to call script B
and script C
.
"scripts": {
"A": "npm-run-all B C",
"B": "rimraf",
"C": "eslint"
}
Note that you can achieve the same result by changing A
to "npm run B && npm run C"
. But npm-run-all
allows you to compose tasks in a platform-independent manner .
The tasks above are run in series. When B
is finished (with a non-zero exit code), only then will C
execute. This is useful for tasks that are dependent on each other. For example, you may want to publish your NPM module only when your code passes all tests and builds without error:
"scripts": {
"release": "npm-run-all build test pub",
"build": "webpack --prod",
"test": "mocha",
"pub": "npm publish"
}
If any error occurs in a task, subsequent tasks will not run.
Running tasks in series with a shortcut
If we re-write our task names slightly, we can simplify this configuration further:
"scripts": {
"test": "npm-run-all test:*",
"test:unit": "mocha",
"test:integration": "jasmine",
"test:system": "codecept"
}
Running npm run test
would execute test:unit
, test:integration
, and test:system
, in that order.
Running tasks in parallel
Most task runners allow you to run tasks in parallel. This is useful when you have tasks that are independent. For example:
"scripts": {
"all": "npm-run-all -p lint test",
"lint": "eslint",
"test": "mocha"
}
The -p
flag is annpm-run-all
flag that means “run the following tasks in parallel: lint
& test
, then exit when both tasks have completed”.
npm-run-all
allows us to run a combination of tasks in parallel and in sequence:
"scripts": {
"all": "npm-run-all build -p lint test -s pub",
"build": "webpack",
"lint": "eslint",
"test": "mocha",
"pub": "npm publish"
}
The all
script runs build
, then lint
& test
in parallel until both tasks are complete, then pub
. The -s
flag means “run the following tasks in series”, and is only needed after using the -p
flag.
Running never-ending tasks
Simple case
So we can run tasks in parallel, in series, and with a combination of these approaches. But what about when you need to run a task that is designed to run forever, such as a web-server?
Let’s see what happens when we apply our existing techniques to try to solve this problem:
"scripts": {
"all": "npm-run-all build serve",
"build": "webpack",
"serve": "serve -p 8000"
}
Running all
would work beautifully, as the serve
task runs last (after the build
task) and it runs forever. Solved!
Complex case
Now what if you want to do something after the serve
task, such as testing the fully-running application?
"scripts": {
"test:system": "npm-run-all build serve test",
"build": "webpack",
"serve": "serve -p 8000",
"test": "codecept",
}
In the above example, the test
task would never execute as the serve
task runs forever. We could try changing the all
script to "npm-run-all build -p serve test"
and that would execute the test
script at the same time as serve
. But if the purpose of the task was to test the system and exit after the tests have finished, then this is not a solution since serve
would continue to run even after test
had finished.
Complex case — solution
npm-run-all
has another trick up it’s proverbial sleeve: the -r
(race) flag. Before any SJWs, activists, or other people with an axe-to-grind start hounding the developers of npm-run-all
, let me explain…
The -r
flag tells npm-run-all
to run the specified tasks in parallel (you can only use -r
with -p
), but when one task exits, all the other parallel tasks exit too. In other words, the tasks are in a race (competition); when one task finishes, the competition itself finishes.
This feature allows us to run any-number of tasks in parallel and exit when one of those tasks finishes:
"scripts": {
"test:system": "npm-run-all build -p -r serve test",
"build": "webpack",
"serve": "serve -p 8000",
"test": "codecept",
}
One more thing…
In the example above, imagine that the serve
task takes 10 seconds before the server is listening on a port. The test
task may try for 5 seconds to connect to the server in order to run the tests. In this scenario, the first few tests may timeout until the server is running. What do you do?
Option 1 — Sleep
This is the approach I initially tried:
"scripts": {
"test:system": "npm-run-all build -p -r serve test",
"build": "webpack",
"serve": "serve -p 8000",
"test": "sleep 10 && npm run test:codecept",
"test:codecept": "codecept",
}
This is dependent on speed of the hardware that the tasks are running on. It might take 10 seconds for you, but maybe it takes 12 seconds for someone else’s machine. So it’s a brittle approach.
Option 2 — Write a Node script
See Ali’s example for more information. It requires a lot more work than Option 1, but it is not as brittle either.
Option 3 — Use wait-on
Thanks to Toru Nagashima for his suggestion to use the wait-on NPM package! From their docs:
wait-on is a cross-platform command line utility which will wait for files, ports, sockets, and http(s) resources to become available…
This allows us to write the following:
"scripts": {
"test:system": "npm-run-all build -p -r serve test",
"build": "webpack",
"serve": "serve -p 8000",
"test": "wait-on http://localhost:8001 && npm run test:codecept",
"test:codecept": "codecept",
}
Very nice!
I may be British, but my favourite race is npm-run-all -p -r
— Lewis Hamilton, F1 driver, circa 2015