NPM Task Running Techniques

Published in

Netscape

5 min readOct 4, 2017

Task running — it’s not sexy, but it makes the world go ‘round. — Madonna

Updated 11 Oct 2017: Thanks to Alireza Motevallian and Toru Nagashima for their feedback on handling never-ending processes. I’ve updated the last section accordingly.

Background

As a software engineer, I use software to automate boring tasks. If I can’t find an existing software tool to do what I need, I can write my own. In the NodeJS ecosystem, this is as simple as writing a JavaScript file and then running it from a terminal window: node my-file.js.

As I’ve automated more tasks, I’ve found I need to organise tasks into sequences to accomplish my overall-task. A certain kind of software called a task runner allows me to do this efficiently. A few years ago my chosen task runner was GruntJS. But for the last few years I’ve used an NPM-like task runner because it is simpler than Grunt and easier to maintain.

`npm-run-all`

In case you have not come across it, allow me to introduce npm-run-all — a powerful task-runner for NodeJS. I use it only within my package.json file to compose smaller tasks (implemented via “scripts” inside package.json) into larger ones. (See the CLI documentation for more information). This avoids creating yet-another-config file and leverages the existing capability of npm to run tasks.

Running tasks in series

In the following example, calling script A (via npm run A) would cause npm-run-all to call script B and script C.

"scripts": {
  "A": "npm-run-all B C",
  "B": "rimraf",
  "C": "eslint"
}

Note that you can achieve the same result by changing A to "npm run B && npm run C" . But npm-run-all allows you to compose tasks in a platform-independent manner .

The tasks above are run in series. When B is finished (with a non-zero exit code), only then will C execute. This is useful for tasks that are dependent on each other. For example, you may want to publish your NPM module only when your code passes all tests and builds without error:

"scripts": {
  "release": "npm-run-all build test pub",
  "build": "webpack --prod",
  "test": "mocha",
  "pub": "npm publish"
}

If any error occurs in a task, subsequent tasks will not run.

Running tasks in series with a shortcut

If we re-write our task names slightly, we can simplify this configuration further:

"scripts": {
  "test": "npm-run-all test:*",
  "test:unit": "mocha",
  "test:integration": "jasmine",
  "test:system": "codecept"
}

Running npm run test would execute test:unit, test:integration, and test:system, in that order.

Running tasks in parallel

Most task runners allow you to run tasks in parallel. This is useful when you have tasks that are independent. For example:

"scripts": {
  "all": "npm-run-all -p lint test",
  "lint": "eslint",
  "test": "mocha"
}

The -p flag is annpm-run-all flag that means “run the following tasks in parallel: lint & test, then exit when both tasks have completed”.

npm-run-all allows us to run a combination of tasks in parallel and in sequence:

"scripts": {
  "all": "npm-run-all build -p lint test -s pub",
  "build": "webpack",
  "lint": "eslint",
  "test": "mocha",
  "pub": "npm publish"
}

The all script runs build, then lint & test in parallel until both tasks are complete, then pub. The -s flag means “run the following tasks in series”, and is only needed after using the -p flag.

Running never-ending tasks

Simple case

So we can run tasks in parallel, in series, and with a combination of these approaches. But what about when you need to run a task that is designed to run forever, such as a web-server?

Let’s see what happens when we apply our existing techniques to try to solve this problem:

"scripts": {
  "all": "npm-run-all build serve",
  "build": "webpack",
  "serve": "serve -p 8000"
}

Running all would work beautifully, as the serve task runs last (after the build task) and it runs forever. Solved!

Complex case

Now what if you want to do something after the serve task, such as testing the fully-running application?

"scripts": {
  "test:system": "npm-run-all build serve test",
  "build": "webpack",
  "serve": "serve -p 8000",
  "test": "codecept",
}

In the above example, the test task would never execute as the serve task runs forever. We could try changing the all script to "npm-run-all build -p serve test" and that would execute the test script at the same time as serve. But if the purpose of the task was to test the system and exit after the tests have finished, then this is not a solution since serve would continue to run even after test had finished.

Complex case — solution

npm-run-all has another trick up it’s proverbial sleeve: the -r (race) flag. Before any SJWs, activists, or other people with an axe-to-grind start hounding the developers of npm-run-all, let me explain…

The -r flag tells npm-run-all to run the specified tasks in parallel (you can only use -r with -p), but when one task exits, all the other parallel tasks exit too. In other words, the tasks are in a race (competition); when one task finishes, the competition itself finishes.

This feature allows us to run any-number of tasks in parallel and exit when one of those tasks finishes:

"scripts": {
  "test:system": "npm-run-all build -p -r serve test",
  "build": "webpack",
  "serve": "serve -p 8000",
  "test": "codecept",
}

One more thing…

In the example above, imagine that the serve task takes 10 seconds before the server is listening on a port. The test task may try for 5 seconds to connect to the server in order to run the tests. In this scenario, the first few tests may timeout until the server is running. What do you do?

Option 1 — Sleep

This is the approach I initially tried:

"scripts": {
  "test:system": "npm-run-all build -p -r serve test",
  "build": "webpack",
  "serve": "serve -p 8000",
  "test": "sleep 10 && npm run test:codecept",
  "test:codecept": "codecept",
}

This is dependent on speed of the hardware that the tasks are running on. It might take 10 seconds for you, but maybe it takes 12 seconds for someone else’s machine. So it’s a brittle approach.

Option 2 — Write a Node script

See Ali’s example for more information. It requires a lot more work than Option 1, but it is not as brittle either.

Option 3 — Use wait-on

Thanks to Toru Nagashima for his suggestion to use the wait-on NPM package! From their docs:

wait-on is a cross-platform command line utility which will wait for files, ports, sockets, and http(s) resources to become available…

This allows us to write the following:

"scripts": {
  "test:system": "npm-run-all build -p -r serve test",
  "build": "webpack",
  "serve": "serve -p 8000",
  "test": "wait-on http://localhost:8001 && npm run test:codecept",
  "test:codecept": "codecept",
}

Very nice!

I may be British, but my favourite race is npm-run-all -p -r — Lewis Hamilton, F1 driver, circa 2015