HOW TO: Build your own task runner with es6 and node

Fed up with flavor of the month task runner? Build your own!


It’s been a while. This could be a long one!


I’ve written previously about the topic of running tasks for front end source. I’ve looked at popular task runners such as Gulp and also alternative solutions such as npm run-scripts and Make.

“I’m fed up with task runner X, I’m just using node scripts” — someone, somewhere… likely

I’ll disclaim now that if you have a build set up that works well for you. I’m by no means suggesting that you throw that out and rewrite the whole thing in node scripts.

For those in camp TL;DR, you can check out a “node script” runner I put together as a result of exploring my own solution here.


If you’ve decided to stick around, let’s start exploring creating our very own task runner.

Motivation

There are various options when it comes to running tasks. Use task runner X or Y, use npm run-scripts, use Make etc.

To be honest, after trying out different things, I thought “why not explore creating my own CLI task runner with node?”

How hard can it really be to create a node CLI tool that will run node scripts for me? Can it be simple enough to do that whilst giving me some nice to haves?

Desired features

Based on the experience of using other solutions, I came up with a list of features I’d like for my task runner to have and things I’d like my task runner to be;

  • It just does what you tell it. Nothing extra.
  • Is it even a task runner? It’s more like a node scripts runner. I don’t want it to do anymore than it has to. Take care of watching and writing to the filesystem within tasks.
  • A specific file for defining things like package/plugin options and source paths, a runtime configuration file.
  • A specific folder for storing task files.
  • Pre and post hooks for tasks that work recursively.
  • Use npm modules directly so you never have the issue of being tied to a task runner specific ecosystem.
  • Ability to run tasks in sequence or concurrently.
  • Ability to pass runtime variables to script runner.
  • Task profiling so I can see how long things are taking.
  • Self documented tasks so that I don’t have to crawl through task code to work out what’s going on in certain tasks. I should be able to see what tasks I have and what understand what they do without opening any task source.

Design

The important part about creating our own task runner is making it easy to use. Design is crucial to how it works. There’s room for improvement, but this was my first take at the design.

CLI

The CLI is going to be pretty simple. We want to be able to do things like;

$ task-runner task:a
$ task-runner task:a task:b
$ task-runner task:a --env dist

In order;

  1. Run “task:a”.
  2. Run “task:a” and “task:b” concurrently.
  3. Run “task:a” with environment variable as “dist”.

But how about self documentation? Some popular task runners have a default task feature. For the first iteration, I’m avoiding the default task. My default task will be to show info about the available tasks;

$ task-runner

Could give me the following output;

Available tasks to run:
  task:a  -  does some magical stuff to get the project running
task:b - does some other stuff, not so magical I'm afraid

Maybe a different approach could be to actually implement the default task as people are familiar with that and use an option to show available tasks like Gulp does with “-T”. Maybe something like;

$ task-runner --info

Tasks

Tasks are going to be node scripts that will be consumed by our CLI tool. I want to keep things minimal. It’s important to make it easy to migrate from using any current node scripts and vice-versa if we decide the script runner isn’t for us. Based on this and our desired features list, my first idea was to expose objects or an array of objects from task files. This is so that we can have “one to many” tasks defined per file and keep a good separation of concerns. The initial design for tasks is as follows;

module.exports = {
name: 'task:a',
doc : 'some task that does something',
pre : 'task:b',
post: 'task:b',
deps: [
'fs',
'stylus'
],
func: (fs, stylus, runner) => {
/* Do stuff and resolve/reject a promise */
}
};

Most of the object keys/properties are pretty self-explanatory. “name” and “doc” name our task and give it a description respectively. “pre/post” defines a pre/post hook where a defined task will be ran either before or after the task we are looking to run.

The “deps” key is where it gets a little different. One factor in the design was to try and reduce the need to have to write the following multiple times;

const somePackage = require('somePackage');

My initial idea was to define an array of dependencies that are then passed to the task “func” function. Maybe a better solution could be something like Gulp’s “gulp-load-plugins” that exposes an object for us to access each module we desire? This can be simply changed. The only issue is whereby something like “gulp-load-plugins” looks for Gulp ecosystem packages, how do we filter out the things we actually need? Maybe define it in our runtime configuration file?

Lastly, let’s take a closer look at the “func” property. This defines a function for our task logic.

func: (fs, stylus, runner) => {}

There’s an extra parameter! “runner” isn’t one of our dependencies. That’s on purpose. This is a special object that exposes elements of our task runner to our task logic. Why? The idea behind the design is to rely quite heavily on Promises so that we can have tighter control over when things finish and when other things start. When we have a Promise, we need to able to resolve or reject it and this needs to happen within our task logic;

func: (fs, package, runner) => {
var result = package.compile('src/scripts/**/*.*');
if (!result) runner.reject('Something went horribly wrong');
runner.resolve();
}

What else can we pass in this object?

How about the content of our runtime configuration file? In here we can store things like;

module.exports = {
sources: {
scripts: 'src/scripts/**/*.*
}
};

Instead of repeatedly writing our source path strings we access them with the “runner” param like so;

func: (fs, package, runner) => {
var result = package.compile(runner.config.sources.scripts);
if (!result) runner.reject('Something went horribly wrong');
runner.resolve();
}

And logging. It felt right to also expose the runners logging instance to be used within tasks;

func: (fs, package, runner) => {
var result = package.compile('src/scripts/**/*.*');
if (!result) runner.reject('Something went horribly wrong');
runner.log.success('Finished');
runner.resolve();
}

We can also pass the any environment variables that have been set.

Lastly and importantly, we need to pass the ability to tell our task runner instance to run a task. This is particularly useful when doing things like watching for file changes and triggering tasks. We can do this by passing a bound instance of our task runner functionality(examples can be seen later on)


Defining task sequences and concurrent tasks should be simple. By default, calling the task runner with multiple arguments will run those tasks concurrently without profiling;

$ task-runner task:a task:b

But to define a sequence or set of concurrent tasks will be along the lines of the following;

module.exports = {
name: 'sequence',
doc : 'runs two tasks in sequence',
sequence: [
'task:a',
'task:b'
]
};

or

module.exports = {
name: 'concurrent',
doc : 'runs two tasks concurrently',
concurrent: [
'task:a',
'task:b'
]
};

Implementing a solution

Enough of the design, let’s start putting something together. For implementation I decided to go with Babel. I also decided on using Commander, colors and winston for the task runner. Commander is an ideal solution for handling the CLI and winston is a great logging solution that also offers built in profiling. The colors package made sense to give our logging some pretty colors :D

For the purpose of trying to keep this post from becoming exceptionally large, the presumption is that you know how to create a basic CLI application in node and that you’re familiar with the commander package. I’m also hoping that you’re familiar with things like Promises. I’m going to primarily focus on the structure of our task runner code and how the features from es6 have been used. I’m also going to forego writing about setting up a custom instance of winston for our task runner. For those wanting to dive into some source, all the source can be see here.


Main

The main entry point to our task runner will be concerned with consuming our command line arguments and setting up a task runner instance. It will then decide whether to tell the instance to show task documentation or to actually run a specific task(alternatively, we might throw an error if there are no tasks or runtime configuration file available).

Structure

Referring to our main entry point, it seems our task runner is an ideal opportunity to make use of es6 classes. We can have a class for our task runner instance and a class for our task runner tasks.

Promises

Mentioned briefly earlier, Promises play a large part in our task runner’s implementation. Providing that tasks resolve or reject Promises we can hook into when a task truly finishes. With that being said, it’s true to say that without Promises our task runner would not be that useful and only really be capable of running a task at a time.

Code

Let’s dive into some code(you can check out the full source here). We will take a closer look at our task runner “instance” and “task” classes. I don’t want it to be all code, so I’ll just highlight the workings and give snippets where appropriate.

Task

Let’s start with the implementation of our “Task” class. This is the smaller of the two classes. Below, is the full source code commented where appropriate;

class Task {
constructor(instance, opts) {
Object.assign(this, opts, { instance });
if (!this.instance) throw Error(‘Missing instance definition…’);
/**
* If the task is dependant on modules and these are declared in
* the task definition, require each and push into an Array to be
* used at run time
*/
const hasFunc = (this.func && typeof this.func === ‘function’) || this.sequence || this.concurrent;
if (!this.name || !this.doc || !hasFunc)
throw Error(‘Task options missing properties…’);
this.deps = [];
if (opts.deps && opts.deps.length)
for (const dep of opts.deps) {
let module;
try {
module = require(dep);
} catch (err) {}
try {
const path = `${process.cwd()}/node_modules/${dep}`;
module = require(path);
} catch (err) {}
if (!module) throw Error(`Module ${dep} not found, installed?`);
this.deps.push(module);
}
}
/**
* runs task
* @return {Promise} — Promise that informs Instance of
* task outcome.
*/

run() {
/**
* Returns a Promise so that runner knows to start on the next
* task. Be mindful that a watching task will never resolve so
* if we wish to run more than one watch, we must use the
* “concurrent” option.
*/

return new Promise((resolve, reject) => {
/* give user feedback so they know which task is running */
winston.info(`Running ${this.name}`);
/**
* profile the running time using winston so user can see
* which tasks may be underperforming or should be
* refactored
*/
winston.profile(this.name);
/**
* run function
* passing the task dependencies(an Array of required
* modules) and a reference Object that contains the
* resolve/reject for the Promise in addition to any
* configuration defined within runtime config file,
* environment and
* the russInstance run. “run” is important for when we have
* a watcher that wishes to run another task or wish to run
* a task within a task.
*/
this.func(…this.deps, {
__instance: this.instance,
env : this.instance.env,
config : this.instance.config,
log : winston,
resolve : resolve,
reject : reject,
run : this.instance.runTask.bind(this.instance)
});
});
}}

Our task implementation is pretty simple. The notable points though are how we pull in dependency packages for our task and how the “run” function is defined for our task instance.


Task instances are created with task objects. Remember the task object that was mentioned previously above?(Note:: we don’t create instances for sequences and concurrent tasks)

I’m not 100% on how I’ve gone about pulling in dependencies but a couple of try/catches was my first attempt. Essentially, we look to loop through the “deps” array that is passed to our “Task”. Then we “require” each dependency so that we can pass them through to our “run” function when the time comes.

Our “run” function will return a Promise and pass through the resolve/reject parameters as part of our special object to our task logic(note the use of the spread operator to pass our dependencies as separate parameters to the run function);

run() {
/**
* Returns a Promise so that runner knows to start on the next
* task. Be mindful that a watching task will never resolve so
* if we wish to run more than one watch, we must use the
* “concurrent” option.
*/

return new Promise((resolve, reject) => {
/* give user feedback so they know which task is running */
winston.info(`Running ${this.name}`);
/**
* profile the running time using winston so user can see
* which tasks may be underperforming or should be
* refactored
*/
winston.profile(this.name);
/**
* run function
* passing the task dependencies(an Array of required
* modules) and a reference Object that contains the
* resolve/reject for the Promise in addition to any
* configuration defined within runtime config file,
* environment and
* the russInstance run. “run” is important for when we have
* a watcher that wishes to run another task or wish to run
* a task within a task.
*/
this.func(…this.deps, {
__instance: this.instance,
env : this.instance.env,
config : this.instance.config,
log : winston,
resolve : resolve,
reject : reject,
run : this.instance.runTask.bind(this.instance)
});
});

If our task logic does not invoke the resolve/reject provided by our Promise, this will unfortunately break the intended behaviour of our task runner.

That’s pretty much it for our Task class.

Instance

Our “Instance” class is a bit more complex.

When we create an “Instance” we check that there is a runtime configuration file available and also that there are some tasks available. If there isn’t we throw an error.

Apart from general instantiation, the most important thing is to register our tasks with our instance.

We base our registration off the exported Arrays/Objects from our task files;

/**
* populates instance tasks
* @param files {Array} array of task objects to use when
* registering
*/

register(files) {
const registerTask = (opts) => {
const ERR_MSG = `Task ${opts.name} missing properties…`;
const hasFunc = opts.func && typeof opts.func === ‘function’;
const isDel = opts.concurrent || opts.sequence;
if (opts.name && opts.doc && (hasFunc || isDel))
this.tasks[opts.name] = opts;
else throw new Error(ERR_MSG);
};
if (files.length === 0) throw new Error(‘No tasks defined’);
this.tasks = {};
for (const file of files) {
const taskOpts = require(`${process.cwd()}/tasks/${file}`);
if (this.tasks[taskOpts.name]) throw new Error(`Task ${taskOpts.name} already defined…`);
if (Array.isArray(taskOpts))
taskOpts.map(registerTask);
else
registerTask(taskOpts);
}
}

Nothing too scary in there.

Once our “Instance” has registered, there are two things it can do. It can either display info about available tasks(self-documentation) or it can run a task.

Let’s start by having a look at our documenting function. The aim of the “info” function is to display a list of available tasks and the content of their respective “doc” properties;

info() {
const header = ‘Available task to run:\n’;
let taskList = ‘\n’;
if (Object.keys(this.tasks).length > 0)
for (const task in this.tasks)
taskList += ` ${task.green}: ${this.tasks[task].doc.cyan}\n`;
winston.info(`${header}${taskList}`);
}

It’s pretty lightweight and making use of template literals we generate a documentation string that is displayed on the command line.


Here comes the big one… It’s not too bad I Promise.

Running tasks isn’t that overly complicated in concept. Essentially it works as so;

  1. Check if given task name is registered, if it is, proceed.
  2. Generate the pool of tasks that need to be run for a given task. This is recursive. This means checking for all pre/post tasks of tasks and also iterating through tasks listed in a sequence or concurrent array.
  3. If our task is a sequence or concurrent set then start the profiler for this.
  4. Begin running tasks. Using Promises, we know when to begin the next task in the pool of tasks.
  5. As tasks complete and when all tasks are complete give user feedback so they know their tasks are complete and also give them profiling information so they know how long tasks took.

Generating the pool of tasks is one of the slightly trickier parts. We need to ensure we don’t push duplicates into the task pool and also that we populate the pool correctly differentiating from tasks and sequential/concurrent tasks.

getPool(name) {
const pool = [];
const pushToPool = (name, parent) => {
if (pool.indexOf(name) !== -1) return;
const task = this.tasks[name];
if (!task) throw Error(`Task ${name} is not defined…`);
const clean = (a) => { return a && (pool.indexOf(a) === -1); };
let tasks;
if (task.sequence)
tasks = task.sequence.filter(clean);
else if (!task.concurrent)
tasks = [task.pre, task.post].filter(clean);
if (parent) {
const parentTask = this.tasks[parent];
const pIdx = pool.indexOf(parent);
const idx = (name === parentTask.pre) ? pIdx : (pIdx + 1);
pool.splice(idx, 0, name);
} else if (!task.sequence) pool.push(name);
const newParent = (task.sequence) ? undefined : name;
if (tasks && tasks.length > 0)
for (const t of tasks) pushToPool(t, newParent);
};
pushToPool(name);
return pool;
}

The last part to explain is when we actually get down to running our task functions. It’s here that we instantiate “Task” instances and invoke their task logic.

The trick to making this work was to use an iterator over our task pool. Each time we finish a task, we check to see if there is something next in the task pool. If there is, we run it. If there isn’t we finish. This gets a little trickier with concurrent tasks. This is where “Promise.all” comes to the rescue and lets us know when all concurrent tasks have finished executing.

run(taskPool) {
const tasks = taskPool[Symbol.iterator]();
const exec = (name, resolve, reject) => {
const cb = () => {
winston.profile(name);
if (tasks.next) {
const nextTask = tasks.next();
if (!nextTask.done)
exec(nextTask.value, resolve, reject);
else
resolve();
}
};
const errCb = (err) => {
if (err) winston.error(`Error: ${err}`);
reject(err);
};
try {
const taskObj = this.tasks[name];
if (taskObj.concurrent) {
const tasks = taskObj.concurrent.map(this.runTask.bind(this));
Promise.all(tasks)
.then(cb)
.catch(errCb);
} else {
let task;
try {
task = new Task(this, this.tasks[name]);
} catch (err) {
winston.error(err.toString());
reject(err.toString());
}
task.run(this.env)
.then(cb)
.catch(errCb);
}
} catch (err) {
reject(err);
}
};
return new Promise((resolve, reject) => {
exec(tasks.next().value, resolve, reject);
});
}

The source might be a little daunting to start with but once you are OK with the Promise inception and you’ve been looking at it a little, it kinda makes sense.


That’s it. With not an enormous amount of code and leveraging the use of Promises and Iterators we can create our very own “script” runner in node. Full control over what’s happening and how it works.

To conclude

We’ve explored creating our very own task/script runner in node leveraging es6 features with Babel.

Would I use it? Yeah sure.

I think it can work pretty well if you don’t intend on swapping out things in your build stack frequently. The reason I say this is that there is a little more overhead when creating your tasks initially. This will be due to the nature of how different packages will behave differently. This is the price for being able to use packages directly and not be tied into a specific ecosystem.

For example; let’s consider a task I put together for compiling JavaScript with source mapping.

{
name: ‘compile:scripts’,
doc : ‘compiles runtime JavaScript files’,
deps: [
‘fs’,
‘path’,
‘glob’,
‘mkdirp’,
‘uglify-js’
],
func: function(fs, path, glob, mkdirp, uglify, russ) {
const isDist = russ.env === ‘dist’;
const outputDir = russ.config.paths.destinations.scripts;
const pushToServe = (filePath) => {
mkdirp.sync(`${outputDir}src/js`);
const content = fs.readFileSync(filePath);
fs.writeFileSync(`${outputDir}${filePath}`, content);
}; mkdirp.sync(outputDir);
glob(russ.config.paths.sources.scripts, (err, files) => {
if (!isDist) files.map(pushToServe);
const res = uglify.minify(files, {
outSourceMap: (!isDist) ? ‘source.js.map’ : null,
wrap: ‘russ-recipes’,
beautify: true
});
fs.writeFileSync(‘public/js/scripts.js’, res.code);
if (!isDist) fs.writeFileSync(‘public/js/source.js.map’, res.map);
russ.resolve();
});
}
}

There’s typically more then you’d need to write for some popular task runners. But. We have full control over what is happening. Let me just say, it also runs very fast from my experience so far.

How about a watching task? This makes use of our bound “run” function to tell our task runner instance to run a task by name.

{
name: ‘watch:scripts’,
doc: ‘watch for script source changes then run and compile’,
deps: [ ‘gaze’ ],
func: function(gaze, russ) {
gaze(russ.config.paths.sources.scripts, (err, watcher) => {
watcher.on(‘changed’, (filepath) => {
russ.log.info(`${filepath} changed!`);
russ.run(‘compile:scripts’);
});
});
}
}

How about generic dev tasks that will run a magnitude of things?

module.exports = [
{
name: ‘compile’,
doc : ‘compiles sources’,
concurrent: [
‘compile:styles’,
‘compile:scripts’,
‘compile:markup’
]
},
{
name: ‘watch’,
doc: ‘watch files and do things’,
concurrent: [
‘watch:scripts’,
‘watch:styles’,
‘watch:markup’
]
},
{
name: ‘develop’,
doc: ‘lets develop’,
concurrent: [
‘watch’,
‘server’
]
}
];

You can see some more examples here.


There are probably some things in the implementation that could be improved and I’d fully welcome some suggestions or alterations to the way in which it works. I’ve put together a repo here.

From my own experience using the script runner so far, it’s been pretty quick. After the initial overhead of creating tasks, for me it’s been noticeably fast.

I think the best step for reducing the task creation overhead would be to most likely share “recipes” for using it. Someone needs a task for compiling and saving some Stylus source? Here, someone wrote it already.

That’s it!

If you’ve got this far…

Well done!

Hopefully it’s been interesting. I found it interesting to explore creating my own task runner for sure.

If you want to check out the full code or where it’s got to, see here.

As always, any questions, suggestions, or if I’ve missed something out, feel free to leave a note or tweet me @_jh3y!