Be a good dev: Offer a dependency.

Two problems of Node.js recently captured my attention.

1. It is easy to find implementations in NPM but hard to find solutions.
   For example: It is easy to find a yaml parser but it is hard to load any given structured file.
2. If you find a solution, it might have a lot of dependencies.
   This makes the installation heavy and slow.

This post looks into a new option of how make developing easier for everybody.

TL;DR: If your package becomes to big and has too many dependencies you might consider using PackageMissingError or require-implementation to improve the life of the users.

Coding for Solutions

It is easy to implement a given algorithm like “parse yaml file”.
There is a file format called YAML and file formats are meant to be parsed.
Looking for a package that parses YAML is straightforward with NPM.

However: Usually that is not the task given.

The task at hand might be “load config file” or “import data that was exported”.

Now, lets continue by imagining that you had to write a package that does “load a configuration”.

The Task: Implement “load-config”

Which config format would you choose? JSON? YAML? CSON? .properties? INI? XML? PLIST? .CFG?
Some of those come with a long rat-tail of dependencies.

Up until now I knew of following approaches to choose an implementation:

a.) Pick one format and go with it.

var loadConfig = require(‘load-json-config’)
loadConfig('./config.json')

Problems:

1. This results in “load-json-config”, “load-cson-config”,… packages which spams NPM and makes it hard to find anything.
2. The API might be completely different for every implementation. In that case the user would become tightly bound to JSON if he uses the “load-json-config”.

b.) Pick one format and offer an implementation hook.

var loadConfig = require(‘load-json-config’)
var jsYaml = require(‘js-yaml’)
loadConfig(‘./config.yaml’, {
  parse: function (blob){
    return jsYaml.load(blob.toString())
  }
})

Problems:

1. The default implementation is always shipped with the code. (bloating the installation)
2. Every new user needs to think of how to satisfy your API.
3. You might not think of one particular way to implement the API. But since the API is public, it will be a pain to upgrade the specification.
   i.e. Some file parsers are built asynchronously.

c.) Pick no format, let the user decide.

var loadConfig = require(‘load-config’)
loadConfig(‘./config.json’, {
  parse: function (blob) {
    return JSON.parse(blob)
  }
})

Problems:

1. Same like b.) you have the problem of the exposed API and that every user needs to think again
2. Your implementation would not be comfortable for every given case.
3. Packages like “load-json-config” would pop-up that simply fill out the
   “parse”-config. This would spam NPM even more than a.).

d.) Implement every format that makes sense to you.

var loadConfig = require(‘load-config’)
loadConfig('./config.yaml')

Problems:

1. It will take a lot of time to implement all configuration files properly.
2. Loading all of the possible dependencies will consume a lot bandwidth (when installing the package) and resources (on the hard disk).

Interface vs. Implementation

When I think of this problem i think of Interfaces. In other programming languages (such as TypeScript or Java) there has long been the concept of an interface. An interface in the best case describes a simplification of how to solve a problem (pseudo syntax).

interface Config {
  load (path:String):Object
}

You can deal with the actual implementation in a later step. So: while we depend in our code to “how it works” we don’t depend on “how it is implemented”. Which is reducing a lot of the headache.
In our case of the config: loading should describe the interface. This is a beautiful way to write c.). Public interfaces then to be hard to maintain but it can be a good way to implement d.).

Now, where is the novelty?

A better world

Let’s assume we use the simplest approach d.):

var loadConfig = require(‘load-config’) 
loadConfig.load(‘./config.yaml’)

This is awesome for the user. He doesn’t need to think! Now we just need to to implement all the formats (we can do that incrementally), but instead of having a very-long list of dependencies, how about getting a good error message, like:

EPACKAGEMISSING: Trying to load yaml file.
This error can be easily fixed by running the following command:
$ npm install --save js-yaml

That would be awesome! Now the user knows he is at the right place and he can fix it by running a simple script.

How could an implementation of “load-config” look like?

Let’s take advantage of the fact that Node.js falls-back up the folder hierarchy to check for a package. This means “load-config” can access a package like “js-yaml” without it being installed as an own dependency, it is enough if the parent folder has it installed!

var impl = require(‘require-implementation’)
function loadConfig (filePath) {
  if (path.extname(filePath) === '.yml') {
    var yaml = impl('Trying to load yaml file.').require('js-yaml')
    return yaml.load(fs.readFileSync(filePath, 'utf-8'))
  }
  return JSON.parse(filePath)
}

“require-implementation” is a simple, fresh package that looks if “js-yaml” exists and if not, throws that nice error message mentioned above. Now it is clear to everyone how to fix the missing dependency.

Application to other problems

This example uses configuration-files but it seems to apply when:

You want to offer a general solution to a problem-domain that has a variety of implementations but usually only a small subset of it is used at a time.

I can imagine other examples:

• An image loader that wants to support a great variety of image formats. (The image you load has one format. Can be applied to other formats as well: i.e. 3D model loader, word documents, presentation files…)
• A command line tool that wants to support translation of the UI to a variety of languages.
  (Usually you want to read in the same language.)
• A gamepad api that has a long list of different gamepad drivers.
  (Usually only one Joystick is plugged in.)
• A Query-builder tool that has a drivers for different databases.
  (The average server connect to one, maybe two, database types.)
• A HTML template rendering engine that wants to offer every template language there is™ . (see: https://github.com/tj/consolidate.js)
  (For the sanity of your coders: use one template language)

Final words

Finding the right package/implementation is a general, hard problem of software engineering. I think with this approach we could make it significantly simpler for people to use any new concepts while still maintaining slim & fast code.

This is not a solution for everything. I respect the Node.js community for the incredible amount of packages that solve one thing well. In fact I think that making such a meta-package would only be possible by using a lot of small, well build packages.

But as a user: instead of digging through tons of documentation and searching/comparing solutions on NPM the process would become a lot easier:

1. Download the solution package (like: load-config).
2. Run it against your code.
3. Install a dependency that allows you to implement your requirements.

To me this could be a very nice process. Nicer than the current solution of trial & error with NPM.

What do you think?

Would you join my efforts to write a general “read-structured-file” package?

[EDIT]: I opened the issue #13869 at NPM asking for proper support of this.