Unit-testing Child Processes in node.js

Introduction

This article attempts to demistify the basics of unit-testing spawned child processes in node.js.

Code that involves spawning and interacting with a child process often ends up in the “let’s just hope that this just works and trust that our acceptance tests will catch any problems” bin in our codebases. This is entirely understandable because child processes require a lot of interaction between crashed processes, synchronizing STDIN/STDOUT/STDERR, etc. Child process interaction is probably one of the harder things to unit test which is why we give it the Fight Club treatment.

With the right architecture, it’s not too hard to test the happy path of STDIN/STDOUT interaction of a process, but it’s really, really hard to mock out behaviors for literally anything that can go wrong or right with a child process. It’s this difficulty that causes developers to shy away from the whole mess in the first place. In the end, however, any tests are typically better than no tests, so this article will layout a basic call to the built-in *nix command wc and test that the interactions between STDIN, STDOUT, and STDERR work as our code expects.

The Setup

It’s hard to come up with a non-convoluted example involving calling out to a shell command whose functionality isn’t easily supported in node itself, but let’s say you want to call out to the *nix command wc to find the number of characters, words, and lines in a file and return in a plain old object.

To demonstrate two ways to interact with the wc command, it takes either a filename and returns something like:

$ wc /etc/passwd
     108     292    6804 /etc/passwd

Or can read from STDIN:

$ cat /etc/passwd | wc
     108     292    6804

The wc command outputs four columns:

1. Number of lines
2. Number of words
3. Number of characters
4. The filename being counted (if a filename was supplied)

See the wc man page for more details.

It’s quite helpful that wc can read from both a file and STDIN since that makes testing far easier as STDIN is just a Writable stream in node.js.

The Non-tested Way

Whether you inherited the code or you’re fixing some of your own tech debt and want to add tests, this is likely the code you’re faced with:

Note that the child process construction is an external dependency and the source data is just a filename.

The Problem

When considering how to unit-test this module, there are several problems to consider:

1. It only deals with locally-accessible files
2. It isn’t even remotely testable due to the child process being hard-coded

The first problem is easily fixed by just injecting the stream returned by fs.createReadStream(‘myfile.txt’) and you get:

This immediately expands the number of things your module can process since practically everything is streamable: files, s3 objects, HTTP responses, strings, etc.

The second problem, however, is a bit trickier to tackle. There are some problems to consider when devising a strategy to unit-test this module:

• The path to wc is hard-coded
• The wc command is itself hard-coded
• Your team may be developing on a system that lacks wc (eg: Windows) whereas the target environment has wc (eg: *nix)
• Your CI environment may lack wc
• While unlikely, the wc command may not execute permissions

These concerns may seem a bit overkill but this is a simplistic example and are more applicable for situations where the command to be tested is a custom binary or cannot be installed locally.

The only way to test this as-is is to do so on a system that has wc in /usr/bin executable by the user running the unit tests and even then wc should be considered an external dependency which should be avoided as much as possible.

One of the core tenets to TDD is to push the hard-to-test bits to the edges, so the solution is to just inject the child process that represents the call to wc, resulting in:

We now have a module that pipes a Readable stream to a “Transform” stream (a child process isn’t technically a Transform stream, but has STDIN written to and STDOUT/STDERR read from, so it helps to think about the problem in a Transform stream fashion) and returns the parsed wc output as an object. This doesn’t solve the entire problem of making sure that the wc process is called correctly, but it pushes that bit out to the edges, leaving the main meat of the function to only deal with the how the wc command is interacted with.

Mocking the Child Process

We now have a module that takes in a stream representing the source data, sends it all to a child process STDIN, and returns the parsed output as an object. Normally you would follow the red/green/refactor approach to writing or refactoring code, but I felt it would be illustrative to show the finished product first.

Child processes are EventEmitters that contain a Writable stream tied to STDIN and separate Readable streams tied to STDOUT and STDERR. As the wc command is interacted with by sending data to STDIN and reading from STDOUT, it becomes quite easy to mock out a child process that can do both:

With the above code snippet, we now have the basic shell of a mock child process that can be fleshed out for individual test functionality.

Testing the Happy Path

First, let’s test the happy path where wc is given a bunch of text on STDIN and outputs the stats on STDOUT:

Testing an Error Case

Second, consider the case where wc encountered an error condition and wrote a message to STDERR. Other than a “file not found” error which isn’t applicable here, I don’t personally know how to get wc to throw an error, but plenty of other commands do so let’s assume that there is some way to get wc to exhibit error behaviors. In that case, we want to write some data to STDERR and verify that an Error is thrown.

There are definitely more robust ways of dealing with errors like checking the exit code (eg: 0 for success, 1 for error), but that is for another article, dealing with the output of STDERR is sufficient for our purposes today.

Improvements

This is by no means an exhaustive list of all the behaviors one can expect to deal with when working with child processes. For example, there are some other behaviors left as an exercise for the reader even for the simple wc command:

• what happens if wc and doesn’t output the three space-delimited numeric fields?
• what happens if wc doesn’t output anything at all?
• what happens if you need to interrupt wc for some reason?

Some of these are obviously a bit far-fetched, but not every child process is as decades-old and time-tested as wc, so, as external dependencies, much care should be taken to think out as many behaviors the child process under test can exhibit.

Pushed to the Edges

I mentioned above that we took the actual instantiation of the child process and moved it up a level, making the interaction with the child process the part under test. This unfortunately still leaves a external dependencies portion of the original code untested:

At this point, you could probably get the remainder of this code covered by unit-tests with liberal and (probably) painful usage of proxyquire to mock both fs.createReadStream and spawn. Depending on your particular situation and dedication to 100% test coverage, that approach may be worth the effort, but just testing the core interaction is usually the important part to get right.

Conclusion

Child processes are instances of EventEmitter that contain Writable and Readable streams for STDIN, STDOUT, and STDERR and are fairly simple to mock out for injection into processing functions. The solution described above shows the basic strategy for refactoring code that calls wc to inject the child process and creating mock child processes to test both the happy path and an error condition in an architecture-agnostic way.

Technologies Used

I wrote the code and tests for this article using node.js v10.13.0, tape for unit-testing, and string-to-stream for easy construction of streams from strings.

Acknowledgements

Thanks to Abhiram Bharadwaj at Esri R&D DC for allowing me to throw him into the deep-end on unit-testing without complaint and whose struggles gave me the inspiration to write this article.