Why I Wrote a JavaScript In-Memory Streams Library.


A few weeks ago I wrote a library called oenyi. It wraps a small subset of Image Processing commands and exposes them with a consistent API. It also implements resizing by 3 different methods: fit, cover and contain. I wrote oenyi as part of a big refactor we are doing at VoxFeed to make our code base much more readable and maintainable.

The reason why I wrote oenyi was to remove all image processing specific logic from our platform and put it in a library that met the interface we wanted, thus inverting the dependency line and replacing hundreds of lines of code to just about 10.

This Friday night I made a test coverage analysis on it and realised that I was missing many branches of code including all logic related to “stream behaviour”. So I wrote some tests for the “pipe” method using Node’s native streams in File System and fixed a few problems but then I thought:

Tests Should Run In Memory

When writing automated tests, we have to make sure they run super fast, each case in a few milliseconds; the full suite in a couple of seconds at most. To achieve this speed is important to keep things small and in memory but I was using File System.

I started by looking into existing node libraries for in memory streams, found a few but after looking at their documentation and code I discarded all but one, however it didn’t have any tests or badges of anything that would speak for its stability.

I changed my methods and tests from using file system streams to use memory-streams but struggled to make it work, in fact I failed. I spent about 4 hours trying before giving up thinking:

I am using these streams the same way as I would use node’s native streams, why is this not working? Oh, the library must be the problem.

I Better Write My Own Library

The Prototype goes first. I wrote a very fast implementation of Memory Streams and wrote just 2 tests, exactly for the 2 cases I needed:

TAP version 13
# should read from readable stream
ok 1 should be equal
# should pipe result
ok 2 should be equal
1..2
# tests 2
# pass 2

I did all this in no more than 15 minutes, something had to be wrong with the library I was using before. So I decided to go forward and implement my own in-memory streams package for node.

Crispy Stream

A few hours later I finished writing Crispy Stream, an in-memory stream implementation that allows you to do everything that node’s native file system streams allow, and since its compliant with the api of the abstract interface it works cross implementations or, from crispy to crispy.

Native Readable to Crispy Writable

var crispyStream = require('crispy-stream');
var fs = require('fs');
var input = '/path/to/input';
var pipable = fs.createReadStream(input);
var writable = crispyStream.createWriteStream();
pipable.pipe(writable);

Crispy Readable to Native Writable

var crispyStream = require('crispy-stream');
var fs = require('fs');
var input = 'pipe this';
var filename = '/path/to/output';
var pipable = crispyStream.createReadStream(input);
var writable = fs.createWriteStream(filename);
pipable.pipe(writable);

After I integrated crispy-stream in oenyi, my tests started to pass with no effort. Great success.