Reacting to filesystem io events with Node.js
I recently was working on a larger ETL process that started with the reception of various data files via SFTP that were delivered on varying schedules. The requirement was that as files are received we generate a unique event in a database, then execute a sequence of commands to archive the files out of the delivery directory and offline to a central immutable annotated file repository.
This new functionality had to integrate with an existing SFTP legacy server, and would likely have other uses outside of this initial use-case.
Looking around for simple solutions based on a scripting language, I really could not find any that would work or be extensible enough for the need. Hence I ended up writing io-event-reactor.
The basic concept is this; you have a monitor that listens for IO events for particular paths on the filesystem. As these IO events occur, they are passed on to one or more evaluators to decide whether or not the IoEvent should be reacted to by one or more configured reactors. The entire latter sequence is encapsulated in an IoReactor instance that manages the flow between the three described components.
With this module, you construct and configure a single IoReactorService which can manage and contain one or more IoReactor instances, as many as you wish, providing for lots of flexibility for reacting to filesystem events.
When you configure the IoReactorService and its IoReactor instances, you specify which plugins you would like to use to fulfill the monitor and reactor roles. For evaluators you simply provide one or more functions which evaluate whether or not an IoEvent should be passed on to one or more reactors.
https://github.com/bitsofinfo/io-event-reactor
The default monitoring plugin is implemented using the great Chokidar library at: https://github.com/paulmillr/chokidar
For reactor plugins, I developed two based on my initial needs.
For an real-world example of the kind of application you could build on top of this, check out io-overwatch (albiet a simple utility) at: https://github.com/bitsofinfo/io-overwatch
Originally published at bitsofinfo.wordpress.com on November 2, 2016.
