Bringing Streams Together with the Node Pipe Function
Node has many tools to manipulate and control the flow of data through the application — its event-based asynchronous architecture lends itself well to layers of abstractions that make day-to-day coding a breeze. For example, flowing Node readable streams into Node writable streams can be abstracted from a 10 to 15 line tangle of code into a single ‘pipe’ method — this concept of piping data between streams offers huge performance/memory boosts and drives much of Express asynchronous data flows. Sometimes it’s useful to take a step back and remember what these tools are actually doing under the hood.
Node streams are EventEmitters — interactions with these objects and their data buffers occur when events are emitted such as ‘data,’ and ‘error’ and ‘end’. Take a look at the code below for a typical read event from a file — the data stream is accumulated until it ends and is then sent to the client.
A better way to do this would be to write the data to the client as we read it. The ‘res’ object provides us with a writeable stream where we can ‘write’ chunks to the stream until we call the ‘end’ method on it. This allows us to bypass the step where we save the fully buffered stream in-memory and reduce the time it takes to send the full payload to the client.
This is how the pipe method works under the hood — it saves response time and memory by combining readable and writable streams as seen above. The above can be condensed into the following:
It’s a fraction of the code and makes so much sense!