Working with Node.js Streams

Poorshad Shaddel
Sep 4 · 5 min read
Image for post
Image for post
Datastream

What are streams?

Streams are a collection of data that might not be complete and we might not fit them in memory.

When we watch an online video we use a stream. Do we need to download the whole video to watch it? the answer is no. thus the video data is not complete but we can watch the parts we want!

Imagine we are uploading a video with 12GB size. The problem is our memory is only 4GB. So how is it possible to upload such a file? the answer is that with using streams I don’t need to use my memory and load that 12GB file in my memory and I’m going to send this file chunk by chunk in a stream.

Where do we use streams in Node.js?

Some of the Node.js modules implementing a stream interface.
HTTP responses are the most common example of streams. HTTP responses are readable streams on the client and writable streams on the server. We are going to discuss different types of streams.

Image for post
Image for post
Screenshot from www.freecodecamp.com

What are the different types of streams?

  • Writable Stream: we can write to this stream.
  • Readable Stream: we can read from this stream.
  • Duplex Stream: we can read and write to this stream.
  • Transform: Transform streams are duplex streams where the output is in some way related to the input.

It is impossible to understand these sentences without proper examples. So let’s see a few examples:

Writable Stream examples

Writing to HTTP response

Here is a simple HTTP server and we want to see if we can write to HTTP response or not :

NodeJS simple http server
NodeJS simple http server
Simple HTTP server

This is a server listening on port 2000. As you can see we have written a few lines using write method on this writable stream. If we send a request to this port the output will be like this:

The output of a request sent to port 2000
The output of a request sent to port 2000
The output of a request sent to port 2000

Another important part of this code is using end method on a writable stream.

If we don’t use the end() method the client is going to wait until the request becomes timed out.

Writing to a file

We want to write a specific sentence a million times in a file.

We can create a string and we can append this sentence a million times but in the real-world, we cannot do this because some times the size of this variable can be more than our memory size. This was part of the stream definition and we can use a writable stream to fill this file with our sentences.

Image for post
Image for post
Writing to a file using fs.createWriteStream

Readable Stream Examples

Reading from a file

Imagine we want to read from a big file like the file we created using a writable stream and we want to check if a word exists or not. Since this is a bad idea to load the whole file into memory we can use a readable stream instead.

Reading from a file using a readable stream
Reading from a file using a readable stream
Reading from a file using a readable stream

Remember that streams are event emitter and we can register event listeners. Here we used data event to log data.

Two modes of Readable Streams

There are two modes of readable streams:

Paused mode and Flowing mode

When we use the Flowing mode the data is coming and we have to use event handlers and listen to data event. Actually, when we use event handlers we switch the readable stream to the Flowing mode. In order to change the stream mode, we can use paused() method and resume(). In Paused mode, we read data using read() method.

How we can read from a readable stream and write to another writable stream?

Let’s see the answer in a simple example. We know that HTTP response is a writable stream on the server and also we know that we can create a readable stream from a file. So let’s give the content of the file as a response to the user. We have a readable stream and we want to write it to a writable stream (response).

The first way we can do this is by using event handlers:

How we can read from a readable stream and write to another writable stream?
How we can read from a readable stream and write to another writable stream?
send a file in response

As you can see we have two event listeners one for writing data to res and one for ending the response.

The second way is using pipe and it is just like pipes in Linux.

The second way is using pipe and it is just like pipes in Linux.
The second way is using pipe and it is just like pipes in Linux.
Using pipe method

Readable streams have a pipe method that we can use to pass a writable stream.

Example of Duplex Streams

We can create any kind of stream using the stream module.

In order to have a duplex stream, we need to implement both read and write methods:

Duplex stream

I used process.stdin which is a readable stream and process.stdout which is a writable stream so when I write something on my console I’m writing to that duplex stream.

Example of Transform stream and its usage

We need to use stream.Transform to create a transform stream and all we need to do is implementing the transform method.

For example, I want to replace the numbers of a stream with *:

Simple transform stream to convert numbers of a string to *

As you can see we must use push method to add something to a readable stream and we have access to chunk which comes from a writable stream.

Now let’s see what is the result:

Console result of using a transform stream

I entered the first line and the second line is written by transform stream to process.stdout.

JavaScript In Plain English

New JavaScript + Web Development articles every day.

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store