How JavaScript works: Streams and their use cases

Published in

SessionStack Blog

8 min readDec 17, 2021

This is post # 56 of the series, dedicated to exploring JavaScript and its building components. In the process of identifying and describing the core elements, we also share some rules of thumb we use when building SessionStack, a JavaScript tool for developers to identify, visualize, and reproduce web app bugs through pixel-perfect session replay.

Streams are an important part of computing that is mostly used to send data from a source to a location in a particular order. They are a way to handle information exchange between applications, introduced to the Unix Operating system which has been helpful to pass data to an application in a sequence without having to wait for the full content which can be time-consuming.

A practical aspect of a Stream is streaming a video on Netflix. The video is served to you as you watch, allowing you not to download the whole video which takes time and memory resources.

In this article, we will look into the good parts of Streams for Node.js. Node.js has provided certain API interfaces for us to use and work on streaming data. We will discuss more on that, the types, reasons to use streams and some implementations of these types of streams.

Basic knowledge of Node.js is required to understand this article.

What are Streams

In Node.js, Streams are implementing the standard definition of streams. What we are going to be looking into are the Streams API and how we can use it to work with the streaming of data. The good thing about Streams is how they help read large chunks of data in pieces and process them instead of reading them all at once which is the traditional way. This helps when you have limited memory space.

Working with streams can be as simple as reading content from a text file using the fs(file system) module. The fs module comes with a handful of methods needed to read from the file system, and it also has the fs.createReadStream() which creates a readable stream and fs.createWriteStream which also creates a writable stream. What the two methods mean is that it allows access to a certain, read/write through them and stream the data to the client instead of waiting for the full data.

Let us see how we could do that the traditional way:

Notice that the readFile method reads the content of ‘file.txt’ and once it is complete, the callback function is called to return the results. If our content were to be very large it will take a significant amount of time to get the complete result.

Seeing how this plays out, let us look at the improvements we get by using a stream.

With the createReadStream, the content of the file is streamed to the client in chunks instead of waiting for the complete content.

Sending of the actual data is done with the pipe method which takes the chunk of data from the source and pipes it to the res object.

Streams also give us the power of composability in our codebase. — being able to have pieces of code as components and then using Streams to pipe data from one piece of code to another. This technique is a Unix philosophy derived by Ken Thompson which advises that code should be simple, clear, modular and extensible for other developers.

For a better understanding, you can send data between commands in Unix like this:

Each command is an executable program and data is sent to the next command using the pipe operator.

The | (pipe) symbol is still represented as the pipe() method in Node.js which sends data from the source to the destination.

Types of Streams

Since our focus is on Node.js Streams, let us look at the various types of Streams that are available for us. The streams module is available in Node.js to work with the streaming of data and also work with the proper type of Stream you need.

Writable: This is basically being used to write data to a file using Streams. With this type of Stream, you can only send data to a file or storage location. We can use the file system module fs.createWriteStream() as we had done above or used the method in the stream module provided for us:

2. Readable: When we want to read data from a file or storage location using Streams, we use this type of Stream. With this, you can pipe from a file into another file. Just like above, we can use the fs.createReadStream() or use the Stream module as well:

Our readableStream is ready to be used to read data.

3. Duplex: This is a combination of both the readable and writeable stream where we can pipe data from and pipe data into. An example of a Duplex stream is a Socket that provides two channels to send and receive data:

A Duplex stream creates a channel for a readable and writable stream to work together.

4. Transform: The Transform is very similar to the Duplex stream but there is one thing that is very different. The Transform stream works on the input data to produce a new output. This Stream transforms the data to what is expected by the user. An example of this type of Stream will follow in the next paragraphs.

Reasons to use Streams

A big benefit and common reason for using Streams is the handling of very large amounts of data that you don’t want to hold in memory. If you are going to be reading a file that is almost 5GB or more, and sending it to your users, doing so would take a while and consume a lot of server memory. Especially if you have a lot of users. The best approach is to read each file in pieces using Streams so you do not run out of memory.

They really help to transfer data efficiently and they work without changing the data also except when you transform it. Like we mentioned earlier, Streams give the power of composability in your code where each function can be made modular and data is conveyed to the next function to be worked on.

Implementations of Streams

We will be implementing the Readable and Writeable Stream together, and also the Transform Stream so we understand when and how to use them.

Let’s see the methods in the package the stream module Node.js provides for us

To read data using a Stream, the `Readable()` is what we use. We also create the Writeable Stream. The read stream is used to read the file or data and piped to the write stream.

To create a Transform Stream, we get the Transform object form the module set the object and implement the method it comes with it:

Our transform stream takes a string of letters and splits them with a comma and into an array.

Streams are very important to us these days because of the size of data we tend to handle. And with the stream module, Node.js provides as an API you could do cool things with them.

You have to be careful about the data structures and approaches you pick when dealing with data and I/O. Making the non-optimal choices could degrade the performance of your product and even make it unusable.

Even if you feel like the proper decisions have been made, it’s always necessary to verify that this is indeed true and your users have a great experience with your product.

A solution like SessionStack will allow you to replay customer journeys as videos, showing you how your customers actually experience your product. You can quickly determine whether your product is performing according to their expectations or not. In case you see that something is wrong, you can explore all of the technical details from the user’s browser such as the network, debug information, and everything about their environment so that you can easily understand the problem and resolve it.

There is a free trial if you’d like to give SessionStack a try.

If you missed the previous chapters of the series, you can find them here:

How JavaScript works: Streams and their use cases

What are Streams

Types of Streams

Reasons to use Streams

Implementations of Streams

Written by Victor Jonah