Multi threading in NodeJS? How and what’s new?

Yes, you read it right — Multi threading in NodeJS. But isn’t Node a single-threaded? — I know this is hitting your mind.

Well, that’s the old story about Node. Node has introduced worker-threadsin version 12+ and It’s provided out of the box in the NodeJs core modules.

But before you get excited — let me make it clear— the way multi-threading is implemented in Node is not the same as we would implement in other languages like Java, Python, etc.

So before we come to worker-threads — Let us first understand how NodeJS works under-the-hood:

Most of the server-side languages, like PHP, ASP.NET, Ruby, JAVA servers, follow multi-threaded architecture. That means, every request by the client results in the instantiation of a new thread or even a process.

However, in NodeJS, all requests are handled in a single thread with shared resources. Then how does NodeJS handles concurrent traffic or requests?

Assume a request is sent to a Node app, this single-threaded app accepts the request (it’s non-blocking IO right, so it has to accept it) and it starts processing the request as soon as it receives it, as per the code mentioned inside the controller representing the route.

At the same time, there can be more than one request hitting this same app, and even those requests get accepted and begin to get processed.

But how is that possible when Node is single threaded?

Let’s take an example of a router code of a node’s expressJs application for better understanding.

Let’s say there is a route /send -email and two requests to send email comes at the same time. Both the requests start processing immediately. By the code, the above functions get executed for each request.

Question is, will both the requests be run in parallel? No, not. NodeJs is single-threaded and it will run each thing at a time.

Then does it mean that each request will run one after another in sequential order? Did I not say that both the requests get accepted and start processing as soon as it gets accepted?

Yes, I did say that. Both of the above statements are correct and the real answer is that things happen concurrently but not in parallel.

Concurrency and parallelism are two related but distinct concepts.

Concurrency means, essentially, that task A and task B both need to happen independently of each other, and A starts running, and then B starts before A is finished.

There are various different ways of accomplishing concurrency. One of them is parallelism — having multiple CPUs working on different tasks at the same time. But that’s not the only way. Another is by task switching, which works like this: Task A works up to a certain point, then the CPU working on it stops and switches over to task B, works on it for a while, and then switches back to task A. If the time slices are small enough, it may appear to the user that both things are being run in parallel, even though they’re actually being processed in serial by a multitasking CPU.

So basically— Node handles requests concurrently but not parallelly. And all this is done using a “Single Threaded Event Loop Model” architecture that runs on top of a single V8 engine instance.

Let’s go through the controller function for the request, to understand how this happens in each line.

1. Extracts the three variables message, from, to from the request body. This is a synchronous operation.
2. Create a template for account confirmation and stores it to a variable called template . This is a synchronous operation.
3. Generates an email object and stores it to a variable called email. This is a synchronous operation.
4. Pushes the email object to the email service. This is an asynchronous operation and takes a bit of time to happen. And whenever the operation is completed, then the response is sent back to the client.

Now that we have gone through the code, let’s see how it gets executed for both requests together.

NodeJs event-loop is single-threaded, and it will execute the controller function for the initial request first, and the following gets executed :

1. Line 1 gets executed and let’s say it takes 4ms for example.
2. Line 2 gets executed and this takes 3ms.
3. Line 3 gets executed and this takes 3 ms.
4. Now, this task to push `email message` to email service gets executed, and let’s assume that it takes 14ms. (These are usually IO calls like HTTP or some messaging call to a message queue, hence it’s asynchronous)

Let’s deviate a bit from this flow and understand the internal thread pool in NodeJS.

Now, something we need to understand about nodeJs is there is an internal thread pool maintained in nodeJs, and these threads in the internal thread pool are used for specific asynchronous tasks such as HTTP calls, Database operation, libraries like `bcrypt` use this for encryption, File operations, etc

Hmmm, So nodeJS do use multiple threads?

Yes, they do, but it’s used internally by nodeJs itself not putting the burden on the developer to deal with the heavy task of managing threads and to bring asynchronous behavior to the code.

Now, this doesn’t mean the internal thread pool has unlimited threads. It just has 4 by default. But you can easily change it according to your system resources by setting an environment variable in two ways :

1. While running the app :

UV_THREADPOOL_SIZE=64 node index.js

2. Inside the app, at the beginning of the main file.

process.env.UV_THREADPOOL_SIZE=64

Whenever there is an opportunity for processing a task asynchronously, it's either transferred to the thread pool or queued until a thread is free. And once the task completes the callback associated with that async task is invoked.

Now that we have enough information about the internal thread pool, let’s move back to the case we were discussing.

In the 4th step in the process we discussed, it takes 14 ms, and here comes the part where nodeJs become different from the rest of the languages.

When the 4th step is transferred to a thread in the internal thread pool, it need not to wait for the task to complete. The main event loop becomes free and hence it starts processing the next request.

So we see that before even a request completes, another request begins to process. And in the middle of the second request being processed, the first request 4th step completes and the callback gets invoked before the second request finishes. And once the callback finishes executing, the second request continues to process.

This is how NodeJs deals with concurrent requests using just one single-threaded event loop.

Old versions of nodeJS worked like this — so the question is how does new versioned NodeJS works?

The simplest answer is — it works the same way it used to. But something new has come which allows the developer to create new threads in their application apart from the already available internal thread pool.

And this is possible using worker-threads module, which is part of nodeJs from version 12+.

Unlike other languages, here in nodeJs these threads can be used and reused by every request or task. That means we create a thread with which we mention what it must do, by passing a JS file to it.

Now the developer can pass data to the worker and it gives the output.

Please have a look at the diagram to relate to what I explained.

Now you can see an example, of how to use a worker thread.

The above code is of the worker file: worker-script.js

You must have understood from the above codes, how you could pass some data to a worker and get the output.

And till the time the worker processes the message, the main application can do whatever task requires to be done. And once the worker finishes the task, the main application can receive the message from the worker and do the next steps.

There are good libraries for using worker-threads in nodeJs with ease, called V-blaze, Greenlet.

By this, we come to an end to this article. I believe the beginners would have got a good understanding of worker threads and also how the concurrent requests are handled in nodeJs.