The Case for Multithreading in Node.js
Multi-threading allows your application to handle long-running synchronous workloads, like AI inference, without blocking the main thread’s event loop. When the event loop is blocked, the system becomes unresponsive. It’s not until the synchronous code completes, that the system begins processing other requests. Let’s look at a use case that requires this kind of workload and see how to handle it.
In the case of async I/O, we know that one thread is more than enough. It’s like a restaurant with one waiter, who can handle more than one table at a time, because he doesn’t stand around in the kitchen waiting for the last order to finish before taking the next one. This is concurrency.
We use an equivalent strategy when it comes to data intake. We avoid unnecessary memory consumption and wasted clock cycles, by streaming content, and processing it in chunks, as it arrives, rather than waiting for the entire transfer to complete. Combining asynchronous programming and streaming, we can efficiently process large or continuous datasets, like frames from a CCTV or LiDAR camera.
But what if we want to identify the faces in those frames. Then we need the help of machine learning. But whereas streaming is I/O bound, AI inference is CPU-bound. How can we handle these incompatible workloads? Concurrency won’t cut it. We need parallel execution. We need another waiter.
This is where multithreading comes in. The main thread hands off the AI workload to another thread to process. Although the second thread will be blocked, the main thread goes on to handle other work. Sometime later it will see the AI thread has finished and run a callback to complete the rest of the request.
It’s only fair to mention that the event loop already works like this. It’s just that the underlying threads are managed for us by libuv. In other words, the only reason the waiter can leave the kitchen to take another order, is because the chef does the work. We are following the same pattern. Only, instead of I/O-intensive, we are targeting CPU-intensive functions, and managing the threads ourselves.
