What is multi-threading?

Multithreading is the ability of a program to perform multiple tasks concurrently. Multi-threading is a programming language term derived from a concept of multitasking. It is used when we want to divide our job into several independent parts. It allows the execution of multiple parts of a program at the same time. These parts are known as threads and are lightweight processes available within the process. These threads are the individual process utilizing the system memory and resources in a better way to ensure high performance. Node.js does not support multithreading. Therefore, Node.js have different ways to achieve that than traditional multithreading in many other high-level languages.

When do we need multi-threading?

When you want to perform series of complex operations (Fetch records from a database, image processing, API request, etc.) irrespective of their sequence, priorities, and response, you have to think of using multi-threading for your application. Single-threaded architecture has many limitations when dealing with performance, memory, resource utilization. When you want to defer a part of your task to some other process and continue executing your main application you should think of multi-threaded architecture.

How to create a multi-threaded architecture in NodeJs?

Node.js is a single-threaded programming language. Being single-threaded means that only one set of instructions is executed at any time in the same process. But Node.js uses multiple threads to execute asynchronous code in the background. But that does not help us to execute our application in multiple threads in real-time.

In NodeJs multi-threading can be achieved by tree different mechanisms.

• Multilevel Architecture – Multiple independent processes communicate over sockets.
• Clustered Architecture – Master forks multiple child process sharing same port and communicate over IPC.
• Worker Threaded Architecture – Single process creates multiple threads with each thread having its own Node instance, Event loop and they all share memory.

Let’s deep dive a little bit more into each of them with the help of a very real-time example. Let’s take an example of a stock exchange application where you are requested to fetch information on live trades and create a 15 min candle/bar chart for a particular stock.
For this application, we will try to create an architecture using each of the methods listed above.

Multilevel Architecture

When you want to defer and execute some of your tasks in the background, you can choose multilevel architecture. Here you will create multiple Node processes to serve the purpose, out of them one will be the master processor or in technical terms, you can call it a server and others are slave or child processes or in technical terms, you can call them clients. Every process has its own instance of a V8 engine, event loop, and memory.

When you create a server and client(s) they will establish a communication channel with the help of sockets where they can exchange messages with each other. Here a server will be the main thread of your application and all clients are helping your main thread to perform a complex operation in an efficient and faster way. This architecture will help you perform your time-consuming complex operations in the background and continue to serve the main purpose.

Now let’s take an example of stock exchange application.

We know that the trading applications have millions of users so handling everything with the help of a single thread or a process would lead to the performance issues. As we know faster response time is the key requirement in such applications. Here you will create a server that will be the main thread to serve every request coming to your application. The main thread of your application is responsible to serve many different requests from millions of users. For faster performance, main thread should defer some heavy operations to other processes. Now we have to identify complex and time-consuming tasks that can be performed in the background and deferred to other processes. In this particular case, we can create a separate process to fetch live trades from the system database or from some API and create 15-minute bar chart. Once the trade starts, child process begins to read live trades, create a bar with an interval of 15 minutes and send them to server. Server then send those bars to the client (Browser in this case) requested for it.

Github Link: https://github.com/sandysalunke/multithreaded-architecture

Let’s understand this with the help of a simple architecture diagram.

Clustered Architecture

“A single instance of Node.js runs in a single thread. To take advantage of multi-core systems, the user will sometimes want to launch a cluster of Node.js processes to handle the load.”
– Node.js Documentation

Clustering refers to the concept of creating child processes across master that all share server ports. Communication between worker processes and master happens through IPC (Inter-process communication). By creating a pool of cluster we can easily distribute the load among different child processes. Clustered architecture is used to speed up the execution of the program and ensure high performance in case of complex operations. This helps the master process to easily distribute the load among its predefined set of child processes.

Every Node (forked) process has its own V8 instance, event loop, and memory. All processes in the cluster can share the server ports but they don’t actually share any state, and they don’t share the event loops.

In comparison to the multilevel architecture, clustered architecture manage the process in a very efficient manner. Here we don’t have to manually create a child process, we can easily fork multiple child processes to better utilize the core capabilities and resources of the CPU. Any process killed can be easily replaced with the new one to ensure high performance.

A child process can perform the desired task and report back to the master process with the result. This way master can defer different requests coming to it to its child processes and send a response back to the requesting clients once get the result of the child process. A clustered architecture with 10 child process can perform 10 operations simultaneously, this will help to improve the performance of the system.

Now let’s take an example of stock exchange application.

We can create the main process to serve all requests coming to an application and create multiple child processes (cluster) to distribute the load. Let’s consider there are multiple clients requesting for bar chart of different stocks. We can create a cluster of child processes to serve these different requests. Clustered architecture can be combined with the multilevel architecture for better scalability.

Note: A built in “cluster” module from the core set of Node.js libraries can be used to create cluster pool.

Let’s understand this with the help of a simple architecture diagram.

Worker Threaded Architecture

NodeJS does not support multi-threading. Therefore, Node.js Worker Threads behave in a different way than traditional multi-threading in many other high-level languages. In Node.js, a worker’s responsibility is to execute a piece of code (worker script) provided by the parent worker. The worker script will then run in isolation from other workers, with the ability to pass messages between it and the parent worker.

Each worker is connected to its parent worker via IPC message channel. A Message Channel is a simple communication channel. It has two ends, which are called ‘ports’. In NodeJS terminology, two ends of a Message Channel are just called ‘port1′ and ‘port2′.

In Node.js, each worker will have its own instance of V8 and Event Loop. However, unlike child processes, Workers can share memory. NodeJS v10.5 introduced a built in library to support “worker threads”.

Worker threads are found to improve the performance of applications by distributing the load among the pool of worker threads. Worker threads are light weighted than the process, hence they perform better than clusters and they also share the memory of a parent. But we need to remember that Even though worker threads are lightweight than actual processes, spawning workers involve some serious work and can be expensive if done frequently. But this can be solved by creating a pool of worker threads so that any incoming request can be passed to the next available worker thread. It is also not advised to use worked threads to perform intensive I/O operations.

Now let’s take an example of stock exchange application.

We can create an application server and multiple worker threads to server requests coming from millions of users. Every new request coming to the server can be routed to the next available worker thread. Worker threads can fetch required information (trades in this case) from the database/API and perform some business logic (Create a bar of 15 mins) and then once it’s done with its desired task it will send the response back to its parent thread.

Let’s understand this with the help of a simple architecture diagram.

Now you might be thinking that all of the above three architecture methods seem identical, I agree they do, but they do actually defer by the way of their implementation, communication, pros, cons, and performance. Based on the business and technical requirements of your application you need to choose the appropriate method to design your system architecture.

In next article I will have you deep dive in more details of each of these methods, so that you can actually design your suitable architecture on your own.

You would also like to read my other articles.
Create simple chat application using Node.JS, Express.JS and Socket.IO
ChatBox: A Peer to Peer Chat Application

Thank you!