Frontend Weekly
Published in

Frontend Weekly

Using JavaScript Generators to Create a Multithreaded Data Processor

Introduction

Understanding JavaScript Generators

function* countFrom(start: number) {
let current = start;
while (true) {
yield current;
current++;
}
}
const iterator = countFrom(0);
console.log(iterator.next()); // { value: 0, done: false }
console.log(iterator.next()); // { value: 1, done: false }
console.log(iterator.next()); // { value: 2, done: false }
const numbers = [...countFrom(10)];
console.log(numbers); // [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]

Using Web Workers to Create a Multithreaded Data Processor

const worker = new Worker("./worker.js");

worker.postMessage("Start processing data");
worker.onmessage = (event) => {
console.log(event.data); // the message received from worker
};

Hands-on, step-by-step implementation

Setting up the Project

mkdir data-processor
cd data-processor
yarn init
yarn add -D typescript
npx tsc --init
yarn add -D webpack webpack-cli

Creating the Generator Function for the Main Thread

const random = (limit: number) => Math.floor(Math.random() * limit) + 1
const dataSource = function*(initialData: Coin[]) {
let previousData = initialData;
while (true) {
// simulate real-life delays
yield new Promise(resolve => setTimeout(() => {
const updatedData = getUpdatedData(previousData);
previousData = updatedData;
resolve(updatedData);
}, random(500))
}
}
const getUpdatedData = (previousData: Coin[]): ProcessedCoin[] => {
const updatedData = previousData.map((coin) => {
const priceUpdate = (Math.random() - 0.5) * 0.05;
const newPrice = coin.price + priceUpdate;
const trend = priceUpdate > 0 ? "up" : priceUpdate < 0 ? "down" : "neutral";
return { ...coin, price: newPrice, trend, new: false, disabled: false };
});

// simulate new or removed coins
if (Math.random() > 0.8) {
const coinDescriptor = random(50000);
updatedData.push({
price: random(10),
name: `New Coin - ${coinDescriptor}`,
symbol: `NC${coinDescriptor}`,
trend: "neutral",
new: true,
disabled: false
});
}
if (Math.random() > 0.8) {
const removedCoinIndex = Math.floor(Math.random() * updatedData.length);
updatedData[removedCoinIndex] = {
...updatedData[removedCoinIndex],
disabled: true
};
}
return updatedData;
};
const iterator = dataSource(coinList);
iterator.next().value.then(updatedData => {
console.log(updatedData);
// [{price: 1.405, name: "Bitcoin", symbol: "BTC", trend: "up", new: false, disabled: false},
// {price: 34.12, name: "Ethereum", symbol: "ETH", trend: "down", new: false, disabled: false},
// ...]
});

Creating the Web Worker and Generator Function for the Worker Thread

const processData = function*(data: ProcessedCoin[]) {
// Perform data processing
const processedData = data.map((coin) => {
// ... perform calculations or transformations
return { ...coin, calculatedValue: coin.price * 2 };
});
yield processedData;
}
const worker = new Worker('worker.js');
const iterator = dataSource(coinList);
iterator.next().value.then(initialData => {
worker.postMessage(initialData);
worker.onmessage = (e) => {
console.log(e.data); // [{price: 2.81, name: "Bitcoin", symbol: "BTC", trend: "up", new: false, disabled: false, calculatedValue: 2.81}, ...]
iterator.next().value.then(data => worker.postMessage(data))
};
});

Using the yield* operator to pass control between threads

const mainThread = async function*() {
const iterator = dataSource(coinList);
while (true) {
const v = await iterator.next().value;
worker.postMessage(v)
}
}
const workerThread = function*() {
while (true) {
const data = yield;
const processedData = processData(data);
postMessage(processedData);
yield* processedData;
}
}

Wrapping Up the Generator Functions

const startProcessing = () => {
const iterator = mainThread();
iterator.next();
worker.onmessage = (e) => iterator.next(e.data);
worker.onerror = (e) => iterator.throw(e);
}
const worker = new Worker('worker.js');
worker.onmessage = ({ data }) => workerThread.next(data);

Final Words

Limitations

  • Web Workers have a limited API and don’t have access to the main thread’s DOM, so they are unsuitable for tasks that need direct access.
  • Complex data structures and objects must be serialized and de-serialized when passed between the main thread and the worker thread, which can be time-consuming and error-prone.
  • Web workers are not supported in all browsers, so you may need to provide fallbacks for older browsers.
  • The amount of web workers you can spawn is limited, so it’s important to be mindful of how many web workers are running at any given time.

Advantages

  • By offloading heavy data processing tasks to web workers, you can greatly improve the performance and responsiveness of your application.
  • Using generators, you can process data in smaller chunks, which can help to prevent the main thread from becoming blocked.
  • Web workers are suitable for computationally intensive tasks and can benefit from parallelism.
  • It enables the main thread to handle the UI interactions while processing.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Radovan Stevanovic

Curiosity in functional programming, cybersecurity, and blockchain drives me to create innovative solutions and continually improve my skills