Async JavaScript, a Pocket Reference

Learn asynchronous programming in JavaScript

The asynchronous nature of JavaScript is one of the aspects of the language that can confuse a lot of people. Having a good understanding of its primary async constructs however, can reduce a lot of confusion about the language. The goal of this guide is to introduce you to async programming in JavaScript and provide you with the necessary techniques to write clean and maintainable async code. We will start by exploring synchronous and asynchronous execution models and their differences. Then, we’ll dive into callback functions and examine how they are used to capture results of async operations. After that, we’ll explore promises and learn how they abstract callback functions to simplify async flows. We will also look at generators and explore how they can be used in async flows. And at the end we’ll explore async functions and illustrate how they can be used along with promises to further simplify asynchronous operations.

Code Examples

All the code examples for this guide are available on Gitlab:

https://gitlab.com/aj_meyghani/asyncjs-code-examples

You can either clone the repo or download it as a zip file.

Other Pocket References

Be sure to check out my other pocket references:

Introduction

If you are new to async programming in JavaScript, you might be surprised with the result of the following piece of code:

setTimeout(() => console.log('1'), 0);
console.log('2');

What do you think the output of the above code will be? You might be tempted to say 1 and then 2, but the correct answer is 2 and then 1. In the following sections we will dive into the asynchronous model used in JavaScript and explain why the above code prints 2 and then 1.

Synchronous vs Asynchronous

When you do your daily tasks it’s highly likely that you do them asynchronously. Let’s look at an analogy to illustrate the difference between the synchronous and asynchronous models of execution.

Imagine that you have three tasks to do in your todo list:

  1. Do laundry
  2. Do groceries
  3. Cook dinner

With a synchronous model, you have to finish each task before moving on to the next. That is, you have to finish laundry first before moving onto doing the groceries. If your laundry machine is broken, you cannot do the groceries. The same applies for the third task, you can only cook dinner if and only if you have completed the groceries (and laundry).

Now with an asynchronous model, you don’t have to wait for each task to finish to move onto the next. You can start the washing machine and do the groceries while your clothes are being washed. By the time you come back from the grocery store, your clothes are washed. Now if you need to dry your clothes, you can put them in the dryer and cook dinner while your clothes are being dried.

That’s basically the main difference between the synchronous and the asynchronous models of execution. In the synchronous model, you have to wait for each task to finish before moving onto the next. But in the asynchronous model you don’t have to, you can schedule tasks in a way to effectively do more in less time and not wait if you don’t have to.

In the next section we will look at the event loop and learn how JavaScript deals with asynchronous tasks.

The Event Loop

Let’s look at the snippet that we saw in the beginning of this section:

setTimeout(() => console.log('1'), 0); // A
console.log('2'); // B

On line A, when you call the setTimeout method, the callback function () => console.log('1') will be pushed on whats called the message queue. After that, on line B, console.log(2) will be pushed on, whats called a stack, and will be called right away and prints 2 to the console. After console.log(2) is called the stack is empty and JavaScript moves onto the queue and executes the items in the queue. In this case we have a callback function () => console.log('1') to be executed. JavaScript executes it and prints 1 to the console.

The mechanism that manages this flow is called the event loop. The event loop is responsible for looking at the items on the stack and the items in the queue and scheduling the executions in the right order. In the figure below there are three tasks on the stack to be executed. Once they are finished, two more tasks are picked up from the queue (for example callback functions) and placed on he stack to be executed:

The event loop is like a task scheduler, it decides what should be executed now and what in the near future

Now that’s an oversimplified explanation of the event loop, but in essence, the event loop is responsible for deciding what can be executed now and what should be executed later.

Callback Functions

Before talking about callback functions in an async context, it’s important to learn how functions can be passed to other functions. Let’s look at an example and see how functions can be passed around just like any value in JavaScript.

let name = 'Tom';
hello(name);

In the code snippet above we define a variable called name and we assign a string to it. Then we pass it to the hello function as an argument. We can do the exact same thing with a function. We can define name to be a function instead and pass it to hello:

let name = () => 'Tom';
hello(name);

Technically speaking name is a callback function because it's passed to another function, but let's see what a callback function is in the context of an asynchronous operation.

In an async context, a callback function is just a normal JavaScript function that is called by JavaScript when an asynchronous operation is finished. By convention, established by Node, a callback function usually takes two arguments. The first captures errors, and the second captures the results. A callback function can be named or anonymous. Let’s look at a simple example showing how to read the contents of a file asynchronously using Node’s fs.readFile:

const fs = require('fs');
const handleReading = (err, content) => {
if(err) throw new Error(err);
return console.log(content);
};
fs.readFile('./my-file.txt', 'utf-8', handleReading);

The fs module has a method called readFile. It takes two required arguments, the first is the path to a file, and the last is a callback function. In the snippet above, the callback function is handleReading that takes two arguments. The first captures potential errors and the second captures the content.

Below is another example from the https module for making a GET request to a remote API server:

// code/callbacks/http-example.js
const https = require('https');
const url = 'https://jsonplaceholder.typicode.com/posts/1';
https.get(url, (response) => {
response.setEncoding('utf-8');
let body = '';
response.on('data', (d) => {
body += d;
});
response.on('end', (x) => {
console.log(body);
});
});

When you call the get method, a request is scheduled by JavaScript. When the result is available, JavaScript will call our function and will provide us with the result.

“Returning” an Async Result

When you perform an async operation, you cannot simply use the return statement to get the result. Let's say you have a function that wraps an async call. If you create a variable, and set it in the async callback, you won't be able to get the result from the outer function by simply returning the value:

function getData(options) {
var finalResult;
asyncTask(options, function(err, result) {
finalResult = result;
});
return finalResult;
}
getData(); // -> returns undefined

In the snippet above, when you call getData, it is immediately executed and the returned value is undefined. That's because at the time of calling the function, finalResult is not set to anything. It's only after a later point in time that the value gets set. The correct way of wrapping an async call, is to pass the outer function a callback:

function getData(options, callback) {
asyncTask(options, callback);
}
getData({}, function(err, result) {
if(err) return console.log(err);
return console.log(result);
});

In the snippet above, we define getData to accept a callback function as the second argument. We have also named it callback to make it clear that getData expects a callback function as its second argument.

Async Tasks In-order

If you have a couple of async tasks that depend on each other, you will have to call each task within the other task’s callback. For example, if you need to copy the content of a file, you would need to read the content of the file first before writing it to another file. Because of that you would need to call the writeFile method within the readFile callback:

const fs = require('fs');
fs.readFile('file.txt', 'utf-8', function readContent(err, content) {
if(err) {
return console.log(err);
}
fs.writeFile('copy.txt', content, function(err) {
if(err) {
return console.log(err);
}
return console.log('done');
});
});

Now, it could get messy if you have a lot of async operations that depend on each other. In that case, it’s better to name each callback function and define them separately to avoid confusion:

const fs = require('fs');
fs.readFile('file.txt', 'utf-8', readCb);

function readCb(err, content) {
if (err) {
return console.log(err);
}
return fs.writeFile('copy.txt', content, writeCb);
}

function writeCb(err) {
if(err) {
return console.log(err);
}
return console.log('Done');
}

In the snippet above we have defined two callback functions separately, readCb and writeCb. The benefits might not be that obvious from the example above, but for operations that have multiple dependencies, the named callback functions can save you a lot of hair-pulling down the line.

Exercise: Simple Callback

Define a compute function that takes two arguments:

  1. An array of integers
  2. A callback function that operates on the passed array

For example, the following would return 6 (assuming that the addAll function is defined).

const result = compute([1,2,3], addAll);

Solution

One possible solution is to check if the first argument is an array of integers and then call the callback function with the args:

// code/callbacks/exercises/simple-callback.js
function compute(nums, fn) {
if(!Array.isArray(nums)) return NaN;
const isAnyNotInteger = nums.some(v => !Number.isInteger(v));
if(isAnyNotInteger) return NaN;
return fn(nums);
}

Exercise: Async Callbacks in-order

In this exercise, we need to make a GET http call to an endpoint, and append the result to the content of a file and finally write the result to another one. For the sake of this exercise, let’s assume that each operation needs to happen in order:

  1. Make the GET http request to get the title of a post
  2. Read the content of a file
  3. Append the post title to the file content
  4. Write the result to a file

Solution

Below is a solution that uses named callbacks to perform each operation. In the Promises section we will see how to use promises to gather asyc results and take advantage of concurrent tasks. But for now, we are going to depend on each callback result to perform the next.

// code/callbacks/exercises/read-write/main.js
const fs = require('fs');
const request = require('request');
const url = 'https://jsonplaceholder.typicode.com/posts/2';

request.get(url, handleResponse);
function handleResponse(err, resp, body) {
if(err) throw new Error;
const post = JSON.parse(body);
const title = post.title;
fs.readFile('./file.txt', 'utf-8', readFile(title));
}
const readFile = title => (err, content) => {
if(err) throw new Error(err);
const result = title + content;
fs.writeFile('./result.txt', result , writeResult);
}
function writeResult(err) {
if(err) throw new Error(err);
console.log('done');
}

Promises

Promises are probably the most important abstractions in async programming with JavaScript. Make sure to read this section very carefully since promises lay the foundation for many other async abstractions.

A promise is an object that represents the result of an asynchronous operation that may or may not succeed when executed at some point in the future. For example, when you make a request to an API server, you can return a promise that would represent the result of the api call. The api call may or may not succeed, but eventually you will get a promise object that you can use. The function below performs an api call and returns the result in the form of a promise:

// code/promises/axios-example.js
const axios = require('axios'); // A
function getDataFromServer() {
const result = axios.get('https://jsonplaceholder.typicode.com/posts/1'); // B
return result; // C
}
  • On line A, we load the axios module which is a promise-based http client
  • On line B, we make a GET request to a public api endpoint and store the result in the result constant
  • On line C, we return the promise

Now, we can simply call the function and access the results and catch possible errors:

getDataFromServer()
.then(function(response) {
console.log(response);
})
.catch(function(error) {
console.log(error);
});

Every promise has a then and a catch method. You would use the then method to capture the result of the operation if it succeeds (resolved promise), and the catch method if the operation fails (rejected promise). Note that both then and catch receive a callback function with a single argument to capture the result. Also, it's worth noting that both of these methods return a promise that allows us to potentially chain them.

Below are a couple of other examples of asynchronous tasks that can return a promise:

  • Reading the content of a file: the promise returned will include the content of the file
  • Listing the content of a directory: the promise returned will include the list of files
  • Parsing a csv file: the promise returned will include the parsed content
  • Running some query against a database: the promise returned will include the result of the query

The figure below summaries the states that a promise can have.

A promise can have three states: pending, fulfilled, or rejected. When an async value is being evaluated the promise is in pending state. When the async operation is finished, the promised will either be resolved with a value or rejected with an error.

Promise Advantages

Promises existed in other languages and were introduced to JavaScript to provide an abstraction over the callback mechanism. Callbacks are the primary mechanisms for dealing with asynchronous tasks, but they can get tedious to work with. Promises were implemented in JavaScript to simplify working with callbacks and asynchronous tasks.

Making a Promise

We can create a promise using the global Promise constructor:

const myPromise = new Promise();

The promise constructor takes a callback function with two arguments. The first argument is used to resolve or capture the result of an asynchronous operation, and the second is used to capture errors:

const myPromise = new Promise(function(resolve, reject) {
if(someError) {
reject(new Error(someError));
} else {
resolve('ok');
}
});

And as mentioned before, we can use the then method to use the results when the promise is resolved, and the catch method to handle errors:

myPromise
.then(function(result) {
console.log(result);
})
.catch(function(error) {
console.log(error);
});

It’s worth mentioning that we can wrap any asynchronous operation in a promise. For example, the fs.readFile is an method that reads the content of a file asynchronously. The fs.readFile method is used as follows:

fs.readFile('some-file.txt', 'utf-8', function(error, content) {
if(error) {
return console.log(error);
}
console.log(content);
});

We can create a function called readFile that uses fs.readFile, reads the content of a file and resolves a promise with the content, or reject it if there is an error:

// code/promises/wrap-readfile1.js
const fs = require('fs');
function readFile(file, format) {
format = format || 'utf-8';
function handler(resolve, reject) {
fs.readFile(file, format, function(err, content) {
if(err) {
return reject(err);
}
return resolve(content);
});
}
const promise = new Promise(handler);
return promise;
}

The same code can be re written more concisely as follows:

// code/promises/wrap-readfile2.js
const fs = require('fs');
function readFile(file, format = 'utf-8') {
return new Promise((resolve, reject) => {
fs.readFile(file, format, (err, content) => {
if(err) return reject(err);
resolve(content);
});
});
}

Now we can simply call our function and capture the result in the then method, and catch errors using the catch method:

readFile('./example.txt')
.then(content => console.log(content))
.catch(err => console.log(err));

Even more concisely we can re write the code above using the util.promisify method that was introduced in Node 8:

// code/promises/promisify-example.js
const fs = require('fs');
const util = require('util');
const readFile = util.promisify(fs.readFile);

readFile('./example.txt', 'utf-8')
.then(content => console.log(content))
.catch(err => console.log(err));

The util.promisify method takes a function that follows the Node callback convention and returns a promise-based version. You may be wondering why doesn't Node make all the methods promise-based. Low level Node methods are not promise based because promises are higher level abstractions over callbacks. It's up to the programmer to decide whether or not they need a higher abstraction like promises to handle async operations.

Promise Static Methods

The Promise constructor has a couple of useful static methods that is worth exploring. All the code snippets are in code/promises/static-methods.js. Some notable ones are listed below:

Promise.resolve: a shortcut for creating a promise object resolved with a given value

function getData() {
return Promise.resolve('some data');
}
getData()
.then(d => console.log(d));

Promise.reject: a shortcut for creating a promise object rejected with a given value

function rejectPromise() {
return Promise.reject(new Error('something went wrong'));
}
rejectPromise()
.catch(e => console.log(e));

Promise.all: used to wait for a couple of promises to be resolved

const p1 = Promise.resolve('v1');
const p2 = Promise.resolve('v2');
const p3 = Promise.resolve('v3');

const all = Promise.all([p1, p2, p3]);

all.then(values => console.log(values[0], values[1], values[2]));

Note that Promise.all takes an array of promise objects, evaluates them in random order, and "waits" until all of them are resolved. Eventually it will return a promise object that contains all the values in an array in the order that they were submitted, but not in the order that they were processed. Note that Promise.all does not process the promises in-order, it will evaluate them in the order that it prefers. In the next section we will look at executing promises in-order.

Promises In-order

If you want to run a couple of asynchronous tasks in order, you can follow the following pattern:

const promiseChain = task1()
.then(function(task1Result) {
return task2();
})
.then(function(task2Result) {
return task3();
})
.then(function(task3Result){
return task4();
})
.then(function(task4Result) {
console.log('done', task4Result);
})
.catch(function(err) {
console.log('Error', err);
});

The promise chain is kicked off by calling the first task that returns a promise. Afterwards, the then method is called which also returns a promise allowing us to keep chaining the then calls. Let's look at an example to say how you may want to use this pattern.

Let’s say we have a text file that contains a bunch of invalid characters the we need to remove. In order to accomplish that, first, we need to read the content of the file. Then, we need to remove the invalid characters, and finally write the results to another file. Assuming that we have a function for each operation that returns a promise, we can define the following promise chain:

// code/promise/promise-in-sequence.js
const promiseChain = readFile('example.txt')
.then(function(content) {
return removeInvalidChracters(content);
})
.then(function(cleanContent) {
return writeToFile('./clean-file.txt', cleanContent);
})
.then(function() {
console.log('done');
})
.catch(function(error) {
console.log(error);
});

Using the above promise chain, each task is finished before the next one starts causing the tasks to happen in the order that we like. Please note that you must return a value in the then block, otherwise your sequence will not be executed in the order that you intended. That’s why you cannot simply call an async function in a then block and assume that it will happen in the right order. Below is an example of such code that you should always avoid:

getUserData()
.then(info => {
authenticate(info)
.then(authResult => {
doSomething(authResult);
});
});

Instead use proper chaining and make sure to return a value at each step:

getUserData()
.then(info => authenticate(info))
.then(authResult => doSomething(authResult))

In the snippet above note that since we are using arrow functions without a {} block, the right side of the => is implicitly returned.

Running Promises Concurrently

When you call an asynchronous function that returns a promise, you can assume that the operation is executed asynchronously. Therefore, if you call each function one by one on each line, you are practically running each task concurrently:

function runAll() {
const p1 = taskA();
const p2 = taskB();
const p3 = taskC();
}

runAll();

Now, if you want to do something when all these operations are finished, you can use Promise.all:

// code/promises/run-all.js
function runAll() {
const p1 = taskA();
const p2 = taskB();
const p3 = taskC();
return Promise.all([p1, p2, p3]);
}

runAll()
.then(d => console.log(d, 'all done'))
.catch(e => console.log(e));

In the next section we will explore how you can combine promises that run concurrently and in-order.

Combining Promises

The main motivation for this section is mainly for the type of tasks that need to run concurrently and in sequence. Let’s say you have a bunch of files that you need to manipulate asynchronously. You may need to perform operation A, B, C, D in order on 3 different files, but you don’t care about the order that the files are processed in. All you care about is that the operations A, B, C, and D happen in the right order. We can use the following pattern to achieve that:

  1. Create a list of promises
  2. Each promise represents the sequence of async tasks A, B, C, D
  3. Use Promise.all to process all the promises. Note that, as mentioned before, the all method processes the promises concurrently:
const files = ['a.txt', 'b.txt', 'c.txt'];

function performInOrder(file) {
const promise = taskA(file)
.then(taskB)
.then(taskC)
.then(taskD);
return promise;
}

const operations = files.map(performInOrder);
const result = Promise.all(operations);

result.then(d => console.log(d)).catch(e => console.log(e));

Below is an actual code that you can run, assuming that you have the three files a.txt, b.txt and c.txt:

// code/promises/read-write-multiple-files/main.js
const fs = require('fs');
const util = require('util');
const readFile = util.promisify(fs.readFile);
const writeFile = util.promisify(fs.writeFile);

const copyFile = (file) => (content) => (writeFile(file + '-copy.txt', content));
const replaceContent = input => (Promise.resolve(input.replace(/-/g, 'zzzz')));
const processEachInOrder = file => {
return readFile(file, 'utf-8')
.then(replaceContent)
.then(copyFile(file));
}

const files = ['./a.txt', './b.txt', './c.txt'];
const promises = files.map(processEachInOrder);
Promise.all(promises)
.then(d => console.log(d))
.catch(e => console.log(e));

It’s worth noting that this kind of processing can introduce a big workload on the CPU if the input size is large. A better approach would be to limit the number of tasks that are processed concurrently. The async library has a qeueue method that limits the number of async tasks that are processed at a time, reducing extra workload on the CPU.

Exercise

As an exercise, write a script that reads the content of a directory (1 level deep) and copys only the files to another directory called output.

Hint You can use the example from the previous section as a starting point

Hint Here is the general idea: read the content of the folder, use the stat method to figure out which entry is a file, make the output folder, read each file, and write to the output folder

Solution

Below is one possible solution that uses the Promise.all pattern to process the read-write promises:

// code/promises/exercise/main.js
/*
List the content of the folder, filter out the files only
then copy to the output folder.
*/
const fs = require('fs');
const path = require('path');
const util = require('util');
const readFile = util.promisify(fs.readFile);
const writeFile = util.promisify(fs.writeFile);
const readdir = util.promisify(fs.readdir);
const stat = util.promisify(fs.stat);
const mkdir = util.promisify(fs.mkdir);
const outputFolder = './output';

function isFile(f) {
return stat(f).then(d => d.isFile() ? f : '');
}

function filterFiles(list) {
return Promise.all(list.map(isFile))
.then(files => files.filter(v => v));
}

function readWrite(result) {
const files = result[1];
return Promise.all(files.map(f => {
return readFile(f)
.then(content => writeFile(path.join(outputFolder, f), content));
}));
}

const getFiles = readdir('./').then(filterFiles);

Promise.all([mkdir(outputFolder), getFiles])
.then(readWrite)
.then(_ => console.log('done!'))
.catch(e => console.log(e));

Generators

Generators usually come up when talking about async programming in JavaScript. Knowing what they are lays the foundation for understanding other abstractions like async/await.

Generators are special functions that generate values when you need them to. When you call a generator it will not execute like a normal function. It will execute to the point where it sees a yield statement and it will exit until you need a new value. When you want a new value, you ask the generator for the next value and it will execute the function again from where it left off until there are no more values to generate. In the following sections we will learn how to create generators, how to iterate over them, how to stop them and more.

Creating Generators

You can create a generator by placing a * after the function keyword:

function* myGenerator() {
//...
}

Next, in the body of the generator function, we can generate values using the yield statement:

// code/generators/simple.js
function* simpleGenerator() {
yield 1;
yield 5;
}
const g = simpleGenerator();
const v1 = g.next().value; // --> 1
const v2 = g.next().value; // --> 5
const v3 = g.next().value; // --> undefined

You can even define an infinite loop and generate values:

// code/generators/inf-loop.js
function* myGenerator() {
let i = 0;
while(true) {
i += 1;
yield i;
}
}

Now if it were a normal function, it would get stuck in an infinite loop. But because this is a generator we can read values generated by calling next on the generator object returned:

const g = myGenerator();
const v1 = g.next(); // --> { value: 1, done: false }
const v2 = g.next(); // --> { value: 2, done: false }
const v3 = g.next(); // --> { value: 3, done: false }
// and so on...

Essentially, we enter and exit the function every time we call next and we pick up from where we last left off. Notice how the value of i is "remembered" every time we call next. Now let's update the code above and make the generator finish generating values. Let's make it so that it won't generate any values if i is bigger than 2:

function* myGenerator() {
let i = 0;
while(true) {
i += 1;
if(i > 2) {
return;
}
yield i;
}
}

or we can simplify the code above and move the condition to the while loop:

// code/generators/inf-loop-terminate.js
function* myGenerator() {
let i = 0;
while(i < 2) {
i += 1;
yield i;
}
}

Now if we read the generated values, we will only get two values out:

const g = myGenerator();
const v1 = g.next(); // --> { value: 1, done: false }
const v2 = g.next(); // --> { value: 2, done: false }
const v3 = g.next(); // --> { value: undefined, done: true }

Notice that after the second value, if we keep calling next, we will get the same result back. That is, a generator object with a value of undefined and the done property set to true indicating that there will be no more values generated.

Return Statements

A return statement in a generator marks the last value and no values will be generated after that:

// code/generators/return-statement.js
function* withReturn() {
yield 1;
yield 55;
return 250;
yield 500;
}
const g = withReturn();
const v1 = g.next().value; // --> 1
const v2 = g.next().value; // --> 55
const v3 = g.next().value; // --> 250
const v4 = g.next().value; // --> undefined

The code above will generate 1, 55 and 250. It will not reach the final yield statement, because the return statement marks the end of the generator.

Passing Values to Next

Using generators, you can pass a value to the next callback to use in place of the previously calculated yield statement. Let's look at a simple example to demonstrate what that means.

// code/generators/pass-next.js
function* myGenerator(n) {
const a = (yield 10) + n;
yield a;
}

const g = myGenerator(1);
g.next().value; // --> 10
g.next(100).value; // --> 101

Let’s go through the snippet above and explore what happens step by step:

  • First we call the generator and we pass 1 for n, and store the iterator object in g. Nothing new here.
  • Then, we call g.next to start the generator. The function is executed until it reaches the first yield statement: const a = (yield 10). At this point the value next to yeild is generated which is 10.
  • Then we call g.next and we pass 100. The function resumes from where it left off: + n but it will replace 100 for (yield 10) resulting in const a = 100 + n where n is 1. It will continue until it hits the next yield. In this case yield a which will generate 100 + 1 = 101.

We will use this special behavior of generators in later sections to implement a helper to handle async flows.

Calling Another Generator Within a Generator

You can use yield* inside a generator if you want to call another generator. In the example below, we have two generators, g1 and g2. We want to call g2 inside g1 and read the generated values:

// code/generators/call-another.js
function* g2() {
yield 2;
yield 3;
}
function* g1() {
yield 1;
yield* g2();
yield 4;
}

const vals = [...g1()];

console.log(vals); // -> [1,2,3,4]

In the snippet above we call the g1 generator and below is a summary of what happens:

  • The 1 value is generated from the first yield statement
  • Next, we hit yield* g2() which will generate all the values that g2 would generate, that is 2 and 3
  • Next, we come back to g1 and generated the final value, which is 4

Iterating Through Values

Using for-of

Since a generator function returns an iterable, we can use the for-of loop to read each generated value. Using the simple generator from above, we can write a loop to log each generated value:

// code/generators/use-for-of.js
function* myGenerator() {
let i = 0;
while(i < 2) {
i += 1;
yield i;
}
}

const g = myGenerator();
for(const v of g) {
console.log(v);
}

The code above will output 1 and then 2.

Using while Loop

You can also use a while loop to iterate through a generator object:

// code/generators/use-while-loop.js
const g = myGenerator();
let next = g.next().value;
while(next) {
console.log(next);
next = g.next().value;
}

In the while loop above, first we get the first generated value and we assign it to next. Then in the while loop, we set next to the next generated value. The while loop will keep going until next becomes undefined when the generator yields the last value.

Spread Operator and Array.from

Because a generator object is an iterable you can also use the spread operator to read the values:

// code/generators/use-spread.js
function* myGenerator() {
let i = 0;
while(i < 2) {
i += 1;
yield i;
}
}
const vals = [...myGenerator()]; // -> [1, 2]

In the example above first we call the generator myGenerator() and we place it in an array. And finally we use the spread operator right before it to essentially read each value out. The result is stored in the vals variable as an array with two values [1, 2].

In addition to the spread operator, you can also use the Array.from method to read the values and put them in an array:

// code/generators/use-array-from.js
function* myGenerator() {
let i = 0;
while(i < 2) {
i += 1;
yield i;
}
}
const vals = Array.from(myGenerator()); // --> [1, 2]

In the snippet above we call the generator and we pass it to Array.from which will read each value and store them in an array, resulting in [1, 2].

It’s worth mentioning that if you are iterating through a generator object that includes a return statement terminating the sequence, you won’t be able to read the last value if you use any of the internal iteration methods like for-of loop or the spread operator:

function* withReturn() {
yield 1;
yield 55;
return 250;
yield 500;
}
for(const v of withReturn()) {
console.log(v);
}

The code above will output 1 and then 55 but it won't output 250. This is also true if you use the spread operator:

function* withReturn() {
yield 1;
yield 55;
return 250;
yield 500;
}
const vals = [...withReturn()];
console.log(vals);

The code above will output [1, 55] and will not include 250. But notice that if we use a while loop, we can read all the values up until the value at the return statement:

function* withReturn() {
yield 1;
yield 55;
return 250;
yield 500;
}

const g = withReturn();
let next = g.next().value;

while(next) {
console.log(next);
next = g.next().value;
}

The while loop above will read all the values, including the value at the return statement, logging 1, 55, and 250 to the console.

Generating Infinite Sequences

In this section we are going to look at creating a Fibonacci sequence using a generator function. Note that the code used in this section is only for demonstration purposes. For practical purposes, you probably would want to use a pre-generated list to retrieve values for better performance.

The Fibonacci sequence is a sequence of numbers that starts with 0, and

  1. And the rest of the numbers in the sequence is calculated by adding the current value with the previous one:
  2. 0, 1, 1, 2, 3, 5, 8, 13, 21, …

or recursively, the sequence can be define as:

fib(n) = fib(n - 1) + fib(n - 2)

We can use the definition above and define a generator to produce n number of values:

// code/generators/fibo.js
function* fibo(n, prev = 0, current = 1) {
if (n === 0) {
return prev;
}
yield prev;
yield* fibo(n - 1, current, prev + current);
}

let vals = [...fibo(5)];
console.log(vals); //-> [ 0, 1, 1, 2, 3 ]

In the snippet above we define the first two numbers as default argument values using prev = 0 and current = 1. Below is a summary of what happens for n = 5:

  1. The first yield will generate the prev value, that is 0. Note that n is 4 now.
  2. Next, fibo(4 - 1, 1, 0 + 1) = fib(3, 1, 1) will generate 1.
  3. Next, fibo(3 - 1, 1, 1 + 1) = fibo(2, 1, 2) will generate 1.
  4. Next, fibo(2 - 1, 2, 1 + 2) = fibo(1, 2, 3) will generate 2.
  5. Next, fibo(1 - 1, 3, 2 + 3) = fibo(0, 3, 5) will generate 3, marking the end since n is 0 and we hit the return statement.

Generators and Async Operations

We can take advantage of the unique features of generators to essentially wait for async operations to finish before moving onto other parts of a function. In this section, we are going to write a helper function to allow us to do just that. But, first let’s review what happens when you pass g.next an argument. If you remember from the previous sections, if you pass g.next an argument, it is going to replace the given value with the previously yielded result:

function* myGenerator(n) {
const a = (yield 10) + n;
yield a;
}

const g = myGenerator(1);
g.next().value; // --> 10
g.next(100).value; // --> 101

We are going to use that as the foundation for our helper function. Now, first let’ start by making an asynchronous function that returns a promise:

const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));

This function returns a promise that resolves to the value 1 after 1 second. Now, let's create a generator function and call our async function inside it:

const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));

function* main() {
const result = yield asynTask1();
}

const g = main();
console.log(g.next());

What do you think the code above will output? Let’s go through it and figure out what’s going to happen:

  • First, we call the generator and store the generator object in g.
  • Then, we call next to get the first yield result. In this case it's going to be a promise since asynTask1 returns the promise.
  • Finally we log the value to the console: { value: Promise { <pending> }, done: false }.
  • After 1 second the program ends.

After the program ends we won’t get access to the resolved value. But imagine, if we could call next again and pass the resolved value to it at the "right" time. In that case, yield asynTask1() will be replaced with the resolved value and it would be assigned to result! Let's update the code above and make that happen with one promise:

const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));

function* main() {
const result = yield asynTask1();
return result; //<-- return the resolved value and mark the end.
}

const g = main();
const next = g.next();
console.log(next); // --> { value: Promise { <pending> }, done: false }
next.value.then(v => { // Resolve promise.
const r = g.next(v); // passing the resolved value to next.
console.log(r); // -> { value: 1, done: true }
});

In the snippet above we added a return statement in the generator to simply return the resolved value. But the important part is when we resolve the promise. When we resolve the promise, we call g.next(v) which replaces the yield asynTask1() with the resolved value and will assign it to result. Now, we are ready to write our helper function. This helper function is going to accept a generator and do what we discussed above. It is going to return the resolved value if there are no more values to be generated. We'll start by defining the helper function:

const helper = (gen) => {
const g = gen();
};

So far nothing special, we pass our helper a generator function and inside the helper we call the generator and assign the generator object to g. Next, we need to define a function that is going to handle calling next for us:

const helper = (gen) => {
const g = gen();
function callNext(resolved) {
const next = g.next(resolved); // replace the last yield with the resolved value
if(next.done) return next.value; // return the resolved value if not more items
return next.value.then(callNext); // pass `callNext` back again.
}
};

This function is going to take a single argument, the resolved value of a promise. Then, we call g.next with the resolved value, and will assign the result to the next variable. After that we will check if the generator is done. If so, we will simply return the value. And finally, we call next.value.then() and we will pass callNext back to it to recursively call the next for us until there no more values to generate. Now, to use this helper function, we will simply call it and we will pass our generator to it:

helper(function* main() {
const a = yield asynTask1();
console.log(a);
});

Now if you run the code above, you won’t see the logged result, and that’s because we have one missing piece. The callNext function in our helper needs to be immediately self-invoked, otherwise no one will call it:

const helper = (gen) => {
const g = gen();
(function callNext(resolved) {
const next = g.next(resolved);
if(next.done) return next.value;
return next.value.then(callNext);
}()); // <-- self invoking
};

Now that we have our helper function, let’s throw an error in the helper so that we can catch it later:

const helper = (gen) => {
const g = gen();
(function callNext(resolved) {
const next = g.next(resolved);
if(next.done) return next.value;
return next.value.then(callNext)
.catch(err => g.throw(err)); // <-- throw error
}());
};

The catch block will throw an error from the generator if any of the promises throws an error. And we can simply use a try-catch in the passed in generator function to handle errors. Putting it all together we will have:

// code/generators/async-flow.js
const asynTask1 = () => new Promise((r, j) => setTimeout(() => r(1), 1000));
const asynTask2 = () => new Promise((r, j) => setTimeout(() => j(new Error('e')), 500));

const helper = (gen) => {
const g = gen();
(function callNext(resolved) {
const next = g.next(resolved);
if(next.done) return next.value;
return next.value.then(callNext)
.catch(err => g.throw(err));
}());
};

helper(function* main() {
try {
const a = yield asynTask1();
const b = yield asynTask2();
console.log(a, b);
} catch(e) {
console.log('error happened', e);
}
});

If you are curious, you can take a look the co library for a more comprehensive implementation. We will however look at the async-await abstraction in the next section which is a native abstraction over generators for handling async flows.

Async Await

Async functions are higher level abstractions around generators and promises that can be used to simplify asynchronous flows. Any JavaScript function definition can be marked with the async keyword. When a function is marked as an async function, the returned value will always be wrapped in a promise. Consider the following simple function:

function add(a, b) {
return a + b;
}

We can simply mark the function as an async function by using the async keyword in the function definition:

async function add(a, b) {
return a + b;
}

You can also mark an arrow function as async:

const add = async (a, b) => a + b;

Now when we call the function we will get a promise that wraps the actual value. To access the actual value we use the then method and read the value through the callback argument:

const result = add(1, 2);
result.then(function(sum) {
console.log(sum);
});

Await Operator

The await operator can only be used inside an async function. When you place the await operator behind an asynchronous operation, the execution is "paused" until the result is available. Let's say that we want to read the content of a file and wait until we have the content. Then we want to write the content to another file only after the read operation has completed. To do that we can define an async function readWrite that awaits on each task in the body of the function:

// code/async-await/read-write-file.js
async function readWrite() {
const content = await readFile('./example.txt', 'utf-8');
const result = await writeFile('./example-copy.txt', content);
return result;
}

The readWrite function is marked async which means we can use the await operator in the function body. On line 1 we wait until reading the content of the file is finished. And then on line 2 we write to the file and then wait until it's finished. On line 3 we simply return the result of the write operation.

Now let’s look at another example involving a couple of async tasks that depend on each other. Below is the summary of the order of the operations to execute:

  1. Make a GET request to a public endpoint to get a post object
  2. List the content of a local directory and pick the file that ends with the .txt extension.
  3. Read the content of this file and and append it to the body of the post obtained from step 1
  4. Write the result to a file locally called final.txt

In the snippet below you can see how each async operation is accompanied with the await operator pausing the main function to wait until the result is available:

// code/async-await/post-body-example/main.js
async function main() {
const response = await axios.get('https://jsonplaceholder.typicode.com/posts/1');
const postBody = response.data.body;
const localFolderList = await readdir('.');
const textFiles = localFolderList.filter(onlyTextFiles);
const localFileContent = await readFile(textFiles[0], 'utf-8');
const finalResult = localFileContent + postBody;
const writeResult = await writeFile('./final.txt', finalResult);
return writeResult;
}

The code above looks synchronous, but it’s actually not. Nothing else is blocked while each task is being performed. That’s an important thing to remember, asynchronous code doesn’t block other operations that are scheduled to be executed, resulting in a more efficient execution.

Now let’s take the example above one step further. Let’s ask ourselves: what are the tasks that can be executed separately, and what are the tasks that absolutely need to be run in a certain order. In other words, how can we split this operation into two or more async parts? One possible option might be to split it up into two operations:

  1. Read folder content, get the text file, read the content of the text file
  2. Make GET request to get a post object

If you think about it, you don’t need any information from the http call to read the content of the local text file. However, to read the content of the local text file, you need to first get the folder content and filter the text files. Now that we have grouped the tasks, we can use the Promise.all pattern that we used in the Promises section to perform all the tasks:

// code/async-await/post-body-example/main-group.js
async function readLocalContent() {
const localFolderList = await readdir('.');
const textFiles = localFolderList.filter(onlyTextFiles);
const localFileContent = await readFile(textFiles[0], 'utf-8');
return localFileContent;
}

async function getPostObject() {
const response = await axios.get('https://jsonplaceholder.typicode.com/posts/1');
return response.data.body;
}

async function main() {
try {
const results = await Promise.all([readLocalContent(), getPostObject()]);
const finalContent = results[0] + results[1];
return writeFile('./final2.txt', finalContent);
} catch(e) {
return Promise.reject(e);
}
}

In the snippet above we have split up the two tasks into separate async functions allowing us to effectively run the two groups of tasks concurrently. In the main function we have used Promise.all to wait for both of the operations to finish and use the result to write the final content to a file.

Basic Error Handling

The interesting thing about async functions is that you can simply use a try catch block around a piece of asynchronous code and catch errors:

// code/async-await/read-write-file-catch-error.js
async function readWrite() {
try {
const content = await readFile('./example.txt', 'utf-8');
const result = await writeFile('./example-copy.txt', content);
return result;
} catch (error) {
console.log('An error happened while copying the file.');
return Promise.reject(error);
}
}

In the snippet above we wrap our async code with a try catch block. If an error happens in any of the steps, we can catch it and return a rejected promise. In later sections we’ll dive deeper into error handling but for now it suffices to say that for most cases you can use a try catch block to handle errors inside an async function.

Async/Await Inside Loops

Let’s say you need to perform a set of operations on a multiple local files. Your initial intuition might be use to use a looping mechanism like the forEach method:

const files = ['./a.txt', './b.txt', './c.txt'];
files.forEach(file => {
const r1 = await task1(file);
const r2 = await task2(r1);
});

There are two issues with the code above:

  1. The anonymous function passed to forEach is not marked async, so we can't use the await operator
  2. Even if we made the anonymous function async, the forEach method wouldn't wait for all the tasks to be finished

Instead of the forEach method, we have two options that would work:

  • c-style for loop
async function main() {
const files = ['./a.txt', './b.txt', './c.txt'];
for (let i = 0; i<files.length; i++) {
const r1 = await task1(files[i]);
const r2 = await task2(r1);
}
return 'done';
}
  • for of loop
async function main() {
const files = ['./a.txt', './b.txt', './c.txt'];
for (const file of files) {
const r1 = await task1(file);
const r2 = await task2(r1);
}
return 'done';
}

The downside of the above approaches is that we won’t be able to run the tasks concurrently, even though each tasks is asynchronous. For example, if the first file is huge, but the last file is very small, the function is going to be paused until all the operations on the first file is finished. A more efficient approach would be to define each operation as a promise and then process them all using the Promise.all method:

const files = ['./a.txt', './b.txt', './c.txt'];

async function operation(f) {
const r1 = await task1(f);
const r2 = await task2(r1);
return r2;
}

const tasksPromises = files.map(operation);

Promise.all(tasksPromises)
.then(r => console.log(r))
.catch(e => console.log(e))
.then(_ => console.log('all done'));

In the snippet above, we define an async function that performs the two tasks in the order that we like. Then we make an array of promises using the map method. Finally, we process the promises using the Promise.all method. Note that the promises will be resolved in different order, but the tasks in the operation function will be done in the order that we wanted. You can take a look at an actual example in the code folder of the repo at code/async-await/loop/main.js.

Conclusion

Asynchronous programming in JavaScript takes practice. Over time you will get better at it if you experiment with different abstractions, especially promises. All the abstractions out there build on top of promises, and because of that learning promises is essential to async programming in JavaScript. The best way to learn them is to come up with your own small programs, like reading or writing to files, or calling endpoints, etc. By doing that, you will have a mental map that can guide you when you get to writing larger programs. As a final note, always try to define your flow structures as promises, and use async/await sparsely and only use them in places where you think you can get a lot of value, otherwise just stick to promises.


Make sure to check out my other pocket references: JavaScript functions, JavaScript Prototypes.