How Node and Javascript handle asynchronous functions

https://www.flickr.com/photos/christopherbrown/9798713456

During the short time that I’ve been studying Javascript, I’ve been confused about some (many) things. What is a closure and when do I use them? What is a higher-order function? Why are some functions referred to as ‘callback’ functions? From the articles I’ve seen posted about the more confusing parts of Javascript, I’m not alone in my lack of understanding about some of these things.

When I encounter one of these topics that I don’t understand, I don’t run off into the hills and hide under a rock. There are plenty of other, better reasons to do that these days. No, I take a breath, realize that my not understanding can be fixed with a little bit of time and effort and then go about finding the answers to whatever questions I’m asking.

One of the questions that came up recently is what exactly is a ‘callback’ function? Why are they called that? How are they different from a non-callback function? I get that Javascript calls the function at a later time, but how does it do that? How does it know what to call when?

Callback functions are part of what is referred to as ‘asynchronous’ operations in Javascript. Asynchronous operations are those that are done in a different order than how they are listed in the program. Javascript has the ability to handle functions this way so that none of them, if written correctly, become a ‘blocking’ function.

A blocking function can be bad since everything in the program stops and waits for that function to finish before it can continue. They can also be necessary since sometimes you have to wait for one part of a program to finish before continuing on to the next.

This necessity happens most often with programs that make requests over the network to retrieve data. These requests take a non-zero amount of time, so Javascript must wait for the data to be returned before it can act on it. If the program just continued without waiting then this would throw errors from undefined variables or empty objects.

A synchronous example

A very basic synchronous program would be constructed something like this:

let asyncTest = () => {
    console.log('first');
    console.log('last');
}
asyncTest();

Nothing complex here. ‘First’ is always going to be logged to the console before ‘last’. There is nothing asynchronous going on here.

An asynchronous example

In order to mimic something like an asynchronous call we can use the ‘setTimeout’ method. This takes two arguments; the first is a function which will be a callback function, and the second is the amount of time to delay calling that function in milliseconds.

Adding setTimeout() to the above looks like this:

let asyncTest = () => {
    console.log('first');
    setTimeout(() => console.log('second'), 2000);
    console.log('last');
}
asyncTest();

Now, the output from running this is a bit different.

Here, the console of ‘second’ happened after ‘last’. Why? Because that console.log() statement is in a callback function which gets executed out of order. And since Javascript is non-blocking it continues on executing more functions. In this case, the ‘last’ console.log() gets executed and then finally the callback function is fired and ‘second’ is seen on the screen.

You might think that the callback gets executed out of order due to the delay in setTimeout(). That’ what I thought at first, but take a look at this example:

let asyncTest = () => {
    console.log('first');
setTimeout(() => console.log('second'), 0);
console.log('last');
}
asyncTest();

Here, the delay is set to zero, but the output is still the same:

What is going on here? If there is now zero delay, why did ‘second’ show up after ‘last’?

Here’s a less contrived example. Here I’m using the Axios library to retrieve some data from the Google maps API. I’m also using the dotenv library to import my API key from the process.env object. (You can read previous articles on Axios and dotenv for more info)

const axios = require('axios');
require('dotenv').config();
let asyncTest = () => {
    console.log('first');
    let url = `https://maps.googleapis.com/maps/api/js? key=${process.env.GOOGLE_API}&callback=initMap`
    axios.get(url)
        .then((response) => {
            console.log('Google map data received', response)
        })
        .catch((error) => {
            console.log('ERROR!')
        });
    console.log('last');
}
asyncTest();

Here is the output:

As you can see, we still get the ‘first’ and ‘last’ logged to the screen before the data from the Google API is returned. This is because when data is returned from the Axios call, it is in the form of a Promise and that Promise can have callback functions applied to it.

These are a couple of examples of how asynchronous functions work in Javascript, but they don’t answer the question of why they work that way. What makes these functions behave this way? To answer that, we’ll have to examine a few behind-the-scenes pieces of Javascript. Namely, the call stack, the event loop, and the callback queue.

The call stack, event loop and callback queue

The call stack is a mechanism in Javascript used to keep track of where it is in a program. It knows which functions are currently being run and which to call next. The catch with the call stack is that you can only add things to the top of the stack. It’s a last-in-first-out (LIFO) idea.

When a function is called, it is added to the top of the stack and run. If that function calls another function, it is then added to the top of the stack and then it is run. Only the function at the top of the stack is the one that can be run.

Here is what the call stack would look like through the first example above. The main() is just a wrapper being used by Node.js and stays in the call stack as long as the main (non-callback) part of the program is running.

The function gets added to the stack, then the statements inside the function get added as they are called and removed as they are executed. Finally, main() is the only thing left and, in this case, the program is finished.

In an asynchronous program, there are a few more pieces that come into the mix. The first is the callback queue. This is a kind of holding area where callback functions go once they are called. The callback queue works together with the event loop to execute callback functions at the right time. The event loop checks the call stack to see if it is empty. If it is then the first callback function in the callback queue gets executed, then the second etc. If the call stack is not empty, then those callback functions wait.

This is what causes output from a callback function to appear after all of the other non-callback functions. The non-callback functions are like VIPs that don’t have to wait in line to get executed. Meanwhile, those callback functions are being held in a separate queue until it is their turn.

Here is the workflow for the axios call from above

• main() goes into the call stack
• asyncTest() gets added to the top of the call stack
• console.log('first') gets added to the top of the call stack
• it is executed and removed from the call stack
• axios.get() is called
• assuming the data request goes through, .then() is called which is a callback and it gets added to the callback queue
• the event loop checks the call stack and since it is not empty, the functions in the callback queue stay there
• axios.get() is finished and gets removed from the call stack
• at this point, the main part of the program continues and console.log('last') gets added to the call stack, executed, and removed from the call stack
• now main() is complete and the call stack is empty
• the event loop sees that the call stack is empty and starts executing the functions in the callback queue
• .then() is now moved from the callback queue to the call stack and the contents of the data request are logged to the screen
• at this point, all functions have executed and the program is complete

This is why callback functions show up after all of the functions in the main part of the program. Because they have to go through the callback queue and only get put into the call stack by the event loop once the call stack is empty.

The execution order of asynchronous actions along side synchronous actions can seem like a bit of a mystery when you first start using Javascript. But the built in workflow and mechanisms of the call stack, event loop, and callback queue are there to handle these requests at appropriate times.

I hope this has helped shed a little bit of light on how Node and Javascript handle these processes. Thanks for reading.