Modern Javascript and Asynchronous Programming: Generators/Yield vs. Async/Await

Since the 6th version of the ECMAScript specification was released back in 2015, there have been several new features introduced or proposed for the JavaScript language that are focused on providing easier ways to handle asynchronous programming, or dealing with collections of objects (like iterables/iterators). These two topics overlap when considering asynchronous streams of data, the observable pattern, and reactive programming in general, with all the relationships between data producers and consumers. This is a very broad topic, so understanding first Generator Functions/Yield and Async/await can serve as a good introduction to some of these concepts.

Back in 2015, ES6 (ES2015) introduced the now very popular promises. A promise is an object that works like a contract, like an agreement that an asynchronous function will either resolve to a value or to an error. Promises are not the focus of this article, but they are really a foundation of the following topics, so make sure you understand them well first. There are excellent resources that explain promises in depth, one of my favorites is Mattias Petter Johansson’s Fun Fun Function video about Promises.

ES2015 also introduced Generator functions and the keyword yield. Generators can be used for both synchronous and asynchronous purposes, but it’s important to understand that they are the foundation on which the more popular async/await syntax is built on. Therefore, if you want to really understand async/await, you should first understand Generator functions, as well as the fact that they can be used for other use cases. Often tutorials start explaining generator functions by showing their usage equivalent to async/await, but this has some drawbacks. First, because generators functions can have other completely different uses which should be mentioned, and two, because it can make it more complicated to understand for someone who is not familiarized with them. Here I will start with the simplest example and then move on to their similarity with async/await.

The newer async/await syntax was initially proposed around 2015 for ES2017, to be able to write asynchronous code in a way that reads like synchronous code (without having to use .then() and callback functions). This is also described sometimes as “waiting code”. Async/await is a really useful way to create asynchronous code, which can be more compact and easier to understand than handling promises the regular way. However, they also have their limitations that are important to consider, which I’ll mention below. In some cases, it is still better to handle promises in the older fashion.

Generator Functions (ES2015)

A generator function is a function that can return multiple values, through intermediate “returns” marked by the keyword yield. The generator function pauses its execution at each of these points until it would eventually reach its final return, using the regular “return” keyword, and the function ends. Generator functions are declared by adding an asterisk after the function keyword.

This is a very simple example of a generator function. The first time that a value is requested from this generator (we’ll see how to execute it below) it will return the value 5, the next time it will return the value 7, and finally it will return the value 11.

An interesting aspect of generators is that they provide multiple values on demand, one at a time: they only continue their execution up to the next yield or return when someone else requests it. This is a concept similar to the observable/observer design pattern, except that generator functions act as a passive producer of values, instead of an active one. An active producer would emit values when it wishes, instead of waiting for someone else to request it. The piece of the code that requests values from the generator function is the active consumer, because it takes the initiative to request values, it isn’t just listening to receive passively.

So how are they executed?

To achieve this, when you call a generator function it doesn’t actually run the code of the function, it will immediately (synchronously) return a special type of object called a Generator object. This object functions as an iterator, among other things, so for simplicity’s sake we’ll refer to it here as “iterator”. As the name suggests, it can iterate through a collection of values, in this case, the different “returns” that the generator function can provide (marked by either “yield” or the final “return” keyword). Then, to run the function up to its first “intermediate return”, you call the method next() on the iterator.

In general, iterator.next() executes the generator function up until its next stopping point, whether that’s an intermediate yield statement or the final return. Calling next() always produces an object with a value property and a done property, which is a boolean. This way, you can get the value that you wanted, but you can also know if you can request more values or not.

In this example, you can see the generator function being executed, which returns the iterator object. We then use the next method of the iterator to get the values, calling it three times. I’m using an inline evaluation tool (Quokka) that displays the values of each variable, lines 13–15. With the third iteration we receive "done:true" to indicate that no more values can be requested.

In real life use cases you probably wouldn’t know beforehand how many times to request values, so a usual implementation would be to keep requesting values as needed until you receive "done:true", for example using a while loop. A loop could also be used inside the generator function itself, to keep yielding items one at a time, for example, values from a long list, as long as there are more items to yield. The generator function could even continue providing values infinitely, from sequences that can be calculated that don't have an end (for example, the Fibonacci sequence).

A generator function is similar to a state machine (a function that keeps track of its own state), because it remembers the progress of its execution. This is one of the aspects that makes generators useful. A generator function can actually be implemented in ES5 through the use of closures, to return an iterator object that will remember its own state. This is, however, more complicated to write, and it would have to be adapted to each use case. See this ES5 example (with the exception of using const and let, but those can be replaced with var without changing the example’s behavior):

This “fakeGenerator” function works exactly in the same way as the real one, for our limited purposes. It returns an object with a “next” method, that can return values keeping track of which value to return, in this case, thanks to a closure.

Async Functions (ES2017)

Now, how do generator functions and yield relate to async/await? Async/await is a newer proposal for the upcoming versions of the Javascript language, defined formally in ES2017, which also allows you to write functions that can be paused, but in a more specialized way than generators. Keep in mind that async/await makes it easier to implement a particular use case of generators. Generator functions/yield and Async functions/await can both be used to write asynchronous code that “waits”, which means code that looks as if it was synchronous, even though it really is asynchronous; it doesn’t use callbacks.

We’ll first start with generators: Since yield stops the execution of a generator function at a certain point, it can allow for waiting for async requests to complete before continuing. Consider the following example:

Let’s say in our application we have an “init” function that will retrieve any necessary information from the backend, for example fetching users, which would be an asynchronous method since it involves an XHR request. If we use promises, we would have to define a callback function to execute when the promise resolves:

With generator functions, we can use yield to wait for the async function “getUsersFromDatabase” to resolve, and then just set the returned value as our users:

This definitely looks easier to read, but it doesn’t work quite so simply. As the term implies, the keyword “yield” really yields the control of execution to whoever called the generator function. This means there must be an external function that is in charge of receiving the yielded promise, waiting until it resolves, and then pushing the result back into the generator function so that it can resume execution and assign that value to the users variable.

This introduces an aspect of generator functions that we hadn’t seen before: At any of the “stopping points” in a generator function, not only can they yield values to an external function, but they also can receive values from outside. This allows for a back-and-forth communication between two pieces of code.

Passing values into a generator function is achieved by passing them as arguments into the iterator’s “next” method. The following example is an implementation of how the “external” code that calls the generator function is in charge of resolving any promises and sending the resolved value back into the generator:

The implementation of getUsersFromDatabase is not important, it just returns a promise that resolves after two seconds returning the string “Test Users”. Note how the external code (lines 6–30) is in charge of calling the generator (lines 6–7), handling the resolved value of the promise (callback starting at line 16), and sending the value back into the generator (line 20), in order to get the generator’s final value. Lines 13, 18 and 28 are just there to show the values of the variables at that point, indicated by the dark blue text added by the inline evaluation tool.

This is a simple scenario because the code is implemented specifically for a generator function that yields only one value. Ideally, the external functions should be able to handle any number of yielded promises until the generator reaches its final return. This functionality is available as third-party libraries, to be able to write asynchronous waiting code in this fashion with generators/yield without having to worry about how the external function handles and resolves the promises. Those libraries can provide a function that can receive a generator function as an argument, executing the generator and handling any promises yielded.

This lengthy example serves to show why Async/await is a very useful feature: It lets you write async waiting code like our generator function example (lines 1 to 4), but it doesn’t need any external helper function to handle the promises! With async/await, you can just write the example like this:

The asterisk notation of generators is replaced by placing the keyword async in front of the function declaration, and “yield” is replaced by “await”. Await can be placed in front of any statement that returns a promise, and the await keyword can only be used inside functions that are marked by the async keyword before their declaration. Now when this test function is executed, it will pause at the await keyword, wait for the promise to resolve, and then automatically assign the resolved value to const users, all without the help of any other function.

Async/await is then really useful to write code dealing with promises without having to call .then() methods, declaring callbacks, and all the nesting that it involves (the infamous pyramid of doom). It seems like a great solution, but there are some important considerations that must be taken into account before using them. In some cases it might be better to just stick with the old way of resolving promises with .then(). You should consider the following:

1) Async functions always return a promise: Since async functions pause their execution at any “await” keyword to wait for asynchronous expressions to resolve, they themselves become asynchronous (and hence why they have the async keyword in front of them). That means that regardless of what you return from that function, functions with the async keyword will always return a promise, that will resolve to whatever you returned, or throw an error. In the previous example, the function “test” returns the string “Test Users Correctly received”, but in reality it will actually return a promise that will resolve to that string. This is important to keep in mind when designing your code and thinking about how other parts of the code are going to interact with any given function, whether they are expecting to receive a promise or not.

2) Await can only wait for promises sequentially, not in parallel. The await keyword can only wait for promises one at a time, not several at once. Therefore, if you have to deal with several promises and you want to wait for them with await, you’ll have to wait for each one to complete before moving on to the next.

This is much slower than processing multiple promises at the same time, although there are cases where it could be the best decision, for example, if you don’t want to overload a network by sending hundreds of requests at the same time.

Assume that “users” is an array of users predefined elsewhere, and that “getProfileImage” returns a promise that resolves to a profile image. This example iterates through each user, pausing at each iteration to push the image into a “profileImages” array. It only continues to the next iteration when the promise from the current iteration is resolved.

An alternative if you want to handle several promises at the same time is to combine the use of await and regular promises. For example, you could unify a group of promises with the method “Promise.all”, which will return a single promise that will resolve whenever all of the promises resolve (or throw an error if any of them fail). Then you can use await just on that single unified promise:

This example iterates through the users array using “map”: For each iteration, it executes “getProfileImage” and returns the pending promise. Notice that map doesn’t pause at each iteration, it just returns an array holding all the promises. Then we can unify all those promises using “Promise.all”, and we use await to pause only at that point, to wait until the single promise resolves.

Keep in mind - Generators and async functions always return a specific type of object:
- Generator functions: If you yield/return a value X, it will always return an iteration object with the form {value: X, done: Boolean}
- Async functions: If you return a value X, it will always return a promise that will either resolve to the value X or throw an error.

Conclusion

Generators are pausable functions that can generate (yield) values on demand, whenever the iterator object requests the next value. In this sense, they are a passive producer, while the iterator is the active consumer (because it’s the one that takes the initiative to request a value). This is in contrast to the usual observer pattern, where there’s an active producer (the observable/subject) that emits values when it wishes, and one or more passive consumers (the observers) that are just listening to receive values. A possible use of generator functions is to yield values from a long series one at a time, possibly being able to produce infinite values as needed.

One particular use of Generator functions is to write async code that looks synchronous (code that “waits”), by yielding any promises, but they require the help of another function that would have to handle all the promises. It is better to use async/await for this purpose, since they are able to write code in this way without needing any helper function.

Async functions and the await keyword are a great way to write asynchronous code that "waits", but they cannot wait for multiple promises at the same time, so it's a good idea to fall back to normal promises when await becomes a limitation.