Use “await” wisely in Test Automation

Published in

Globant

15 min readFeb 2, 2024

With the development of web technologies, there is a substantial increase in browser automation frameworks in the Node.js environment. These frameworks give many features to support the latest web applications and end-to-end automation, and in common, all are asynchronous. Test automation frameworks like Playwright, WebdriverIO, and Cypress leverage asynchronous operations to efficiently interact with web browsers, handle network requests, manage element locators and waits, support parallel execution, and utilize promise-based architectures for clean and structured code. The asynchronous nature aligns with the characteristics of web environments, promoting efficiency and flexibility in automation.

In regular practice, test automation engineers use the await keyword to call the asynchronous framework functions and make the test flow synchronous. While this gives better readability and handling of promises, it can increase the execution time if not used wisely.

In this article, we will learn the efficient use of asynchronous programming in test automation rather than just calling await in every line. We will learn to optimize await calls in the code to increase the efficiency of the test script, thus optimizing the execution time. Improper use of asynchronous functions can also lead to freaky tests; we will learn how to create stable automation tests, too.

What is Asynchronous Programming?

Let’s understand this with a real-life experience. Assume you are at the food counter and ordered items like a burger, pizza, fries, and a hotdog. Now, if the chef prepares these items one by another, you must wait longer to get the order ready. However, as these items can be prepared in isolation and parallel, the chef can do the same to save time. Therefore the order will be ready in less time.

Similarly, the computer executes functions one after another in synchronous programming, so all functions get executed in the order they are invoked. In this case, the total execution time to complete the job is the sum of all individual functions.

In the above example, the total execution time to complete three functions will be t1+t2+t3 secs. If these functions take longer to execute, then the total time will be expensive. If these functions are not interdependent, executing them in parallel would be a good idea.

Asynchronous programming enables the execution of independent functions parallel in isolation and the collaboration of results after task completion. This approach is beneficial for handling operations with variable durations, such as I/O tasks, network requests, or user interactions, to optimize performance and responsiveness. So, asynchronously, we can execute the functions below.

JavaScript and Asynchronous Programming

JavaScript supports asynchronous programming through features like callbacks, Promises, and the async/await syntax.

Callbacks: Callbacks are functions that are passed as arguments to other functions and are executed later, often after the completion of an asynchronous operation.

function fetchData(callback) {
 // Simulating an asynchronous operation
 setTimeout(function () {
     callback("Data fetched successfully");
 }, 1000);
}


fetchData(function (result) {
 console.log(result);
});

Promises: Promises provide a cleaner and more structured way to handle asynchronous operations. A Promise represents a value that might be available now, in the future, or never. It has three states: pending, fulfilled, or rejected. Promises allow chaining and handling errors more effectively compared to callbacks.

function fetchData() {
 return new Promise(function (resolve, reject) {
   // Simulating an asynchronous operation
   setTimeout(function () {
     resolve("Data fetched successfully");
   }, 1000);
 });
}

fetchData()
 .then(function (result) {
   console.log(result);
 })
 .catch(function (error) {
   console.error(error);
 });

Async/Await: async/await is a syntactic sugar built on top of Promises, making asynchronous code look more like synchronous code. The async keyword is used to define a function that returns a Promise, and the await keyword is used to pause the execution until the Promise is resolved. Async/await simplifies the syntax and improves code readability, especially when dealing with multiple asynchronous operations.

async function fetchData() {
 return new Promise(function (resolve) {
     setTimeout(function () {
         resolve("Data fetched successfully");
     }, 1000);
 });
}

async function fetchDataAndLog() {
try {
     const result = await fetchData();
     console.log(result);
 } catch (error) {
     console.error(error);
 }
}

await fetchDataAndLog();

Usage in Test Automation

Asynchronous programming is crucial in test automation, especially when dealing with tasks that involve waiting for external resources, reading HTML text, handling parallel execution, or managing non-blocking operations. Here are some key areas where asynchronous programming can be applied in test automation frameworks:

File I/O operation: Asynchronous programming can be applied when dealing with file I/O operations, such as reading and writing log files or test data. This helps prevent blocking the test execution flow while waiting for file operations to complete. Reading test data from a file and waiting for page load to be performed asynchronously.
API Calls: API calls are asynchronous, so there is a good scope for practicing asynchronous programming to test APIs. Let’s have a look at the following scenarios.
— In an integration test scenario, if two or more independent API calls are required to complete the test flow, we can invoke those independent APIs asynchronously.
— If we are generating test data or completing the pre-requisite of a UI test through API calls, then APIs can be called asynchronously with the UI operations. For example, API calls and loading of the webpage in the browser should be performed asynchronously.
Database queries: Like API calls, Database queries are also asynchronous. So, in a test scenario where multiple database calls are needed, we can perform those asynchronously. This can be useful in ETL test automation to verify the target Databases, data integrity, and consistency.
Browser automation: When using browser automation tools like WebdriverIO or Playwright, web pages often have asynchronous operations. Asynchronous programming helps wait for these operations to complete before proceeding with the test.

Case study with Playwright

For this article, I have chosen the Playwright framework with TypeScript. Playwright is an open-source testing framework designed to automate end-to-end testing of web applications. Microsoft developed it and is mainly focused on providing a high-level API for browser automation.

The concepts and examples explained here hold for other asynchronous test frameworks like WebdriverIO, Cypress, etc.

Use case #1: Asynchronous assertion

First, let’s see how asynchronous assertion helps the performance of execution.

Let’s consider this simple test case where I validate ten web page values. This is a simple way of doing this in Playwright or any other similar framework.

test("Verify the studio list", async ({ page }) => {
  await page.goto("https://www.globant.com/our-services#studios-list")
  const studioLocator = `.accordion__item-container--desktop 
                         li.studios-list--item.column-1:not(.hidden) button`
  const menuItems = page.locator(studioLocator)
  await expect(menuItems.nth(0)).toHaveText("Business Hacking")
  await expect(menuItems.nth(1)).toHaveText("Sports")
  await expect(menuItems.nth(2)).toHaveText("Media & Entertainment")
  await expect(menuItems.nth(3)).toHaveText("Finance")
  await expect(menuItems.nth(4)).toHaveText("Smart Payments")
  await expect(menuItems.nth(5)).toHaveText("Airlines")
  await expect(menuItems.nth(6)).toHaveText("Healthcare & Life Sciences")
  await expect(menuItems.nth(7)).toHaveText("Automotive")
  await expect(menuItems.nth(8)).toHaveText("Edtech")
  await expect(menuItems.nth(9)).toHaveText("Games")
})

In the above code, first, we hit the website and create a locator using a CSS selector, which will point to all studio names in the first column. Then, by their index order, we validate the texts.

While this looks fine, if we observe no interdependence of the ten assertions, they can be executed asynchronously. The following sample shows asynchronous assertions.

test("Verify the studio list", async ({ page }) => {
  await page.goto("https://www.globant.com/our-services#studios-list")
  const studioLocator = `.accordion__item-container--desktop 
                         li.studios-list--item.column-1:not(.hidden) button`
  const menuItems = page.locator(studioLocator)
  await Promise.all([
    expect(menuItems.nth(0)).toHaveText("Business Hacking"),
    expect(menuItems.nth(1)).toHaveText("Sports"),
    expect(menuItems.nth(2)).toHaveText("Media & Entertainment"),
    expect(menuItems.nth(3)).toHaveText("Finance"),
    expect(menuItems.nth(4)).toHaveText("Smart Payments"),
    expect(menuItems.nth(5)).toHaveText("Airlines"),
    expect(menuItems.nth(6)).toHaveText("Healthcare & Life Sciences"),
    expect(menuItems.nth(7)).toHaveText("Automotive"),
    expect(menuItems.nth(8)).toHaveText("Edtech"),
    expect(menuItems.nth(9)).toHaveText("Games"),
  ])
})

To make these assertions asynchronous here, we just passed all expectations into Promise.all() function in an array. So here we are using a single await instead of 10. Therefore, all expected statements will be executed in parallel in the above code, and the control flow will wait until all promises are fulfilled. It will stop and throw an error if any of the expectations fails.

Fewer lines of code can achieve the same result. Let’s have a look at it.

test("Verify the studio list", async ({ page }) => {
  await page.goto("https://www.globant.com/our-services#studios-list")
  const studioLocator = `.accordion__item-container--desktop 
        li.studios-list--item.column-1:not(.hidden) button`
  const menuItems = page.$$(studioLocator)
  const actualValues = await Promise.all(
    (await menuItems).map((item) => item.innerText()),
  )
  const expectedValues = [
    "Business Hacking",
    "Sports",
    "Media & Entertainment",
    "Finance",
    "Smart Payments",
    "Airlines",
    "Healthcare & Life Sciences",
    "Automotive",
    "Edtech",
    "Games",
  ]
  expect(expectedValues).toEqual(actualValues)
})

In the above code, we have used $$instead of thelocatorfunction to point the elements. The $$ function finds all elements matching the specified selector in the webpage and returns a promise-array of elements. await menuItems will wait to resolve and produce an array of elements. Then, using a map function, we map the elements to innerText function. The innerText function is asynchronous and Promise.all() will wait for all to generate an array of texts. Ultimately the actualValues will be anarray of actual texts from the webpage. Then, we can easily compare two arrays at the end.

To compare the execution time benefit, each of the above code blocks was executed 100 times in a row, and the execution time was recorded. Then, those were mapped in a histogram chart for analysis. The horizontal axis represents execution time, and the vertical axis shows the count of execution finished in a bucket size.

The leftmost column of the histogram of sequential await shows more than 26 executions were completed between 50–60 milliseconds. The next column shows more than 50 executions were completed between 60 and 70 milliseconds. If we sum up the count of the leftmost two columns, nearly 80 executions completed between 50–70 milliseconds in sequential await. This means that 80% of the executions were finished between 50–70 milliseconds.

The two leftmost columns in the above histogram show nearly 80 executions completed in less than 50 milliseconds when we executed them asynchronously.

The above histogram shows code optimization improved the execution a little more, and here, nearly 20% of execution was completed in less than 40 milliseconds and 80% within 50 milliseconds.

Let’s also have a look at the following table with statistics.

We can see that the minimum execution time of all 100 runs of sequential await is 54 milliseconds compared to 36 or 38 milliseconds of asynchronous executions. The average of both asynchronous executions is 50 milliseconds compared to 68 milliseconds for sequential. So, reading and validating ten values from a webpage asynchronously takes 26.5% less time than the equivalent sequential.

It is clear from the histogram analysis and the above statistics that the asynchronous assertion is much faster than the sequential execution. Therefore, we should always read and assert texts from webpages asynchronously in test automation.

Use case #2: Flaky tests

Inappropriate use of await and asynchronous functions can lead to flaky tests. Asynchronous functions try executing all independent statements inside it parallelly. This feature will cause inconsistent results in test execution. Let’s understand this with the following example, where we want to validate web table values with a stored .tsv file with expected values.

I have created the following function to read the .tsv file and generate a 2D array. I created this method asynchronously so that it takes less time to execute. It takes less than two milliseconds to read values from the file. Readers of this article can try to create a synchronous read method and comment on execution time.

async function getExpectedData(filePath: string): Promise<string[][]> {
 try {
   const fileContent: string = await fs.promises.readFile(filePath, 'utf8');
   const lines: string[] = fileContent.trim().split('\n');
   const tsvData: string[][] = lines.map(line => line.split(/\t/g));
   return tsvData;
 } catch (error) {
   throw new Error(`Error reading CSV file: ${error.message}`);
 }
}

Now, look at the following test method to read the web table and compare the values with the file.

test('Validate table values', async ({ page }) => {
 await page.goto('https://www.datatables.net/examples/ajax/deep.html');
 await page.locator('[name="example_length"]').selectOption("100")
 const tableRowElements = await page.$$('#example tbody tr')
 const actaulValues: string[][] = []
 for (let i = 1; i <= tableRowElements.length; i++) {
   const rowCells = await page.$$(`#example tbody tr:nth-child(${i}) td`)
   const rowData: string[] = []
   for (const cell of rowCells) {
     const text = await cell.innerText()
     rowData.push(text)
   }
   actaulValues.push(rowData)
 }
 const expectedValues = await getExpectedData(pathToExpectedResult);
 expect(expectedValues).toEqual(actaulValues)
});

The first two lines of the above test are to open the webpage and select 100 in the dropdown to expand the table. The actualValue 2D array is defined to hold the cell values of the table. tableRowElements is holding the row elements of the table. Then, inside the first for loop ,rowCells is holding all cell elements of each row. The inner for loop is iterating over the cells and getting the texts from each cell to create an array. After the inner for loop the array is getting pushed to actualValues array.

The above function looks fine and is working. If we execute the same, it will also work most of the time. However, this function is Flaky!

The reason is that if the file read operation completes before the completion of the for loop to read texts, the expect function will be executed. That will fail the test as tableRows will be empty then. Let’s check the solution in the following use case.

So, we have to take utmost care when using loops and creating arrays in the asynchronous functions so that it does not result in an unstable function.

Use case #3: Asynchronous read HTML table

Another issue with the above function is it reads the cells synchronously, so it takes ~3 seconds to get the inner text of all cells in the web table.

The following code will be stable and execute faster.

test("Validate table values", async ({ page }) => {
  await page.goto("https://www.datatables.net/examples/ajax/deep.html")
  await page.locator('[name="example_length"]').selectOption("100")
  const actualUIValuesPromise = async (): Promise<String[][]> => {
    const tableRows: string[][] = []
    const tableRowElements = await page.$$("#example tbody tr")
    const rowCount = tableRowElements.length
    const colCount = (await tableRowElements[0].$$("td")).length
    const tableCellLocators: Locator[] = []
    Array.from({ length: rowCount }, (_, i) =>
      Array.from({ length: colCount }, (_, j) => {
        tableCellLocators.push(
          page.locator(
            `#example tbody tr:nth-child(${i + 1}) td:nth-child(${j + 1})`,
          ),
        )
      }),
    )
    const cellValues = await Promise.all(
      tableCellLocators.map((eachCell) => eachCell.innerText()),
    )
    for (let i = 0; i < cellValues.length; i += colCount) {
      tableRows.push(cellValues.slice(i, i + colCount))
    }
    return tableRows
  }
  const results = await Promise.all([
    actualUIValuesPromise(),
    getExpectedData(pathToExpectedResult),
  ])
  expect(results[0]).toEqual(results[1])
})

Here, I have created a new asynchronous function actualUIValuesPromise to complete the read operation from the webpage. In this function, first, I am getting the column and row counts. Then, create an array tableCellLocators which holds the selector value of each cell. Once these locators are created by Promise.all() function the actual cellValues array is populated. Now, reformatting this 1D array to 2D array using another for loop to compare with the expected values. After restructuring the array return the 2D array. So now we have two asynchronous functions: to read values from the webpage and to read from the file. Now, passing these two function invocations to a Promise.all() function helps to execute both asynchronously. This will return an array with promise-settled values. Once both functions' promises are settled, match results using the expect function.

In the above example, we have obtained parallel execution in two places. Firstly, the UI read and file read operations are independent so that we can execute both in parallel. Secondly, we read the cells in parallel by creating an array of all cell locations and waiting for the text.

This function is executed faster. The average execution time of this method is ~1.5 sec, a 50% reduction from the previous approach. Also, as the expected function executes only after the proper population of actual values, this approach is robust, too.

Therefore, we must ensure the asynchronous reading of the values for the performance when dealing with web tables. Also, we need to ensure the object storing the values is populated before any other operation on it.

Use case #4: API calls

As we have observed in previous cases, clubbing independent calls together helps to improve execution time. We can implement the same concept for API testing as well. Let’s assume an API integration test with 3 APIs involved.

GET API_A
GET API_B
GET API_C

And to call API_C, we need values from the responses of API_A and API_B, and API_A and API_B are independent. Synchronous scripting will look like the below.

const respA = await getAPI_A()
const respB = await getAPI_B()
const respC = await getAPI_C(respA, respB)

We can improve it like below.

const respA = getAPI_A()
const respB = getAPI_B()
const responses = await Promise.all([respA,respB])
const respC = await getAPI_C(responses[0], responses[1])

The script will wait for API_A and API_B together and thus will be executed faster.

Therefore, in any scenario, if there are independent API calls, we should invoke those APIs asynchronously to reduce the execution time.

Use case #5: Error in Asynchronous promises

The Promise.all() rejects when any of the input’s promises rejects, with the first rejection reason. Hence, assertions in use case#1 will stop on the first assert failure. We will get an incomplete result if there is more than one assertion failure. In the following example, I have changed three expected values in the assertions to generate failures.

  await Promise.all([
    expect(menuItems.nth(0)).toHaveText("Business Hacking"),
    expect(menuItems.nth(1)).toHaveText("Sports"),
    // Added extra string to create error
    expect(menuItems.nth(2)).toHaveText("Media & Entertainment - Error"),
    expect(menuItems.nth(3)).toHaveText("Finance"),
    // Added extra string to create error
    expect(menuItems.nth(4)).toHaveText("Smart Payments - Error"),
    expect(menuItems.nth(5)).toHaveText("Airlines"),
    expect(menuItems.nth(6)).toHaveText("Healthcare & Life Sciences"),
    expect(menuItems.nth(7)).toHaveText("Automotive"),
    expect(menuItems.nth(8)).toHaveText("Edtech"),
    // Added extra string to create error
    expect(menuItems.nth(9)).toHaveText("Games - Error"),
  ])

The following is the error output mentioning the first without any information about the other two failure points.

Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)
Locator: locator('.accordion__item-container--desktop li.studios-list--item.column-1:not(.hidden) button').nth(2)
Expected string: "Media & Entertainment - Error"
Received string: "Media & Entertainment"

We can overcome this problem by Promise.allSettled() function. This function waits until all promises are settled(resolved or rejected). The benefit of this approach is that the test will not stop at the first failure point and will validate everything. So, if multiple failure points exist, all will be captured in a single execution.

This returns an array of objects that describe the outcome of each promise. The point to note here is that unlike Promise.all() function Promise.allSettled() function never fails. If all of its promises are rejected, then it will generate an array with all rejected promise objects. We have to add another expect function to check if there is any error. Let’s check the following code.

const results = await Promise.allSettled([
    expect(menuItems.nth(0)).toHaveText("Business Hacking"),
    expect(menuItems.nth(1)).toHaveText("Sports"),
    // Added extra string to create error
    expect(menuItems.nth(2)).toHaveText("Media & Entertainment - Error"),
    expect(menuItems.nth(3)).toHaveText("Finance"),
    // Added extra string to create error
    expect(menuItems.nth(4)).toHaveText("Smart Payments - Error"),
    expect(menuItems.nth(5)).toHaveText("Airlines"),
    expect(menuItems.nth(6)).toHaveText("Healthcare & Life Sciences"),
    expect(menuItems.nth(7)).toHaveText("Automotive"),
    expect(menuItems.nth(8)).toHaveText("Edtech"),
    // Added extra string to create error
    expect(menuItems.nth(9)).toHaveText("Games - Error"),
  ])
expect(results.filter((result) => result.status === "rejected")).toHaveLength(0)

Here is the output.

Error: expect(received).toHaveLength(expected)
Expected length: 0
Received length: 3
Received array:  [{"reason": [Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)·
Locator: locator('.accordion__item-container--desktop li.studios-list--item.column-1:not(.hidden) button').nth(2)
Expected string: "Media & Entertainment - Error"
Received string: "Media & Entertainment"
Call log:
…
], "status": "rejected"}, {"reason": [Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)·
Locator: locator('.accordion__item-container--desktop li.studios-list--item.column-1:not(.hidden) button').nth(4)
Expected string: "Smart Payments - Error"
Received string: "Smart Payments"
Call log:
…
], "status": "rejected"}, {"reason": [Error: Timed out 5000ms waiting for expect(locator).toHaveText(expected)·
Locator: locator('.accordion__item-container--desktop li.studios-list--item.column-1:not(.hidden) button').nth(9)
Expected string: "Games - Error"
Received string: "Games"
Call log:
  -...
], "status": "rejected"}]

The above error log captured all three expected failures with the expected and received strings. This approach will identify all errors by a single test execution.

There is another function to handle multiple promises which is Promise.any(). This function accepts an iterable(e.g. array) of promises and fulfills when any of the promises in the given iterable fulfils. The fulfillment value is the fulfillment value of the first promise that was fulfilled. The return value is rejected when all of the promises in the given iterable reject. The rejected value is an AggregateError containing an array of rejection reasons.

This Promise.any()function can be helpful in scenarios with random popups on the homepage of any application. In some cases, the homepage may have popups that can’t be predicated and depend on multiple factors beyond the control of a tester. So, we have to close those popups before acting on the home page elements. We can create two locators to automate such a web page. One for a homepage element and another for the popup. Then, pass those two promises to the Promise.any(). If the homepage locator promise resolves first, then we are good to continue with homepage actions. Otherwise, if the popup locator resolves, we know the popup is there, so action to close the popup is to be performed.

Therefore, handling possible errors in asynchronous programming is important for better reporting test failures. Promise resolving functions like Promise.all(), Promise.allSettled() and Promise.any() should be used appropriately to produce the best result in case of any rejection of input promises.

Summary

Choosing an asynchronous framework for test automation means more than just writing steps with await calls. The real strength of these frameworks shines when we embrace an asynchronous mindset. It’s crucial to grasp how our tests flow and identify step dependencies. Then, script accordingly with the appropriate use of Promise.all(), Promise.allSettled() and Promise.any(). By understanding these aspects, we can make the most of frameworks like Playwright, WebdriverIO, and Cypress, ensuring our tests run smoothly and efficiently.