Puppeteer best practices
These helped me reduce automation flakiness
Introduction
Puppeteer is pretty easy to use as it provides many APIs to interact with the dom and carry out complex automation and testing. But for beginners, there are some complexities that the developers might not be aware of.
So here I am sharing some best practices (in my opinion) that I learned from my experience. Let's dive in!
Search using text/content instead of CSS selectors
The CSS classes or ids used in third-party websites change frequently. Most well-known websites use random encrypted/hashed strings which might get in the way of using CSS selectors.
The CSS can also get changed when the developers are refactoring a website. So, it is best to use the text content of the HTML elements as the source of truth.
For example, you might have to click a login button. So instead of selecting that button using CSS selectors, use the button text “Login” instead.
Here are some ways for achieving the above:
// using XPath api, the easiest wayconst loginBtn = await page.$x('//button[contains(., "Login")]')await loginBtn[0].click()
// using query selector apiconst buttons = await page.$$('button')for(const button of buttons) { const btnText = await (await button.getProperty('textContent')).jsonValue() if(btnText.includes("Login")) { await button.click() }}
// using evaluateawait page.evaluate(() => { const buttons = document.querySelectorAll('button')
buttons.forEach(button => { if(button.innerText.includes("Login")) { button.click() } })})
There are other methods like $$eval and $eval that can be used instead of the above methods, but you got the gist of it, right?
Wait for elements to load
It might be tempting to use the page.waitForTimeout method to wait for an element to load, but there are better ways below. This is not an ideal solution because you don't know how long the element will take to load.
Always use page.waitForSelector before performing any operation on an element, especially after a new page has been loaded. This is important because the element you want to access might take time to load or it is being inserted into the DOM using JavaScript after some processing.
You can't guess when will be an element available on the dom and if you try to perform an operation let's say click , it might throw an error as the element hasn't been loaded into the DOM yet.
Therefore, its almost always essential to use waiting methods like the below:
// for CSS selectorsawait page.waitForSelector('button[id=loginBtn]')await page.click('button[id=loginBtn]')// for XPathawait page.waitForXPath('//button[@id="loginBtn"]')await page.click('button[id=loginBtn]')
Sometimes, it might also come in handy to include a custom timeout as an option and the visible property.
timeout means that if the element is not found after n minutes, throw an error. visible means that the element is found in the DOM and is visible. Otherwise, it may be hidden behind another element or may have a CSS property that makes it invisible.
// for CSS selectors// throw an error if the element is still not found after 60000 ms
await page.waitForSelector('button[id=loginBtn]', { visible: true, timeout: 60000 }) // timeout is in millisecondsawait page.click('button[id=loginBtn]')// for XPathawait page.waitForXPath('//button[@id="loginBtn"], { visible: true, timeout: 60000 }')await page.click('button[id=loginBtn]')
Wait for navigation
When the browser navigates from one page to another or reloads, it is always necessary to wait for the navigation to complete. Otherwise, if you try to perform operations on the to-be-navigated page and it hasn't completed loading yet, it will throw an error.
The events that might trigger navigation generally are page.goto , page.click , page.reload , page.goBack , and page.goForward .
Waiting for navigation (till the dom has completely loaded)
await page.click('button[id=submitForm]') // triggers navigationawait page.waitForNavigation() // wait till the next page has loaded// your operations on the next page...
The above might suffice if you just want to wait for the DOM to load. If you also want to wait for the fetch/XHR calls in the background to finish, then page.waitForNavigation method allows you to pass that option.
// wait till there have been atmost 0 requests in 500ms timeframe// for surity, use this instead of 'networkidle2'await page.waitForNavigation({ waitFor: 'networkidle0' })// wait till there have been atmost 2 requests in 500ms timeframe
await page.waitForNavigation({ waitFor: 'networkidle2' })
You can also extend the default 30s timeout to wait longer if needed
await page.waitForNavigation({ waitFor: 'networkidle0', timeout: 60000 })Another good practice is to use a Promise.all when waiting for navigation. Pass the first promise as page.waitForNavigation and pass the second promise as the action that might trigger navigation.
await Promise.all([ page.waitForNavigation(), page.click("#loginBtn") // triggers navigation])
This is necessary because the below code might not work as expected as the page.waitForNavigation might keep on waiting and eventually timeout.
await page.click("#loginBtn") // triggers navigationawait page.waitForNavigation() // use the above appraoch instead!
Wait for network calls
You might face the common use case of waiting for an API call to populate the page content. In these types of use cases, page.waitForNetworkIdle might come in handy.
await page.click("#fetchUsers") // makes an API call (no navigation)await page.waitForNetworkIdle() // wait till the page has no more fetch/XHR calls pending
There are two optional properties that can be passed to this method, idleTime and timeout .
idleTime : Specifies the minimum number of ms till there are no API calls
timeout : Specifies the time limit in ms to wait for till there are no API calls
Also, for page.waitForNetworkIdle we should use Promise.all for the reasons mentioned above.
await Promise.all([ page.waitForNetworkIdle(), page.click("#loginBtn")])
Use evaluate method whenever possible
The code we write in Puppeteer is converted to vanilla JavaScript that can be understood by the browser and is executed in the browser context. This can be inefficient if you are writing lots of Puppeteer methods that could have been written in an evaluate method.
evaluate method provides better performance as
1) The code is written in vanilla JavaScript that takes less processing from Puppeteer and is easier to write for most devs
2) Most separate Puppeteer calls can be combined into one evaluate script, this means fewer calls between the browser and Puppeteer
I am speaking of evaluate here, but all these points are also applicable to other helper methods like evaluateHandle , $eval and $$eval .
For example, let's say you want to read the text in the first td of each table row tr
// with usual Puppeteer methodsconst tdList = []const trs = await page.$('table > tr')for(const tr of trs) { const td = await tr.$('td') const tdValue = await (await td.getProperty('textContent')).jsonValue() tdList.push(tdValue)}
As you can see above, Puppeteer will make calls to the browser for every tr in the table , this means a lot of back and forth. Instead, we can use a single evaluate script and comparatively shorter code to implement this.
// with evaluate methodconst tdList = await page.evaluate(() => { const tempList = [] document.querySelectorAll('table > tr').forEach(tr => { tempList.push(tr.querySelector('td').textContent) }) return tempList})
This takes only a single call to the browser!
When NOT to use evaluate
evaluate can seem to be the go-to solution for everything once you get the hang of it. But, there are times when you should avoid using evaulate and instead are better off using Puppeteer methods.
This generally includes user interactions like mouse clicks, keyboard events, typing, hover, focus, etc. These can be implemented using evaulate but might not trigger the side effects.
So, as a rule of thumb, always use Puppeteer methods like click , type , select , hover , etc when you need to do browser interactions.
Don’t forget to close your browser instance
That was it, that was the advice. Remember to do await browser.close().
That's all folks! I hope this will help you to do awesome stuff with Puppeteer.
See you at the next one!
Useful Links:
https://stackoverflow.com/questions/55664420/page-evaluate-vs-puppeteer-methods
