Selenium v PlayWright

Today, I’ll wright about the difference between Selenium and PlayWright including my feeling in using them.

About Selenium

Selenium controls the browser through the WebDriver. So you have to install and set the WebDriver for each browser to use Selenium.

WebBrowsers which can be used in Selenium

• Google Chrome
• Edge
• Firefox
• Safari

Coding languages which can be used in Selenium

• Java
• JavaScript
• Python
• Ruby
• C#

How do we wright codes

I used to use Selenium to make crawling routes for OZ(the crawling system used in Goalist).

So, I’ll show you the actual code I wrote then.

import model.AddData;
import model.CrawlListData;
import org.openqa.selenium.*;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.openqa.selenium.support.ui.Select;

import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--remote-allow-origins=*");
        WebDriver driver = new ChromeDriver(options);

        String rootUrl = "https://www.pasona.co.jp/jobsearch/";

        System.out.println("url,pthStr1,pthStr2,pthStr3,listType");
        getArea(driver, rootUrl);

        driver.quit();
    }

    public static void getArea(WebDriver driver, String rootUrl) throws InterruptedException {
        driver.get(rootUrl);

        String preBase = "list?p=1&place_pref_cd=";
        String cityBase = "&place_area_cd=";

        List<String> subRootUrls = getSubRootUrls(driver);
        List<CrawlListData> datas = new ArrayList<>();

        for(String subRootUrl : subRootUrls){
            driver.get(subRootUrl);
            String areaStr1 = driver.findElement(By.cssSelector(".tool__district > div > p > span")).getText();
            WebElement dropdown1 = driver.findElement(By.id("place_pref"));
            Select preSelect = new Select(dropdown1);
            List<WebElement> prefectures = preSelect.getOptions();
            for(WebElement prefecture : prefectures){
                String areaStr2 = prefecture.getText();
                if(areaStr2.equals("都道府県")){
                    continue;
                }
                String preValue = prefecture.getAttribute("value");
                preSelect.selectByVisibleText(areaStr2);
                Thread.sleep(1000);
                WebElement dropdown2 = driver.findElement(By.id("place_area"));
                Select citySelect = new Select(dropdown2);
                List<WebElement> cities = citySelect.getOptions();
                for(WebElement city : cities){
                    String areaStr3 = city.getText();
                    if(areaStr3.equals("エリア")){
                        continue;
                    }
                    String cityValue = city.getAttribute("value");
                    String url = subRootUrl + preBase + preValue + cityBase + cityValue;

                    CrawlListData data = AddData.addData(url, areaStr1, areaStr2, areaStr3, 1);
                    datas.add(data);
                }
            }
        }
        for(CrawlListData data : datas) {
            System.out.println(data);
        }
    }

    public static List<String> getSubRootUrls(WebDriver driver) {
        List<String> subRootUrls = new ArrayList<>();
        driver.findElement(By.cssSelector(".tool__district a")).click();

        WebElement areaFlame = driver.findElement(By.cssSelector(".modal-district-selector__list"));
        List<WebElement> areaBoxes = areaFlame.findElements(By.cssSelector(".list__item"));
        for(WebElement areaBox : areaBoxes){
            String subRootUrl = areaBox.findElement(By.cssSelector(".list__content a")).getAttribute("href");
            String urlWithoutQuery = null;
            try{
                URI uri = new URI(subRootUrl);
                urlWithoutQuery = uri.getScheme() + "://" + uri.getHost() + uri.getPath();
            } catch (URISyntaxException e) {
            }
            subRootUrls.add(urlWithoutQuery);
        }

        return subRootUrls;
    }

I apologize for my sucked code but I didn’t suppose to show someone this code when I wrote this.

There’re some specific points in the code of Selenium. I’ll show them below. FYI, I’ve only coded in Java, so I only know the codes for Java. But I wanna show you codes that have same meanings in some languages. I apologize if I took mistakes.

1. WebDriver Setting

As I said above, we have to install WebDriver to use Selenium. So, you have to write the path to the driver you use, in the main method.

2. The methods prepared in Selenium

There’re some methods prepared in Selenium to get elements from HTML. So, I’ll show you some of them.

— driver.get(“URL”); [Java] | driver.Url = “URL”; [C#] | driver.get(“URL”) [Python]

This is the most important method for Selenium, that boots up the browser.

— driver.close(); [Java] | driver.Close(); [C#] | driver.close() [Python]

This is to close the browser. If you booted some browsers and wanna close all of them, you can do by using “driver.quit(); [Java] | driver.Quit(); [C#] | driver.quit() [Python]”.

— driver.findElement(By.~(“*****”)); [Java] | driver.FindElement(By.~(“*****”)); [C#] | driver.find_element_by_~(“*****”) [Python]

This is used to get elements. If you wanna get elements by “classname”, insert the word “className” in [~] of these. you can use with “id” and “xpath” by doing same way as “classname”. (When using C#, you need to insert like “ClassName”, “Id”, “Xpath”. And using python, only in the case of class name, need to insert like “class_name”.)

—driver.findElement(By.~(“*****”)).click(); [Java] | driver.FindElement(By.~(“*****”)).Click(); [C#] | driver.find_element_by_~(“*****”).click() [Python]

This is to click the selected element. You can use “className”, “id”, “xpath” in this method as well. Because Selenium is the browsing tool, this function is also important.

— The method to scroll [Java] [C#] [Python]

public static void scrollToTarget(WebDriver driver, WebElement target) {
        JavascriptExecutor js = (JavascriptExecutor) driver;
        js.executeScript("arguments[0].scrollIntoView(true);", target);
}

public static void ScrollToTarget(IWebDriver driver, IWebElement target) {
    IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
    js.ExecuteScript("arguments[0].scrollIntoView(true);", target);
}

target_element = driver.find_element(By.ID, "target-element-id")
scroll_to_target(driver, target_element)

Selenium usually boots browser and visually crawling elements, so it can’t crawl or click the invisible element, for instance in the case the target element is out of the window. In such cases, you have to implement the function to scroll. This is one of the important feature of Selenium.

— ~.getText(); [Java] | ~.Text; [C#] | ~.text [Python]

These need to follow the code to find elements. By these methods, you can easily get the text in the selected element.

— ~.getAttribute(“value”); [Java] | ~.GetAttribute(“Value”); [C#] | ~.get_attribute(“value”) [Python]

These need to follow the code to find elements as well. By using these, you can get the attribute elements like, “value”, “name”, “href” and so on.

— To select the dropdown elements

WebElement dropdown= driver.findElement(By.~("*****"));
Select(element).selectByVisibleText("text");

IWebElement dropdown = driver.FindElement(By.~("*****"));
Select(dropdown).SelectByText("text");

dropdown = driver.find_element_by_~("*****")
Select(dropdown).select_by_visible_text("text")

By using these methods, you can select the dropdown. And if you use these with “.getOptions()”, you can get the options in a dropdown box. Please see the code above I pasted at first to see the concrete way to use.

What I felt in using Selenium

I’ve used Selenium several times and I felt some good points and bad points.

The good points are because it’s visible, I can feel fun. Of course not only that. Because of its visibility, I can easily find the error during crawling. For example, when the function of clicking doesn’t work normally, I can see the window and find out click the other element or button. This is one of the good points.

The bad points are that in some cases, Selenium needs to scroll and get the target into the window. If the element you wanna get or click is out of the window, an error occurs. This is the most annoying point I felt in using Selenium.

So then, I’ll wright about PlayWright. Plese compare it to Selenium and find the differences.

About PlayWright

PlayWright also controls the browser but doesn’t need WebDriver. This is one of the differences between Selenium and PlayWright. You don’t have to install the WebDriver.

WebBrowsers which can be used in PlayWright

• Google Chrome
• Edge
• Firefox
• safari

Coding languages which can be used in Selenium

• Java
• Python
• Node.js

How do we wright codes

Recently I often use PlayWright to make routes. And I make the same code as I pasted in the Selenium part but in PlayWright. So I’ll show you the actual code.

import com.microsoft.playwright.*;

import model.AddData;
import model.CrawlListData;

import java.net.URI;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.List;

public class Main {
    public static void main(String[] args) throws InterruptedException {
        try(Playwright playwright = Playwright.create()) {
            BrowserType browserType = playwright.chromium();
            Browser browser = browserType.launch();
            BrowserContext context = browser.newContext();

            String rootUrl = "https://www.pasona.co.jp/jobsearch/";
            Page page = context.newPage();
            page.navigate(rootUrl);


            getArea(page);
        }
    }

    private static void getArea(Page page) throws InterruptedException {
        List<CrawlListData> datas = new ArrayList<>();

        String preBase = "list?p=1&place_pref_cd=";
        String cityBase = "&place_area_cd=";

        for(String subRootUrl : getSubRootUrl(page)){
            page.navigate(subRootUrl);
            String areaStr1 = page.querySelector(".tool__district > div > p > span").innerText();
            ElementHandle dropdown = page.querySelector("#place_pref");
            List<ElementHandle> preOptions = dropdown.querySelectorAll("option");
            for(ElementHandle preOption : preOptions){
                String areaStr2 = preOption.innerText();
                if(areaStr2.equals("都道府県")){
                    continue;
                }
                String preValue = preOption.getAttribute("value");
                dropdown.selectOption(preOption);
                Thread.sleep(1000);
                ElementHandle dropdown2 = page.querySelector("#place_area");
                List<ElementHandle> cityOptions = dropdown2.querySelectorAll("option");
                for(ElementHandle cityOption : cityOptions){
                    String areaStr3 = cityOption.innerText();
                    if(areaStr3.equals("エリア")){
                        continue;
                    }
                    String cityValue = cityOption.getAttribute("value");
                    String url = subRootUrl + preBase + preValue + cityBase + cityValue;

                    AddData addData = new AddData();
                    datas.add(addData.addData(url, areaStr1, areaStr2, areaStr3, 1));
                }
            }
        }
        for(CrawlListData data : datas){
            System.out.println(data);
        }
    }

    private static List<String> getSubRootUrl(Page page){
        List<String> subRootUrls = new ArrayList<>();

        page.querySelector(".tool__district a").click();

        List<ElementHandle> areaBoxes = page.querySelectorAll(".modal-district-selector__list .list__item");
        for(ElementHandle areaBox : areaBoxes){
            ElementHandle bigElement = areaBox.querySelector(".list__content a");
            String subRootUrl = bigElement.getAttribute("href");
            String urlWithoutQuery = null;
            try{
                URI uri = new URI(subRootUrl);
                urlWithoutQuery = "https://www.pasona.co.jp" + uri.getPath();
            } catch (URISyntaxException e) {
            }
            subRootUrls.add(urlWithoutQuery);
        }
        return subRootUrls;
    }
}

I apologize for my sucked code again.

I want you to compare this code to the code of Selenium.

1. WebDriver Setting

As I said, you don’t need to set the WebDriver in PlayWright. This is because PlayWright supports multiple browser engines.

2. The methods prepared in PlayWright

There’re some methods prepared to get elements from HTML also in PlayWright. So, I’ll show you some of them. (Meaning of the code below is same as ones of Selenium, so I’ll paste only the codes.)

— page.navigate(“URL”); [Java] | page.navigate(“URL”) [Python]

— page.close(); browser.close(); [Java] | page.close() browser.close() [python]

— page.querySelector(“*****”); [Java] | page.query_selector(“*****”) [Python]

In PlayWright, the elements have to be selected by CssSelector. But you can also use the code below.

getElementById(“”); | getElementByCssSelector(“”); etc.

But I usually use [querySelector(“”);].

— .click(); [Java] | .click() [Python]

— The method to scroll

None in PlayWright. PlayWright doesn’t boot the browser visibly.

— ~.innerText(); [Java] | ~.inner_text() [Python]

— ~.getAttribute(“value”); [Java] | ~.get_attribute(“value”) [Python]

— To select the dropdown elements

ElementHandle dropdown = page.querySelector("*****");
List<ElementHandle> options = dropdown.querySelectorAll("option");

dropdown = page.query_selector("*****")
options = dropdown.query_selector_all("option")

What I felt in using PlayWright

I felt good and bad things for PlayWright as well.

One of the good Thing is its speed. Once I compared same codes but one is in Selenium and the other is in PlayWright. And the result was the code written by PlayWright ran two times faster than Selenium. *The selenium updated recently and its speed was improved a little.

And the other is when coding, I don’t need to care about the location of the element. Even if the element is out of window, it’s not its business when I use PlayWright.

The bad thing is its invisibility. Compared to Selenium, It’s difficult to find out the cause of the errors when I use PlayWright. So I sometimes re-wright the same code in Selenium again. It’s losing everything.

Summary

In this article, I show some differences between Selenium and PlayWright. Both of them have the advantages and disadvantages. So in conclusion, I can’t decide which is superior. All I can say is when we use them, we have to think which to use by seeing the condition and situation.

Thank you for reading.