An analysis of Selectors on Test Automation
By Vinicius Seganfredo
While writing automated tests we have numerous forms of selecting a WebElement. In this article we are going to analyse multiple selectors and their performance under different scenarios/different browsers and check what their data points to.
There are three things we need to ask ourselves before we continue:
- “What is a selector?
Good question! If you’re new to test automation and you’re reading this, there’s a good chance that you don’t know what a selector is, so let’s quickly go through the definition of a selector. In summary, selectors are different patterns used to locate WebElements on a given webpage/DOM so we can interact with them and perform certain actions. A WebElement in the other hand is an element that is present on the webpage/DOM. Elements can be buttons, text fields, input fields, radio buttons and so on. - “Why?”
As Automation Engineers, we need to worry not only about delivering high quality code but also an automation solution that runs fast. It is useful to know which selectors are faster under certain situations so we can make smarter decisions. Selectors can impact maintainability, reliability and the outcome of your tests. That’s why it’s so important that we choose our selectors wisely. - “How can we measure the execution time of a selector?”
We need to find balance. I was struggling to figure out which would be the best way to collect all the data that was needed for this article in order to reliably compare all of the selectors. At the beginning, I wrote a code that found an element 1000 times using a given By and logged the time spent to find all the elements. Later I reduced this number to 500, tried running the tests in parallel but I wasn’t satisfied with the results as I was getting too different results on each execution. I found balance when I decided to save the execution time of each FindElement and calculate the average time it took to find that given element after 1000 repetitions. This process was executed 58 times per selector, the 4 highest and the 4 lowest execution times were disregarded for each selector.
It’s good to mention that the execution time of each selector is measured in milliseconds and performance benefits can only really be spotted when we take a look at multiple selectors being used sequentially. And what other name we can use for multiple selectors being executed sequentially? Exactly!
A test case (if we also have validations on it). Let’s check an example to illustrate the benefits of choosing the correct selectors. Imagine a project that has over 3,000 test cases. Each of those test cases finds 10 links and all of them are using the worst performing selector to find the links on Chrome, which means it takes up to 10ms for the element to be found. That would sum up to 300,000ms (or 5 minutes). Now imagine that the best performing selectors were used for finding the links — if we select the elements using CSS Selector for instance, finding an element by its link takes 4.83ms, running all our tests in that case takes 144,900ms (or 2.4 minutes). If we use this same example but on Firefox instead, we can notice an even bigger performance improvement. The worst link selector on Firefox takes up to 27ms to be executed, what would lead us to 810,000ms (or 13.5 minutes) of execution time, using the same selector that we used on the previous example (CSS Selector), that has an average of 1.56ms of execution time, we end up with an execution time of just 46,800ms (or 46 seconds). Usually these automated tests are scheduled to be executed multiple times a day and saving some minutes in only one test execution just by choosing the selectors wisely, can end up saving us hours in the long run.
Before we dig in the performance measurement of each selector, let’s go quickly through the different definitions of some selectors:
- XPath: Allows to navigate through the HTML structure of a page. XPath allows the navigation through the XML structure of any document; it can be used both on HTML and XML. Provides an option to dynamically search for an element within a Web Page, giving the user flexibility. Depending on the XPath expression, it can be hard to read. Each browser has its own implementation.
TagName — By.XPath(“//div”);
By Id — By.XPath(“//div[@id=’myId’]”);
By Class — By.XPath(“//div[@class=’myClass’]”);
Contains Class — By.XPath(“//div[contains(@class, ‘myClass’)]”);
Parent Element — By.XPath(“//div[@id=’myId’]//parent::div”);
- CSS Selector: String pattern used to identify elements based on combination of HTML tags, Ids, classes and attributes. Readability is better than XPath in some situations. Has the same implementation across all browsers.
TagName — By.CssSelector(“div”);
By Id — By.CssSelector(“#myId”);
By Class — By.CssSelector(“.myClass”);
Contains Class — By.CssSelector(“div[class~=’myClass’]”);
Parent Element — Not supported.
- By Id: Finds an element providing a given unique Id.
By.Id(“myId”);
- By Class: Finds an element providing a given Class.
By.ClassName(“myClass”);
- By Link Text: Finds an element based on the text of a link.
By.LinkText(“My Link”);
- By Partial Link Text: Finds an element based on the partial text of a link.
By.PartialLinkText(“Partial Link”);
- By Tag Name: Finds an element based on a tag name.
By.TagName(“div”);
Let’s analyse how the selectors behave under different scenarios/different browsers:
Surprisingly, the fastest selections happened on Edge (average 780 microseconds), followed by Firefox (average 3.8 milliseconds), Chrome (average 5.3 milliseconds) and lastly, Internet Explorer (average 116 milliseconds). Let’s dissect each browser individually and inspect the results closely.
Edge
After a quick look on the above chart, we can identify that CSS Selectors are performing better on Edge than XPaths. We can also see that the direct selectors (By.Id, By.Class and so on) are also performing a bit better than XPath. It is good to mention that we are talking about a difference of 0.3 microseconds in this situation and this is something that will only impact your test results on the long run. We can dissect it even further and divide it by different groups:
Now we can easily identify which selector is the best performing one for different kinds of situations. If you have an element with an Id, and you are testing on Edge, you can go for By.Id without thinking twice, as that is the fastest/most reliable way of selecting an element. On the other hand, if you want to find an element By Class, CSS selector is faster in that case as well as in almost all other charts — Link Text, Name and Tag Name.
Firefox
We can see that on Firefox the CSS Selectors are also performing better than the XPaths. However, we can also see that on Firefox some of the direct selectors (By.Class, By.LinkText) are performing worse than the XPaths, differently than what we saw on Edge. Let’s dissect the selectors in different groups, just like we did for Edge.
With this new information, we can start to make a better and smarter choice of which selector to go for. Let’s take a look at the Id chart for instance, the fastest selector is CSS — Contains Id followed by CSS — By Id, on Edge, it is By.Id followed by CSS — By Id. It doesn’t make sense for us to use “Contains” as most of the times the Id is unique. In that case we could go for CSS — By Id, that performs well both on Edge and Firefox. From this new groups, we can also see that By.LinkText performs really bad on Firefox although it performs well on Edge. In that case we have other 4 selectors to choose from who will perform much better.
Chrome
On Chrome, CSS Selectors are also performing better than XPaths. We can see that on Chrome the Direct Selectors are pretty much mixed together with the other selectors with exception of the By.Link/PartialLinkText that are the worst performing selectors of all. We can also see that finding an element by Class seems to be faster than by Id on Chrome. Let’s divide the selectors into smaller groups:
After dividing the selectors into groups, we can see that indeed it is faster to select an element by class on Chrome than by Id (let’s remind that we are talking about milliseconds). We can also see that CSS Selectors are faster in almost all groups and that LinkText is still being a bad selector choice as it is the worst performing selector of all on Chrome as well.
Internet Explorer
Analysing the data regarding Internet Explorer, we can see a twist of events and verify that XPath is faster than CSS Selector on IE. We can also see that the direct selectors are mixed up among the CSS Selectors but none of them are among the Xpaths. The LinkText on Internet Explorer is very slow, taking almost an entire second to process (798ms). So, let’s go ahead and divide this into smaller groups:
After doing the division, we can see that XPath is the best performing selector on Internet Explorer on almost all the groups. The execution time average of XPaths on IE is 46ms while the CSS Selector average is 72ms. The worst selector, as I stated before, is the LinkText/PartialLinkText, but we have different options to deal with that if we look into our charts.
Now that we have analysed all of the different data for all different browsers, how do we choose which selector is the best for each situation? Well, that depends. There are some questions that you need to ask yourself before making a choice. Are you running cross-browser testing? Does your application have dynamic elements? How does the element that you want to select looks like? Does it have an Id, Class, Attribute? Let’s use the below example:
Andrew wants to select the following element:
<a href=”/en/about/”> About Us</a>Let’s ask Andrew the questions:
- Are you running cross-browser testing? Yes! I am! I’ll run on all the browsers.
- Does your application have dynamic elements? No, you can see the element over there, you don’t need to ask me this!
- How does the element that you want to select looks like? It has an “anchor” tag, with LinkText = ‘About Us’ and href = “/en/about/”
With the answers in hands, we can think about which selector is the best for Andrew’s automation script taking a further look on the charts that we have in hands. As the LinkText is the worst performing selector on the majority of browsers (except Edge), let’s eliminate that possibility.
So, we’re down to CSS Selector, XPath, and TagName. Should we use TagName? Well, we don’t know if Andrew’s page has more links other than the About Us link. So due to that it is better if we can avoid selecting it by TagName for now. Taking a quick look into the charts, we can identify that the fastest selector that is left for us on Chrome, Firefox and Edge is CSS Selector — Link Text. On the other hand, on IE it is XPath. In this situation there are three possible solutions. We could either use Xpath for IE and CSS for the other browsers, or we could choose among one of them — in this case, I’d go for CSS Selector, as it is better performing on 3 out of 4 browsers.
After analysing the collected data, we can conclude that none of the selectors are perfect. At the end, it is up to us which selector we are going to use. In order to make the best choice, we need to always keep in mind the reliability, usability, performance and maintainability of the selectors that are available for selecting a given element. In that way, we can guarantee the improvement of our test’s quality and our analytical skills as Automation Engineers.
If you’d like to take a further look into all the available charts for all browsers, you can find them below:
