Shadow DOM in Python Selenium

What changed in Chromium 96 and what to do about it

Selenium logo in Shadow with Python

As soon as Chromium v96 was released, the Selenium and Google teams started seeing issues about broken shadow DOM code. The change is a feature, not a bug! Here’s how to use Python to locate Shadow DOM elements in Selenium for each browser.

Note: versions of this article are also available in: Java, .NET and Ruby

Chromium v96 is the first browser to be compliant with the W3C WebDriver specification for the shadow root. The spec now includes definitions for getting an element’s shadow root and locating elements in a shadow root. This applies to all Chromium browsers (Chrome, Edge, Opera, etc). You can expect Firefox and Safari to have implementations in the near future.

At this point, this is the code you’re likely using if you’re working with shadow DOM elements in Python:

shadow_host = driver.find_element_by_id('shadow_host')
script = 'return arguments[0].shadowRoot'
shadow_root = @driver.execute_script(script, shadow_host)
shadow_content = shadow_root.find_element_by_id('shadow_content')

(see full example on GitHub)

This code still works in both Selenium 3 & 4 for Safari and Chromium versions earlier than 96. The return value from the#execute_script command gets parsed, and any element references are automatically converted. The change in Chromium v96 is that the return value is not parsed as an element, so the above code results in this error:

AttributeError: 'dict' object has no attribute 'find_element_by_id'

Python didn’t address this use case in Selenium 4.0, so you need to update to Selenium 4.1 for this to work. Unfortunately, this still won’t fix your code because Python does not implement the method we’re using with the shadow Root. The above code in 4.1 gives us this:

AttributeError: 'ShadowRoot' object has no attribute 'find_element_by_id'

This is easily fixed by replacing find_element_by_id() with the new By class implementation like this:

shadow_content = shadow_root.find_element(By.ID, 'shadow_content')

Because Python doesn’t have the strict typing concerns of Java and .NET, the above code will now work just fine with Selenium 4.1 in all versions of Chromium and Safari. If you’re only working in the latest Chromium browsers, you should be using the new shadow_root property.

shadow_host = driver.find_element(By.ID, 'shadow_host')
shadow_root = shadow_host.shadow_root
shadow_content = shadow_root.find_element(By.ID, 'shadow_content')

(see full example on GitHub

None of this code works for Firefox. The JavaScript command to get the shadow root does not work, but you can get all the children and then iterate over the list:

shadow_host = driver.find_element(By.ID, 'shadow_host')
script = 'return arguments[0].shadowRoot.children'
children = driver.execute_script(script, shadow_host)

shadow_content = next(child for child in children
    if child.get_attribute('id') == 'shadow_content')

(see full example on GitHub)

What if you are holding off on upgrading to Selenium 4? I still have a solution for you, well multiple solutions. The first is— just update to Selenium 4, it’s almost definitely easier than you think it is, and it will save you time in the long run. The other option works, but is kind of hacky:

shadow_host = driver.find_element_by_id('shadow_host')
script = 'return arguments[0].shadowRoot'
shadow_root_dict = driver.execute_script(script, shadow_host)
id = shadow_root_dict['shadow-6066-11e4-a52e-4f735466cecf']
shadow_root = WebElement(driver, id, w3c=True)
shadow_content = shadow_root.find_element_by_id('shadow_content')

(see full example on GitHub)

This should clear up the confusion over the new behavior and give you the information you need to work with shadow DOM elements in every browser. Happy testing!

Additional References:
* Language-agnostic version of this article on my website,
* Video with Java examples on YouTube