Inspecting The Shadows: The Secrets of Shadow Doms

Photo by Sergiu Nista on Unsplash

Dive into the world of Shadow DOM, a powerful browser feature! In this article, we’ll uncover its secrets: what it does, why it’s important, and how to harness browser APIs like shadowRoot and MutationObserver to interact with its hidden elements. We'll even craft a new type of selector to consistently save them in our DB

Hello everyone! My name is Maor Zigel and I’m a part of a company of experts-on-demand called Tikal.

While working on a project, I encountered a client who had created a Chrome extension. The extension allowed the client to define custom workflows. They could select elements on a webpage, specify events to trigger actions and save these configurations. Upon revisiting the page, the extension would automatically identify the saved elements using their unique selectors, listen for the designated events, and initiate the pre-defined actions.

I made a simple pen to demonstrate it:

Element Inspector before the challenge

Who is lurking in the shadows?

The challenge I faced was a bug that prevented the extension from recognizing individual elements on a webpage. Instead, it treated the entire page as a single element, making it impossible to inspect or interact with specific parts.

For example, I made another pen to demonstrate what happened when they tried to inspect an input or button inside a web component:

As you can see in the last pen you can’t mark the input, h2, or button inside our new web component and why is that?

Good question! the next slide will be exactly on that!

to understand what is happening let’s look at the following code snippet from the pen:

inspectorRoot.addEventListener('mouseover', (e) => {
  if (onInspection) {
    const elementToInspect = e.target;
    
    rect = e.target.getBoundingClientRect();
    ...
    updateInsectionInfo(e.target);
  }
});

This simple code is listening to the mouse move events on the inspection area and if we are in inspection mode so we take the target of the mouse event and mark it.

So what is wrong with this code? While Web Components typically combine Custom Elements, Shadow DOM, and Templates, not all components require all three. (ref from MDN).

In our example, we did use the Shadow Dom feature and that started the problem.

But why? Because Shadow Dom has a couple of abilities that make them special for the use of Web Components.

Shadow Dom

Shadow Dom encapsulates all the inner elements, logic, events, and styles and makes the web component prone to outer interactions, events, styles, and more and that’s why we want to use it inside our web components so that will be like a black box that is not affected by the parent application.

So now you can start to understand why we are getting the parent web component and not the inner parts, that’s because of the Shadow Dom that blocks the way inside, so the event.target value will be the web component and not the inner input or button.

Composed Path

So how can we fix this you ask? actually, the fix is very simple!

the following line:

const elementToInspect = e.target;

will turn to:

const elementToInspect = e.composedPath()[0];

The <Event>.composedPath function will give us the full path as an array even if the path goes through shadow Dom in the way, from the inner current element to the top document root like so:

Yeah! So now we can get the actual element by just taking the first element in this array of Composed Path trees as you can see in this following pen:

Inspecting the Shadows

Now we need to face with our next challenge: saving a selector to target the buttons consistently. We want this selector to work even after the page reloads. Ideally, the selector “.container button” should grab both buttons — the one outside the Shadow DOM and the one within.

But again our Shadow Dom friend is playing with us and when we try to grab that 2 button with the following line:

document.querySelectorAll('.container button');

But we will get only one like so:

you can see the next pen if you want to check for yourself:

But what is the problem now? so as we already said the Shadow Dom is blocking the browser from searching for elements that is inside the Shadow Dom but what can we do about it?

We will need to run the querySelectorAll function inside the Shadow Dom in order to catch the inner button with the following line:

const innerButton = document.querySelector('parent-web-component').shadowRoot.querySelectorAll('.container button');

like so:

As you can see we need to explicitly tell the browser that our new Document root is now the inner Shadow Root of our Web Component so the querySelectorAll function will look in this context for the selector.

BTW: a Shadow Root node is actually of type of DoucmentFragment as you can see here:

Therefore you can activate all the function that runs on your root document.

So now that we know how to select those inner elements we can create our own kind of selectors that can direct us to the inner paths by adding all the parent web components to the path and putting the native selector in the end.

Selecting the Shadows

So, for example, I create the following types of selectors with the >>> sign between the parts:

parent-web-component>>>inner-web-component>>>#div-id.my-div-class

In the following example, my #div-id.my-div-class is inside a tree of 2 web components (2 shadow roots in the way).

And by writing a simple function to read this selector we can easily get the inner elements like so:

 private getPosibleRootDocumentElements(
  currentDocumentRoot: DocumentFragment | null,
  nextChildSelectors: string[],
  elementSelector: string,
  documentRoots: DocumentFragment[]
 ): void {
  if (currentDocumentRoot === null || currentDocumentRoot === undefined) return;
  if (nextChildSelectors.length === 0) {
   if (lazyJQuery.getValue()(elementSelector, currentDocumentRoot).length > 0) {
    documentRoots.push(currentDocumentRoot);
   }
   return;
  }
  const nextChildSelector = String(nextChildSelectors.shift());
  const posibleChildren = currentDocumentRoot.querySelectorAll(nextChildSelector);
  for (let posibleChildIndex = 0; posibleChildIndex < posibleChildren.length; posibleChildIndex++) {
   const posibleChild = posibleChildren[posibleChildIndex];
   this.getPosibleRootDocumentElements(posibleChild.shadowRoot, [...nextChildSelectors], elementSelector, documentRoots);
  }
  return;
 }

function Params:
1. currentDocumentRoot: just the window.document for the start
2. nextChildSelectors: is just a split of the special selector (without the last part of the native selector)
3. elementSelector is the last part of the split selector (the native selector part)
4. documentRoots: an empty array that will be filled as the recursion goes with the actual correct shadow root parents that match for the native selector

As you can see the following recursive function is getting the current document fragment and the next inner Web Component names and the inner native selector and just going into the tree and looking for possible paths like we can have the 2 of the same Web Components brothers in the same parent so it covers that too.

Great! now we can save and reselect any element that we want even if it is inside a Shadow Dom or even a tree of Shadow Doms.

Listening Between the Shadows

So to my last challenge: now we want to check if an element has appeared or disappeared from the page so we can notify about this event.

Ok so we just save our new selector and just listen for changes in our page with our good friend: Mutation Observer we can listen to any changes on the subtree or attributes of a given element.

This Mutation Observer is a very good thing for another article but for now, we need to know its simple powers of notify on change events on the element (in our case the element is the root document) like so:

  const mutationObserverDiscoveryOptions: MutationObserverInit = {
   childList: true,
   subtree: true
  };

  this.mutationObserverDiscovery = new MutationObserverManager(
   mutationObserverDiscoveryOptions,
   document,
   this.onMutation);

This code will run the onMutation function whenever our document subtree is changed so we can again look for those special selectors and check if they exist or not.

But again! our Shadowy friends are avoiding our Mutation Observer and not notifying us of the inner Shadow Dom subtree changes :-(

So here we have 2 options to handle this:

Our first option is to add the Mutation Observer to any Shadow Root in the page for example:

const mutationObserverDiscoveryOptions: MutationObserverInit = {
   childList: true,
   subtree: true
  };

  this.mutationObserverDiscovery = new MutationObserverManager(
   mutationObserverDiscoveryOptions,
   document.querySelector('my-web-component').shadowRoot,
   this.onMutation);

And we will need to manage and do it for every Shadow Root in the page.

The other more simple and working option that ended up in production is to manually check for existence every 2 seconds (Not the perfect solution but not so heavy in performance).

  this.manualCheckIntervalId = setInterval(() => {
   this.checkForExistence();
  }, 2000);

Conclusions

We dived into the world of Shadow DOM and its role in the Web Components world.

We explored how to leverage browser APIs like the .shadowRoot property and the Mutation Observer to inspect, composedPath function, and interact with these hidden elements.

We'll uncover techniques to effectively locate and monitor Shadow Roots, even with its efforts to remain elusive.

Hoped you enjoyed learning about Browser APIs and Web Components with me! There’s so much more to explore in this area, but I’ll save that for future articles.

Love,
Maor Zigel, FE Architect in Tikal