Visual coverage: Why and How Preply Measures the Impact of the Design System (part I)

The Design System visual coverage dashboard with all the per-team and per-page data.

We implemented a custom way to measure the impact of the Design System, and we measure it on user’s devices. Why? Why were the existing solutions and approaches not enough? Why is it so crucial for us that everyone, management included, can understand and visualize the design system’s impact?

We worked for months to come up with our own way to measure the design system impact, and we shared all the whys and the hows in this article. A longer, technical one (The Implementation Details of Preply’s Design System Visual Coverage), explains what we did. Last but not least, the code is open source (https://github.com/preply/design-system-visual-coverage).

Why Did We Need a Custom Way to Track the Impact of the Design System?

Preply is a data-driven company. All the teams elaborate on a long-term strategy (following the Good Strategy, Bad Strategy book approach) and define the metrics that allow them to measure their successes toward that strategy. The metrics are evaluated, challenged, discussed, tweaked, and approved. The design system team makes no exceptions.

We found that none of the existing ways to track the impact of design systems could tell the actual influence Path (Preply’s Design System) has on Preply’s users. Explaining static analysis data (tokens and components used in the codebase) to the CEO is hard. We needed to measure the impact on what the users see and interact with.

Since 2021, static analysis has shown a positive trend regarding the design system’s adoption, reporting a boost during Preply’s 2023 re-brand. However, this data expresses the influence on the codebase, not the users.

A screenshot of the internal design system stats we collected. The labels at the top left are the internal project names.

What Have We Implemented? The Design System Visual Coverage

The idea is:

1. The borders of the DOM elements are colored green for those generated by Path design system components and red for the others.
2. The borders’ size depends on the importance of the components.
3. The formula is green pixels / (green pixels + red pixels) (the final formula slightly changed, but this is the core concept). So, 100% means that “all the DOM elements of a page come from Path Design System components.”

Here is a screenshot of the visual coverage on Preply.com’s home page.

Here, you can see the visual coverage in action with borders colored and weighted based on the type of components.

The visual coverage tracks how much the Path Design System components are used in Preply’s products.

How Is the Visual Coverage Used?

The visual coverage serves different purposes:

The Design System team can:

1. Constantly track the outcomes of its initiatives. Please note that visual coverage is not the only metric we use.
2. Identify where Path Design System components are used the most and the least, and work with Product teams to increase the visual coverage.

The product teams have:

1. Numbers they can use as OKRs.
2. Straightforward tools to gather the visual coverage in seconds straight where they work: in production, locally, or in Storybook.
3. Straightforward tools to split the ownership of the pages across different teams. Most of Preply.com’s pages are made of components that belong to various teams.

To serve the mentioned purposes, we built an internal dashboard.

Our dashboard with teams, pages, and overall visual coverage.

Product Teams’ Ownership and Coverage Containers

On Preply.com, some pages are owned by a single team, whereas different teams own others. Visual coverage containers are areas of the page that are analyzed and counted independently.

The coverage containers in action: in the screenshot, there are three different containers: the Header, the Calendar, and the rest of the page. A different Preply team owns each of them.

While speaking with teams, we have always stressed that there is no good or bad visual coverage. In Preply, Path Design System is heavily used, and when it’s not used very much, there are strong reasons. For instance:

1. Calendars: Only one team uses calendars, and Path Design System does not offer pre-made calendar components. That team has low coverage compared to the others, and that’s fine!
2. A payment button that comes from a third-party integration. It looks like a Preply button, but it is not.
3. Every custom component has unique needs that the design system components do not offer.

At the same time, all the newly implemented product parts or the frequently refactored parts have the highest score, which has grown organically.

This topic is essential because we never ask teams to reach a specific visual coverage score. Visual coverage is a litmus test for many other factors heavily influenced by the Design System team and initiatives, not a goal to achieve at any cost.

Where to Run the Visual Coverage Count? On Users’ Devices!

The implementation of visual coverage (we shared an “implementation details” article here) ensures that it doesn’t affect the user’s UX. That’s why we can track visual coverage in production on users’ devices. This choice made everything more complex from a technical perspective (we also a/b tested it to ensure UX was not affected). Still, it also made the design system team independent in implementing, deploying, evolving, and gathering thousands of data daily.

If we decided to embed the visual coverage inside Preply’s E2E tests, we would have needed product teams to help us. However, product teams have product priorities, and depending on them would have meant delaying the project.

Some considerations for measuring the visual coverage on users’ devices:

1. Users use different devices with varying screen ratios and sizes, making using a fixed device as a reference impossible.
2. Users see different things based on the user-generated content and custom interactions with the Product.
3. The more a page is visited, the more calculations it sends. The final visual coverage score is an average of all the calculations.

As a result, the visual coverage is not 100% stable and can vary daily for the same page. We embraced this variability because the coverage would reflect what our users see!

Pages’ popularity impact

Again, the more a page is visited, the more calculations it sends. I want to stress this topic because it’s crucial to visual coverage. When we count the visual coverage, we send two pieces of data to the server: the number of pixels of the Path Design System components’ perimeters and the number of pixels of the other DOM elements. In DataDog (used for our dashboard), we calculate the average.

Imagine this scenario (numbers are imaginary):

The Home page is visited 5 times, and it sends the following events:

1. One event for the Design System pixels count. Let’s imagine the value is 10K
2. One event Non-Design System pixels: 5K

In total, the Home page sent 10 events. The Home page’s visual coverage is 67%.

The Setting page is visited 1 times, and it sends the following events:

1. One event for the Design System pixels count. Let’s imagine the value is 1K
2. One event Non-Design System pixels: 20K

In total, the Settings page sent 4 events. The Settings page’s visual coverage is 4%.

The math done in the dashboard is: all the Design System pixels count / divided by all the pixel counts. (10K*5 + 1K*2) / (10K*5 + 1K*2 + 5K*5 + 20K*2) = 44%. Can you see how the high popularity of the Home page weighs more than the low popularity of the Settings page? The final 44% is closer to the Home page’s 67% than the Settings page’s 4%.

Now, consider that our public and indexed pages are visited millions of times, and other “private” pages are visited hundreds of thousands of times. Other B2B pages could have “just” tens of thousands of visits. This is crucial information because, when we do the math, the teams behind the B2B pages could be entirely obfuscated by the popularity of other teams’ public pages. That’s why the Design System team cares more about the average per team than the final number (at the time of writing, our visual coverage is 77%, but most of the teams are above 85%) because the per-team visual coverage allows more granular monitoring.

The Components’ Weights

What counts for the visual coverage are the component’s perimeters. The perimeters can’t be the same for all the components for some reasons:

1. Path Design System components are UI components, while the pages contain hundreds of layout containers, which don’t impact the UX. UI components are smaller than containers, so the latter weigh more if the weight is the same.
2. A Design System’s divider doesn’t impact the UX as much as a searchable dropdown. At the same time, a Dropdown saves developers weeks of work while a Divider saves seconds.

So, the component weights are crucial for the visual coverage. We had a workshop to define the component’s importance. The idea is

1. To identify a 0–5 score for the component’s impact in terms of UI, UX, and accessibility.
2. We did it empirically to identify the importance of every factor: every team member voted and gave a 1–3 score to every factor; the importance is the sum of the votes.
3. Then, multiply the scores we identified at point 1 for the importance identified at point 2.
4. The result is a relative component’s score. In our case, it goes from 0 to 60.

The spreadsheet we used to identify the weight to assign to every component.

The actual weight we use for the visual coverage is reduced by a factor we identified through multiple iterations. For each iteration, we analyzed the score of the most representative pages (some of them are the most important ones, and some have low Path Design System’s components usage), empirically looking for the right coverage for the pages.

Other Design System Metrics

While elaborating on a long-term Design System strategy and a yearly roadmap, we thought of which meaningful metrics we could use to track the Path Design System’s success. The visual coverage seems to group all of them (because an increase in the visual coverage is the litmus of many other initiatives). Still, at the same time, it can’t reflect everything the design system team does. The metrics we care about are:

1. The Design System NPS score, of course.
2. Designers’ and developers’ efficiency: initially, this is a soft metric we ask through surveys without a real process in place to calculate it
3. The percentage of Path Design System foundations and components documented.
4. Path Design System platform features parity: the three platforms are Figma, Web, and App. Path Design System should support all of them at the same level (considering some unique capabilities and needs. For instance, Figma does not require Path’s components to be SSR/RSC-friendly).

And, more in general, we monitor

1. Path’s components usage: static analysis is insightful for the design system team.
2. Figma’s components detachments through Figma Analytics.

We also discarded implementing some metrics:

1. Figma adoption: We differentiate coverage (usage) from adoption (correct usage), but after some experiments and chats with other Design System teams, it seems too early for us to work on it, given the time required to implement and maintain something reliable.
2. The amount of CSS introduced per PR: This is one of the metrics Cristiano Rastelli identified in his Measuring the Impact of a Design System, which is the best resource on this topic. We suggest reading the article to find links to other companies’ activities to track their Design System impact.

Implementation details

Implementing the visual coverage, collecting feedback, improving it, validating it through a company-wide visual coverage increase initiative, etc., required months of work. Other companies we discussed showed interest, too.

If you want to experiment with it, we've got you covered:

1. Read (or send it to your engineers) the second part of this article, which is dedicated to the implementation details.
2. Fork, tweak, and use the visual coverage code.

Let us know what you think, and contact us at design-system [AT] preply.com, or directly with me on X, GitHub, or LinkedIn.

Resources

Preply looks very different from September 2023, and Path Design System is now quite connected to the new brand. If you are curious about how Preply executed a company-wide revolution in a few months, please read Maryna’s dedicated article, Turning the Page: Preply’s Successful Journey Through the Rebrand).

Special Thanks

I want to publicly kudos all the people involved in this project:

1. The whole design system team: Justine, Seif, Alex, Javi, Alisa. I love this team 😍
2. Vadym, Igor, Josh, Dmytro, Dominika, for the support and feedback 😊
3. Oleg, for creating the dashboard and helping with all the data.
4. Daniel, Alberto, for the early internal feedback.
5. All the Preply front-end engineers for supporting the design system, and this initiative. All the Preply app engineers for helping with the React Native implementation.
6. Javier, Serhii, Volodymyr, for supporting me with the React Native implementation.
7. Matteo, for trying the visual coverage with a different company.
8. Massimiliano, for the performance suggestions.
9. Dmitry, Omri, for the feedback.
10. Nicolas, for the frigging detailed review ❤️.

Would you like to join me and work in a purpose-driven organization where work, growth, and learning happen at the same time? Preply continues growing and we are actively looking for talented candidates to join our Engineering team! If you are excited about taking on a new challenge, check out our open positions here.