Real time - Core Web Vitals Monitoring

In today’s digital world, if your application doesn’t provide a user experience, you would lose your users! A bad user experience impacts conversion rate and SEO ranking of your page. Thanks to Google, we have a tool that allows us to check page performance by calculating web vitals.

What are Web Vitals?

Web Vitals are quality signals that are essential to deliver good web user experience. But Google gives weight-age a subset of web vitals, critical to web user experience are known as Core Web Vitals (CWV), that assess the user experience on criteria such as loading performance, responsiveness to user input, and layout stability.

The three Core Web Vital metrics are as follows.

• Largest Contentful Paint (LCP): Measures loading performance, how long it takes to load the largest image or block of text in the viewport.
• First Input Delay (FID): Measures interactivity, how long it takes for the browser to respond when a user engages with the page (button click, tap, etc).
• Cumulative Layout Shift (CLS): Measures visual stability, to determine whether there is a major shift in the content on-screen while elements are loading.

Tools to assess web vitals

There are various tools that measure Web Vitals, and help you diagnose and fix user experience issues, with ease.

Field data tools (CrUX report, Page Speed Insights, Google search console) provide restricted data-set, and it is difficult to debug the issue with limited information. And,these tools report gives web-vitals field data using a granularity of approximately one month. Waiting around one month to see the impact in CrUX data — only to realize your fixes weren’t sufficient — is way too slow.

To get a more detailed and immediate feedback on your site’s performance, it is best to choose your own data set and get a tool to assess these points. For this, you may either choose a third-party solution or create your own RUM tool. This way, you can start collecting field data at a granular level, making it easier to identify issues and test possible solutions​.

Creating your own tool for Web Vitals assessment

Google’s web vitals library will help you collect web vitals, in a more real-time manner than what the CrUX data allows. However, you’ll be responsible for how this data is reported, stored, and analyzed.

Reporting Web Vitals: You can either record these metrics in any analytics tool like google analytics (through GA events) or you can have an end point ‘/analytics’ service to collect web vitals metrics to measure user experience.

Below is the code snippet for collecting metrics at ‘/analytics’ endpoint through web vitals library, keeping a queue of all metrics that were reported and flushing the queue whenever the page is unloaded.

Basis requirement, you can send metadata along with web vitals to an analytics service. Below is the sample request payload that has additional metadata (like application ID, pageUID, event name, pageName etc.). If needed, you can also pass the metric details such as the HTML elements impacting web vitals values.

** FCP, TTFB, LCP values are in milliseconds

1. System Application → It can be any application/ service, having ‘/analytics’ service that ships the Web Vitals data to KAFKA topic
2. KAFKA → Here, Kafka adds as event source for logstash
3. ELK Stack → Below components of this stack, helps to store, and analyze Web Vital metrics

• logstash: To pick the data from KAFKA topics and process data (transform/ aggregation/ enhance) as per our requirement
• Elastic search: To store processed data in Elastic search Index
• Kibana: To visualize the web vitals metrics, Kibana dashboard create views that pull together charts, maps, and filters to display the full picture of your Elasticsearch data (web vitals data)

Core Web Vitals metrics & thresholds

Each Core Web Vitals metric has associated thresholds, which categorize performance as ‘good’, ‘needs improvement’, or ‘poor’

To pass on CWV assessment, each CWV metrics should be in a good threshold category. A good threshold to measure is the 75th percentile (p75) value of all page views to that page or site, segmented across mobile and desktop devices.

To analyze CWV, we created stacked bar charts and tables in the Kibana dashboard. Stacked bar chart shows CWV data categorization of all page-views, meaning what percentage of page-views have ‘good’, ‘need Improvement’ or ‘poor’ LCP. And tables show p75 percentile value of each CWV for all page-views and page-type (tag) wise.

In the snapshot below, 78.154% page-views have a good LCP score, 2242 ms is p75 LCP score of all pages, and 1802 ms is p75 LCP score of home page.

Combining multiple web vitals values

For a page-view, if we need to send web vital entries multiple times to ‘/analytics’ service (e.g. CLS on each layout shift). We need to combine these all values, triggered from the same page.

To achieve this, we can pass a unique identifier (pageUID — unique for a page-view) as a payload in every ‘/analytics’ request(s) for a page and combining/ merging all web vitals entries, having same pageUID, can be done logstash layer and same can be visualized​ at Kibana dashboard.

Thanks for reading this article and share your views !

Reference links

https://medium.com/naukri-engineering/centralizing-logs-at-naukri-com-with-kafka-and-elk-stack-cdce4fcc4c44
https://developers.google.com/learn/pathways/web-vitals
https://www.smashingmagazine.com/2021/04/complete-guide-measure-core-web-vitals