We Can Collect Data, Now What?
Part I of this article goes over the process and tools we use in order to collect data, store it, parse it, and make it query-ready. Now that we had a workflow for all this, we needed to determine what kind of values we wanted to store and how to do so.
Meaningful Data Points
Now we needed to determine the points in the processes of our codebase that would provide the most telling information and help us to pin-point areas where we could reduce latency and improve our performance.
Since this particular code is used to make ad requests and render the response as an ad on a publisher’s page, some initial data points were pretty self-evident. Of course, we started with a Mark when our file first loads on the page. Then, as we send ad requests and receive responses, we set a Mark for each of those and use those Marks to create Measures for how long our requests are taking. However, these speak more to the user’s network speed as well as the performance of our back-end team.
To really begin seeing data that’s informative of the front-end codebase’s performance, we needed to add Measures (and the necessary Marks to create these Measures) for the processes that occur between responses and subsequent requests as well as the creation of objects used for rendering and when they are finalized. This final Mark occurs when the ad has fully rendered on the page.
As we began to store and analyze data from these points, of course additional metrics began presenting themselves as useful or necessary inclusions. Some such metrics include when we receive consent data for regions that currently require users’ knowledge and opt-in for usage of data (such as GDPR in the European Union and CCPA in California). Also, we found that it would be better to assign specific Marks for the different ad products and implementation methods (e.g. Prebid.js versus a direct relationship with the publisher page). Later on, this will help us narrow down problematic areas and troubleshoot issues that arise.
Finding What Matters Most…
In order to create meaningful visualizations of the data we had begun collecting, we first needed to determine which metrics should be used.
After analyzing the initial information we received, we quickly found that the Marks alone were too heavily influenced by outside factors. For instance, many pages do not load our initial script file until a user has interacted with the page. On a page that has a fair amount of content above the fold, a user can read through content for a good amount of time before actually interacting with the page, causing the value of all Marks to be higher than if our initial script file was loaded immediately when the page loaded. Another outside factor that influenced our Marks was whether or not we were served below the fold. For our ad products that will be viewed only after a user has scrolled, we do not finish creating or rendering the ad until the ad space is within the user’s viewport. Again, this caused inflated values for some Marks.
We did, however, find that these Marks would still prove useful in finding information between two different parts of our process. For example, some of our ads require asynchronous requests to be made with the first needing to be parsed and processed before making the second request. In these cases, we can subtract the value of the Mark created on receipt of the first response from the value of the Mark created when sending the second request in order to find how long it took our code to process the response of the first request.
…And Visualizing It
Once we had found which metrics and calculations we would need in order to provide the most meaningful reports, the final step for this performance monitoring tool was to be able to see the outcome in graph form. For this, we decided to use the third-party tool that is already used by our company for visualizing other reports: Looker. Looker allows for our data team to make SQL queries to our data that’s stored within Athena (how it gets there is covered in Part I).
Within Looker, custom reports can be created with built-in equations to handle computations - like the example above - for calculating time between processes. It also allows us to see metrics over time, averages, and compare multiple metrics at once on the same graph and with different graph types.
As an additional feature, we added the ability to see the data for any ad we were viewing on any page in real-time.
We already have a tool for displaying information about a page and any of our ads that appear on it for users who are currently logged into our company-wide dashboard. This tool allows the user to see more information about each individual ad on the page by expanding an overlay with tabbed sections of information. All that was needed was to add a new tab for the performance data we could now collect.
In this view, the values of the most important Marks for this particular ad instance are printed into a table. For the most informative Measures, we added a waterfall visualization that would feel familiar to those who have used browser developer tools.