How Zillow tests Clickstream Analytics

At Zillow, the largest real estate network on the web with the most unique visitors in the U.S.¹, we enable people to unlock life’s next chapter, by turning on the lights on the housing market. In order to find out how the users are behaving on the site, we use analytics. Analytics implementations allow developers to embed tracking events in pages that can then be correlated with user activity on the site.

The web pages are instrumented with calls to an analytics library. The instrumentation is usually of two types:

1. Page Views: These events signal that a particular page was viewed by the user.
2. Events: These are non-pageview events, and can be used at various places for different purposes. For example, if a user scrolled down and looked at a section of the page that is not normally visible, an event would be generated.

The analytics library is usually a third party product. In our case, we use Google Analytics. Other brands in the Zillow Group umbrella use other providers.

In this post, we will explore about the different ways we test clickstream analytics events, and how we got to this place.

Analytics Events Lifecycle

Analytics events are defined by product owners, and instrumented by developers into their products. Sometimes, the events are instrumented without input from product owners. Until now, Zillow did not have an overall schema or guidance on how to author events for the website. In order to fix that, we have been working on a standard definition.

Analytics events ultimately end up in Zillow’s datalake. We have a process that reads the events from Google BigTable, and dumps them to our datalake. Here they are put in a hive table for easy query by product owners and analysts.

Since the events are instrumented by developers, and there is a lack of standardization in naming, existing events might get changed. For example, a developer might see an event for a contact to an agent that is currently named as “agent contacted” and might change it to “Agent Contact by Phone”. Or they might change the casing of the event to suit their personal preferences. When that happens, it can break downstream consumers, because the event they were searching for has now been renamed.

Further, as product development goes on, and features are added or removed, events might stop being fired accidentally. That also causes trouble to downstream consumers or events that are being fired could be missing critical fields in the payloads, which makes analysis challenging.

Event Testing Strategies

In order to solve this problem, we have two kinds of tests, functional tests and schema tests.

In functional tests, the test case completes some activity on the web page, and verifies that the correct events are fired. These tests are usually performed with an embedded browser. At Zillow, most of our test cases are written in Python, and we use Selenium along with ChromeDriver for browser automation.

These tests have the advantage that we are testing functionality that generates the events. However, one downside of these tests is that depending on timing, events may get lost, or not fired at all. In this case, tests do not give a good signal for analytics.

The other kind of tests we use are passive tests. In these test cases, we validate that events being fired meet the minimum requirements for the event schema. For example, we make sure that every event has a non-null AnonymousId, and the page URL is in one of the custom dimensions. These tests are not functional, but they do provide advanced warning if someone ships a change that breaks the event schema.

In a previous post, we described the Data Streaming Platform. This system is used to collect and send analytics data to the datalake. It supports both clickstream analytics, and backend eventing.

We use the same system internally in test environments to collect analytics events from users, automation tests.

Analytics testing with Data Streaming Platform

In the test environments, we configure all our microservices to use the DSP’s test instance. In addition to sending analytics events to the real provider, we also send the events to DSP. DSP writes those events into a Kafka topic and then they are backed up to S3 using a Kafka-connect job.

The clickstream events are sent to Google analytics and are available for consumption from the Google Analytics console. However, since the same raw event is also being sent to DSP, it is available for real time consumption from the Kafka topic, and for batch consumption from S3.

Functional Tests

The functional testing pipeline works as follows:

• A selenium test runner loads pages from internal sites, and invokes activity on the page as if a user was looking at it. This might include scrolling, clicking on buttons, entering text, etc.
• It verifies that certain outcomes happened, for example, a UI text element was rendered with the correct value.
• Finally, it reads the events for that anonymous user from the kafka topic and verifies that the correct events were fired. For example, if it was verifying a PageView event, it would make sure that the event had all the data necessary for analytics.

Passive Tests

The passive tests are executed by a test runner. Each test is executed periodically, typically on a frequency of once every 5 minutes.

Each test works as follows:

• Seek to a prior position in the kafka topic. This is typically set to Now-10 minutes.
• Read each message, and verify that its structure meets the schema requirements.

For passive tests, the test case can be as simple or complicated as required. Usually the simple tests look at each event in isolation and make sure that they meet requirements. The complicated tests can look at multiple events at once. For example, you could have a test that verifies that if a Contact event was seen it was preceded by a PageView event for a Home Details Page.

Conclusion

With this system, we are now able to monitor the quality of our analytics events. It allows us to surface problems to feature owners sooner, before the features ship. It is an extensible system, which makes tests easy to write and maintain.

¹ Source: Comscore Media Metrix® Multi-Platform Key Measures, Real Estate, Total Audience, September 2020, U.S.