Data polling and DOM comparison with React, guided by tests

A football match is composed of many different events. Before a match begins, for instance, football fans crave details about line-ups, previous matches’ results, and related news. After the start of the game, on the other hand, fans desire to see facts like the match’s score, video streams, statistics, the competition’s table, and so on.

For reasons like these, the Match Page is one of the most important pages at OneFootball.com. On this page, football fans are able to find all these details and facts with real-time updates.

Example of a match with a live stream.

Some of these pieces, such as the competition’s name and the kick-off date, are static pieces of data that don’t require constant updates. However, most of them need to be frequently updated during a single session. The match events and match score, for instance, are two of the most critical ones.

In this context, one of the challenges we faced while developing this page was figuring out how to keep the data always up to date without compromising its performance. After all, we hate websites which drain our computer’s resources, but we also hate to lose the exact moment a goal is scored.

Example of a match with live match events.

In order to build a frequently updated webpage we had two main options: we could either wait for a notification (push) or keep asking our backend server for new data from time to time (polling).

Regardless of the choice we would make, a few requirements needed to be met. These were:

• The data should start being updated in “real-time” two hours before the match’s kickoff
• The data should stop being updated in “real-time” after the final whistle
• Only components with new data should be re-rendered

Taking these requirements and a few more factors into consideration (such as our current architecture, implementation simplicity, and a time-to-market need), our engineering team decided that the second option (polling) was the best fit for our scenario.

This article talks about how we implemented it and ensured that the code we developed was doing what we expected it to do.

Task Breakdown

Rome ne s’est pas faite en un jour (or Rome wasn’t built in a day) — Li Proverbe au Vilain

Following the Continuous Integration (and, going further, Continuous Delivery) philosophy we’ve been implementing in our team over the last few months, we broke this task into smaller pieces and delivered it in small batches for a shorter feedback loop. This way, we eased our code review process and made sure that no surprises would be added.

In addition, before starting to write our very first line of code, we designed all the scenarios we would need/like to cover for this task using the Gherkin syntax; and, before writing our first line of implementation code, we created our test scenarios. First things first; Test Driven Development (TDD) rocks 🤘🏼

Talk is cheap, show me the code

OneFootball.com is brought to you through the following stack:

• TypeScript as our programming language
• React as the library for web user interfaces
• Vitest as the unit test framework
• React Testing Library for React DOM testing utilities

With that in mind, let’s dive a little deeper into how we did all these things happen in production.

Data Polling

In order to get data from time to time, we needed to prepare some tools/utilities: we needed to be able to properly play around with timers.

Moving from Imperative to Declarative

After struggling with the imperative APIs provided by setTimeout and setInterval (facing react-hooks/exhaustive-deps errors during the development), we took a step back and restarted our task.

This time, our first step was wrapping the existing imperative APIs into declarative ones. Based on this great article written by Dan Abramov, we created “declarative” equivalents for setTimeout and setInterval.

Following TDD practices, we defined our case scenarios:

// useTimeout.spec.ts

describe("Scenario: start timeout", () => {
  test(`
    GIVEN a delay period in milliseconds
    AND a callback function
    WHEN this delay period has passed
    THEN it should invoke the callback
  `)
})

describe("Scenario: stop Timeout before unmount", () => {
  test(`
    GIVEN an already started process
    BUT the delay has not passed yet
    WHEN the component which renders the hook is unmounted
    THEN it shouldn't have invoked the callback
  `)
})

// useInterval.spec.ts

describe("Scenario: start Interval", () => {
  test(`
    GIVEN an interval period in milliseconds
    AND a callback function
    WHEN this interval period has passed
    THEN it should invoke the callback
  `)
})

describe("Scenario: stop Interval", () => {
  test(`
    GIVEN an already started process
    WHEN a undefined interval is passed
    THEN it should stop (clear the timer)
  `)
})

describe("Scenario: stop Interval before unmount", () => {
  test(`
    GIVEN an already started process
    WHEN the component which renders the hook is unmounted
    THEN it should stop (clear the timer)
  `)
})

After that, we implemented each test scenario through the Red, Green, Refactor technique. The final versions of both files are available here and here.

With these building blocks in place, it was time to work on our polling algorithm.

Polling Algorithm

The requirements for our polling algorithm were:

• We should be able to start it immediately, and keep polling based on a given interval
• We should be able to start it after a delay, and keep polling based on a given interval
• We should be able to stop it
• It should stop polling before the component is unmounted

Once again, we started with the testing scenarios:

// usePolling.spec.ts

describe("Scenario: start polling", () => {
  test(`
    GIVEN an interval period in milliseconds
    AND a callback function
    WHEN the hook is rendered
    THEN it should start polling (invoke the callback)
  `)
})

describe("Scenario: start polling after a given delay", () => {
  test(`
    GIVEN an interval period (in milliseconds)
    AND a delay period (in millisecond)
    AND a callback function
    WHEN the hook is renreded
    THEN it should wait the delay period
    AND start polling (invoke the callback)
  `)
})

describe("Scenario: stop polling", () => {
  test(`
    GIVEN an already started polling process
    WHEN a undefined interval is passed
    THEN it should stop polling (clear the timer)
  `)
})

describe("Scenario: stop polling before unmount", () => {
  test(`
    GIVEN an already started polling process
    WHEN the component which renders the hook is unomounted
    THEN it should stop polling (clear the timer)
  `)
})

Then, we implemented one by one through Red, Green, Refactor cycles. You can find the whole test file here.

Having all the hooks we needed ready to use, we could move forward and start to build the last hook, useMatchPage, which would be responsible for handling the polling strategy and for updating the Match Page components.

Dom Comparison

Currently, the Match page receives all the components it should render (and the order that they should be rendered) from the backend, following a server-driven UI approach. These components are named XPA components, which stands for experience architecture components.

The challenge, here, was to ensure that only XPA components with new data would be re-rendered.

Since we have a unique key for each state of each XPA component, it turned out that the comparison part was just a matter of comparing the previous and the current key. With that, we just needed to figure out how to prevent unmodified components from being re-rendered.

After playing around a little bit with React’s built-in memoisation capabilities, we got a solution with the React.memo function:

const arePropsEqual = (
  prevProps: props,
  nextProps: props,
): boolean =>
  prevProps.contentType.$case === nextProps.contentType.$case &&
  prevProps.uiKey === nextProps.uiKey;

export const XpaMemoisedComponent: FC<props> = memo(
  ({ contentType, uiKey, ComponentsResolver }) => (
    <ComponentsResolver uiKey={uiKey} contentType={contentType} />
  ),
  arePropsEqual,
);

Profiling

In order to make sure everything was working as expected, we did some tests using the Profiler tool provided by React Dev Tools.

We started to randomly serve our development page with two mocks: one based on a real match (Fortaleza vs Ceará), and the second one being a slightly modified version of the first one.

Once the page started to load (and started to poll these mocks), we saw that all children (XPA components) were being rendered on the first render:

Match page first render

Once Google Ads loaded, a second render was made, but only the XpaGoogleAdsPlaceholder component was re-rendered:

Match page second render

On every fetch, the LoadingProgressBar component was rendered. This was the page's third render:

Match page third render

A new fetch with no data changes (fourth render):

Match page fourth render

(Fifth render) Another fetch, this time with data changes (competition name on MatchInfo):

Match page fifth render

(Sixth render) Another data change on the competition’s name (MatchInfo):

Match page sixth render

(Seventh render) No data change = no components re-rendered:

Match page seventh render

(Eighth render) Again, no data change = no components re-rendered (besides the LoadingProgressBar component):

Match page eighth render

After some iterations, we were pretty confident that everything was working as we planned.

Polling Strategy

Last, but not least, we needed to implement our real polling strategy: when should we start to ask for new data? What about the delay? In which match periods should we start (or stop) polling?

With all the building blocks we’ve made in the previous steps, we just needed to define our scenarios.

Basically:

• No polling until two hours before a match’s beginning
• Constant polling for live matches
• No polling for matches that have ended

After defining and meeting the parameters that we needed for our product strategy, we implemented all the scenarios, once again, through TDD and the Red, Green, Refactor technique.

Extra Time

Struggling with Timers

While writing some tests which required the usage of waitFor, from react-testing-library, we realised that waitFor and fake timers are not a great match (#631).

In order to overcome this, we had to alternate between fake and real timers:

describe("Scenario: fetch and update a new layout", () => {
  test(`
    GIVEN an initial Layout
    WHEN the polling interval has passed
    AND a new Layout is fetched
    THEN it should return this new Layout
  `, async () => {
    [...]

    vi.useRealTimers();

    await waitFor(() => {
      expect(result.current.layout).toEqual(newLayout);
    });

    vi.useFakeTimers();
  });
});

Yeah, it doesn’t look like the most beautiful piece of code in the universe. However, it does what we needed at that moment. In addition, since automated tests covered all the scenarios, we were confident to ship it to production.

Fixing a Bug in Production

After a few days of running this solution in production, we realised that we had a bug: some matches were falling into an infinite loop.

Our first guess was that the issue was only happening for matches that was going to start in one month. From this starting point, after some investigation, we discovered that:

Assuming “current date and time” as 26. Apr 2023, 10:34:

• A match happening on the 21st of May, in the afternoon, had a problem.
• A match happening on the 21st at 02:00 didn’t have the problem.

Diving a little deeper, we discovered that:

• A 2,193,025,674 ms delay was breaking the usePolling algorithm.
• A 2,124,205,918 ms delay worked fine.

In the end, the answer to the issue was Mechanical Sympathy.

We realised that the maximum value for useTimeout was 2,147,483,647 ms. That's why kickoffs happening after about 24.8 days were buggy.

Limiting pre-match delays to 2,147,483,647 ms solved the problem :)

It’s interesting to notice that, having a fully tested stack made our lives much easier. Once we got into a reproducible scenario, we left the browser and worked only on test files. It saved us a lot of time and, more importantly, gave us the confidence that our changes would fix the problem and wouldn’t break any existing rules.

Tests rock! 🤘🏼

Conclusion

This was a really interesting task to work on. We got a big task, broke it into smaller pieces, used TDD to develop it, and took advantage of tools like React Dev Tools’ profiling to finally deliver a smooth and seamless experience to our customers.

It’s also worth mentioning that using a test-driven development approach gave us the confidence to ensure that our code was doing what we expected. In addition, it guided us to build a friendly API after refining it throughout the red, green, refactor cycles.

In the end, we realised that, with the right tools and the right techniques, we can build software that we both enjoy building and are proud of.

Ps.: a special thanks to Juanmanuelveleztorres and Diego Mosquera Soto (engineer and manager of the Consumer Web team respectively) for all the support and pair-programming sessions we made during the development of this task 💚