Using Cloud Firestore for Real-Time Alerts

At the start of 2018, I started work on MyTelegraph, a project to build the first personalisation product for the Telegraph. This culminated in presenting at Google Next London 2018 alongside Dan McGrath, on the topic “Building Scalable Apps with Cloud Firestore”. My involvement at Google Next 2018 was to showcase how the Telegraph had used Cloud Firestore to implement a seemingly innocuous feature; a feature that was both crucial to the success of the project and without Cloud Firestore, one that would have taken a significant engineering effort to build and maintain.

With the recent announcement by Google that Cloud Firestore is now out of beta and in General Availability it seems timeous to write a blog post based on my presentation, about how we used Cloud Firestore. Unfortunately the presentation was not filmed, so hopefully this blog post will give those unable to attend a brief overview of how we built the real-time alerting feature into MyTelegraph.

My aim is to follow up this post with posts on how we built it from a team perspective, the benefits we gained from using Swagger to define contracts for team interaction allowing independent progress, and how ultimately this has provided the foundation for the rebuilding of our Content API.

Context

The Telegraph is one of the UK’s most popular news sites, with journalists working across the globe, creating approximately 250 new stories on a variety of topics on a daily basis, resulting in millions of page views.

A number of these articles will not make it onto the main page, either above or below the fold, either because of the topics they cover, or the relative importance of the article to the main headline. In cases such as this, users have to navigate to areas of the site using a menu in order to locate new articles of interest.

Idea for MyTelegraph

One of our company goals is to grow our active registered readers to 10 million.

With that in mind, MyTelegraph was conceived in late 2017/early 2018 as a tool to “Make it fast and easy for registered users to discover and engage with relevant content”, to encourage users to return to the site.

• Fast — by alerting users as soon as new content they are interested in is available on the site, across all of their devices.
• Easy — by allowing users access to all the articles they are interested in from a single location on the site, on all of their devices.
• Registered users — we need to know who our users are so we can show them the articles they are interested in.
• Discover and Engage — by allowing users to choose which topics and authors they want to follow, and provide the ability to save articles to read later.
• Relevant — personalised content feed for each logged in user.

Our aim was to build and release an MVP in time for the 2018 FIFA World Cup.

Features Overview

The zoomed in screenshot taken from a mobile device shown above, shows how a user can follow an author (or topic), and the alert (red dot on the My Feed menu) to let the user know when there is a new article on the site they may be interested in.

Clicking on the “My Feed” button that shows the alert, will take the user to a screen that displays a list of articles, in reverse chronological order, based on the user’s interests, as shown above. When new articles are available on the feed, a toast is shown to the user to allow them to refresh the feed page.

If you look at this through different glasses it looks fairly similar to a chat application, where users are part of a channel (think of users following a topic or journalist), and need to be alerted when someone has added something to the conversation they may want to know.

Building real-time infrastructure to support a chat application is not core to our main business objective, which is to deliver content to and improve engagement between users and The Telegraph, and would have hampered our ability to deliver on time.

Our mobile teams had reported good results with Firebase and we started looking at using Cloud Firestore for a number of the project requirements, including the ability to push articles to the user’s device automatically for offline use, and alert them to these articles across all of their devices, making use of the SDK and out of the box functionality.

With a project goal to deliver by the FIFA World Cup, using Cloud Firestore became appealing.

However, we were hampered with moving forward with Cloud Firestore due to a lack of support (at the time) for using service workers in one major browser we support, a feature which the Cloud Firestore SDK used in order to sync data between the client and server.

In the interest of time, we started development by building on a microservice based architecture to create well-defined boundaries for the features and decided to implement the alert later.

Fortunately, browser support for service workers was introduced a couple of months into the project, which allowed us to use Cloud Firestore for the main feature, namely alerting the user to new content.

Architecture Overview

Having already implemented the follow feature, and feed feature, we looked at rolling Cloud Firestore into the application using its real-time database capabilities, together with managing millions of concurrent connections specifically for the alert. This would allow us to experiment with using Cloud Firestore, and remove the overhead of implementing and managing a real-time database and alerting infrastructure.

The diagram above shows an overview of the architecture of MyTelegraph. At a high level:

• When the user loads up the Telegraph home page a request is made to load the user’s preferences.
• These preferences are then registered via an Event listener, on the user’s browser or native app, with Cloud Firestore to listen for any changes to the article metadata.
• When the user navigates to their feed, checks are made against the preferences tables to get the topics and authors that the user follows.
• A query is then performed against the content store in real-time, to retrieve articles the user would be interested in and build a personalised feed showing a list of articles in reverse chronological order.
• Once the feed is loaded, the preferences store is updated with a timestamp to indicate when the user last loaded their feed, and this timestamp is used to listen for changes to Cloud Firestore in real-time, alerting the user when a new article is published.
• Finally, when a new article is published on our content management system, the associated article metadata is loaded into Cloud Firestore.
• The timestamp of the article is then used to compare against the last time the user refreshed their feed and triggers the event listeners across all of the user’s devices if the user is following the author who wrote the article, or topics that are covered in the article.

Cloud Firestore Document Structure

This screenshot shows the authors collection, and the date and time the last article a particular author created was published.

Topics Collection

This screenshot shows the topics collection, and the date and time an article was created and published that was tagged with a particular topic.

LastChecked Collection

Lastly, this is the collection of users and the date and time they last visited their feed page.

User ID’s are masked for GDPR reasons.

The collections used for the basis of this architecture are simple in nature, with minimal information. But they could easily be extended to include additional information, such as the article contents, that would then be pushed to the user’s device in real-time.

Code Overview

I do not have space to cover the entire implementation, but below I show how simple it is to setup Cloud Firestore to be used in a project.

Setup

The MyTelegraph application is a single page application built using React and Redux, with a number of libraries, of which Cloud Firestore is one.

The snippet of code shown above demonstrates how to connect to the database and set up the collections that we will be using, specifically the topics and authors to listen for changes, and the lastChecked collection to get the time the user was on the Feed.

Retrieving documents

There are two ways to retrieve data stored in Cloud Firestore, as follows:

• This first is a call to get() method to retrieve the data.
• The second is to set a listener to receive data-change events via the onSnapshot() method.

Either of these methods can be used with documents, collections of documents, or the results of queries.

Method 1: get()

The code snippet above shows the first method, by calling a method to get the data.

In this case, we are getting the last time the user opened their feed page.

You can retrieve multiple documents with one request using a .where() to retrieve documents that meet specific conditions, and use get() to retrieve the results.

Method 2: Event Listener

The snippet of code shown above shows the second method, using an event listener, to respond to data change events. When you set a listener (via the onSnapshot method), Cloud Firestore sends your listener an initial snapshot of the data, and then another snapshot each time the document changes.

It is this feature that we are using to alert users when there are changes to the backend when a new article is published.

Conclusion

Cloud Firestore was simple to implement and allowed us to build a real-time alerting system to keep users up informed when new articles they may be interested in are published, across all their devices. Navigating to the feed on one device clears the alert from all other devices that the user may be logged into.

We did this without needing to become real-time messaging and database experts and this reduced the amount of infrastructure that we would have had to implement and maintain had we not used Cloud Firestore. This allowed our engineering teams to focus on delivering content to our end users.

Alex Mansfield-Scaddan is a Solutions Architect for The Telegraph.