Replatforming the Lantern API

Background: What Is the Lantern Api?

All of this informs not just what we do today, in terms of the stories that make it to the homepage, get promoted on social, but what happens in the future: where do we put effort in following up, where should we perhaps start a series, and what might we just want to do less of.

A defining factor in finding the answers to these questions is audience data, powered by our internal editorial analytics viewer: Lantern, and Pink Lantern, our overview of the homepage in real time. The data to drive these tools comes from the dedicated ‘lantern-api’.

Article ‘performance’ is reported using several metrics:

  1. Article pageviews (number of times article pages are requested)
  2. Median Time on Page (having opened the article page, how long on average does the user spend there before moving on?)
  3. Percentage ‘Quality Reads’ (what proportion of page-views are deemed to be ‘quality’, ie. the user seems to have read a decent proportion of the article, and not just skimmed over)
  4. Number of significant reader interactions (eg. comments posted, social shares)
  5. Click-through-rate (CTR): [homepage only] for each view of the homepage, how many times is a particular article link clicked on.

The site-wide metrics are calculated for a variety of ‘slices’ of website traffic, according to various content/reader attributes, for example:

  • Editorial ‘Desk’ (section of the website)
  • Content type (eg. regular article, video, special package)
  • Reader type (anonymous, subscriber)
  • Reader location (country/region)
  • Traffic source (eg. has the page-view originated from clicking a link on search engines result pages / social media)

Lantern needs access to a ‘realtime’ datasource (up-to-the-minute data), but when Lantern was first built, no such datasource was available at the FT. The FT’s data warehouse (Redshift) was populated by a daily upload process, meaning the freshest tracking data available could be as old as 24 hours.

Therefore, a new database of (almost) realtime traffic data was created especially for lantern-api’s needs. To do this, tracking data coming through the existing data pipeline was processed and dispatched to a new, dedicated database (ElasticSearch).

Why Change It? Project Objectives

  1. Reduce costs. The infrastructure supporting this dedicated data pipeline/database was costing ~$17k per month)
  2. Reduce maintenance / complexity / key-person-dependency. The dedicated pipeline was prone to blips; only one engineer was fully au fait with the set-up; the technologies used were not well understood by the wider team.
  3. Eradicate duplication / divergence of business data logic. We wanted, as far as possible, to use the same data and logic as employed by our BI teams, to present a consistent view of key metrics to the business.

The Solution

Switching to using BQ tables that are populated by the standard FT data pipeline, supported by the team (‘Data Platform’) whose main focus is to provide such a service team would fulfil all three of the project objectives. Thus it was a straightforward decision, although various challenges followed…

Data Latency

Dependencies on other Teams

Speed

Some queries were predictable and commonly needed (to serve the main landing pages of Lantern), so it was possible to pre-fetch these on a scheduled basis, and cache them locally. This has two benefits:

  1. Very quick results served back to client
  2. Reduced BQ query costs

Many requests had to remain on an ad-hoc basis, because they were called too infrequently to be worth caching, and couldn’t be predicted, anyway. (There are countless combinations of filters available to the end-user in the Lantern UI).

We ended up employing a variety of techniques to improve speed:

  1. Caching query results.
  2. Pre-fetching data that is known to be commonly and frequently requested, eg. data for the main page.
  3. Fetching partial datasets in parallel, and then aggregating.
  4. Optimising the structure of database tables for the queries needed by our api.
  5. Pragmatism about how the queries we support (eg. certain queries disregard tracking data for articles older than 5 years)

Cost

Some of the techniques used to improve speed also had the effect of keeping reducing query costs, eg.

  1. Caching.
  2. Optimised structure of database tables.
  3. Pragmatism about how the queries we support (eg. rounding time-ranges to the nearest minute.

Some of the speed-saving techniques may have introduced extra cost, such as splitting queries up and running in parallel. Where possible, the splitting was done by raw dataset so that it didn’t incur too much extra cost. But speed v cost was a perpetual trade-off. Likewise the tradeoff with simplicity: some queries could probably have been written to run more cheaply/quicker but readability of the code would have suffered enormously.

Aligning Business Data Definitions

The Result

We have decommissioned the Lantern-specific legacy infrastructure, saving ~$13k per month in AWS costs (where most of the the legacy infrastructure was running).

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store