The Times Digital Transformation Programme

Part 2: Choosing the Technical Approach

This programme is a major project to re-engineer the technology behind the digital versions of The Times and The Sunday Times. It’s transforming a disparate tech stack that evolved from the print editions of the newspaper to a new scalable, performant and rationalised architecture that can support future subscriptions growth and technical innovations.

This article is part two of four:

Part 1: Our business rationale and processes

Part 2: Choosing the technical approach

In this article, I describe the decision process around the technical approach to the digital transformation project.

Part 3: The technical implementation

Part 4: Lessons learned and key takeaways

Identifying the problems and objectives drives out the right solution

The problem

The Digital Transformation Programme came about because we recognised that our platform for publishing content on the web and in apps was not optimal. It’s largely evolved from systems designed to support print rather than digital, which has led to a complex ecosystem that’s hard to support or grow. Some of our consumers are missing out on the rich, interactive components our digital editors create to help with their story telling. These render brilliantly on the web, but are completely missing from our apps.

To move fast and deliver cross platform features we are held back by both the architecture and the front-end as a delivery model. Moving into voice, AR, VR or some new thing is much harder with the current siloed stacks.

So we recognised that we have a complicated platform that doesn’t serve our readers well, is more difficult to develop than it could be and is not future proof. We decided to fix that.

Our objectives

Our criteria are clear, if not simple. We want a single codebase generating a consistent UI across web, Android and iOS that is easy to develop and gets us ready for the future.

We are touching everything, front to back. We are:

• Migrating the datastore from ElasticSearch to PostgreSQL.
• Consolidating a myriad of content serving APIs down to one central, public, GraphQL.
• Rationalising and optimising all the processes and software that prepare and serve the content.
• Rebuilding the user interfaces from a single code base.

More than just building a better product suite for the company, we wanted to build a development environment where:

• We are working on product differentiators that give us a competitive advantage; making The Times an exciting place to work.
• We have a single, extensible code base.
• We have performant applications that are easier to develop, innovate, support and maintain.
• We are creating a better data store with a simple and consistent way to access the data through a single API for all content reading and writing.
• We work in smaller, more flexible, truly cross functional and properly agile development teams.
• We have a continuous delivery pipeline with testing at its heart.
• We have a single platform to render interactive content to our digital readers however they consume it, with a better suite of content creation tools that integrate smoothly with the rest of the tech stack.
• We have the potential to easily adjust and extend the platform in the future.

It’s a major commitment that means rebuilding our digital platform from the ground up, taking more than a year to do so.

The solution

Our chosen path to being truly cross-platform is JavaScript based; Node.js, GraphQL and React Native.

Being cross-platform is hard and delivering this programme throws up some interesting challenges. React Native is still young. Not many companies are building web and native from a single code base. Embedding videos, ads and other third party components cause constant issues. Server side rendering (SSR) is tough. Version management with JavaScript and npm is hard when packages don’t follow semver, you use a monorepo and have 8000+ 3rd party dependencies with React/React Native incompatibilities thrown into the mix. Do you use Haul or Metro? Documentation and community support is thinner than it could be. Third party libs (such as svg libs) are tricky. Android support is still second class. Sometimes it just crashes (Red Screen of Death). Storybook add-ons cause complications.

But, for us, the case to go ahead was compelling. Let’s look at how we are delivering the technical aspects of the project.

How do you get from a myriad of APIs down to one?

Build a single API and migrate

Because our old current system has evolved organically over time, developed by different teams with different requirements and different codebases, it has become unwieldy and complex. The architecture is old, the connections between the pieces are clunky and the data is spread all over the place.

The SmartPhone API (SMAPI) that serves content to our mobile apps is very hard to develop on, has long build times and is virtually impossible to release. Omnis Plus was written in a language that none of our current developers know. Kindle support was bolted on as an afterthought using an external agency. There has been a divergence in approach and it has lost cohesion.

We are progressively eradicating them all and replacing them with a unified back end that solves those problems, provides a single source of truth and enables being cross-platform.

Our new Times Public API (TAPI) is fast and safe to develop on. It is strongly typed, written with JavaScript and has an excellent test approach. It is easy to deploy and provides a single place that serves all our content. Being strongly typed leads to rigorous coding standards and a strong testing strategy.

The underlying RDBMS is AWS RDS Aurora PostgreSQL. We decided to use a managed database service that frees us to focus on the product and domain knowledge.

TAPI uses GraphQL, which is very consumer friendly. Whereas Rest APIs rely heavily on tools such as Swagger for documentation, GraphQL is evidently self descriptive, making integration and querying safe and easy.

With traditional APIs you have to get the forest when all you want is a tree because you cannot control the data you receive. GraphQL allows many clients with many cross-cutting concerns to retrieve exactly what they need, however different their requirements. The queries that GraphQL exposes provide transparency around the hierarchy of the data components, making it super easy for clients to understand the model and retrieve the data they need. It provides a true server/client divide, with the client in the driving seat. Restful APIs provide homogeneous queries across clients, whereas we now share queries across platforms.

TAPI is defined in GraphQL with the infrastructure described in Cloudformation. It’s serverless, runs on lambdas and retains no state, leading to reliable deployments, blistering speed and super scalability. Having many APIs in your ecosystem leads to many logging and dashboard solutions rather than a unified worldview of your real estate that a single API enables. We have added Cloudwatch and the full suite of alerting so we always know the system health.

Adopting GraphQL took some thought and persuasion. We had to win the hearts and minds of the technology function, because along with the benefits comes compromise and concession. Querying is now client driven rather than server supplied. We cannot deliver breaking changes because GraphQL does not support API versioning. The new API no longer serves nicely formed HTML, so the clients need to transform and render it. GraphQL won the debate because the benefits are greater than the drawbacks.

How do you write a single UI across web, Android and iOS?

This is a bold objective. In April 2017 we started a series of spikes to consider replatforming The Times. This included lots of technical analysis, proof of concepts and thinking about what we wanted to achieve for our readers now and in the future.

We looked at several ways of achieving it (back at the beginning of the project; things might be different when you read this, of course!)

Option 1: Stay native

We recognised that we need discrete iOS and Android applications as well as a web site, but building them with native code (HTML/Swift/Java/Kotlin) was not efficient. Hiring native developers is hard and you end up with three teams and codebases instead of one, risk having different feature sets and certainly having uneven implementations.

Option 2: Progressive web apps

One option was to abandon the native apps and move to a Progressive Web Application. That would certainly solve some of our problems, but it is not what our readers expect. Research shows that The Times is seen as a luxury brand and we believe that native apps are much better positioned to service that.

Option 3: Flutter

We felt Flutter was still too young (it only reached Beta many months into our project), it was ‘too Google’ and did not provide web views.

Option 4: Angular/NativeScript

We felt this was still surprisingly niche and therefore hard to hire for.

Option 5: Microsoft’s React XP

This was not mature enough, provided a vendor lock in we did not want and there was generally no appetite to choose it.

Option 6: React Primitives

We liked this and would have considered it, but we felt it was more of a pet project of Leyland Richardson than something fully embraced by a wider community.

Option 7: React Native/React Native Web

We felt this was most mature, had larger and broader community engagement, it leveraged our existing React knowledge (and would also make hiring easier) and it emphasised sharing code across native and web, which limits us to a subset of features. Add to that that it was already in production at Twitter, and we found our winner.

Once this was complete the whole of the estate was looked at, epics drawn up and sized relatively with an estimate of six years worth of work. This was prioritised down to a programme likely to take one to two years (we are nearing the end of the first year now).

Overcoming technical challenges

• The most obvious challenge was that we did not have enough people. We had to go on a massive hiring spree for developers, product owners and scrum masters.
• React Native is very young and has been fraught with difficulties, particularly on Android. Navigation is still sketchy.
• Replatforming the entire front to back platform created a lot of moving parts and interdependencies.
• As far as we can tell, no one else is building a monorepo of cross platform components to be consumed on three platforms with proper testing and flow types. We needed a strong team of engineers to carry this through.
• Fonts are massively neglected by existing React Native apps. We had to diagnose and build cross platform fonts.
• Adverts are not cross platform friendly, but we now have cross platform webviews.
• Lambdas don’t naturally provide feature branches. We tried aliasing but it didn’t work with Terraform or anything else really.
• Putting React Native in a brownfield app is a lot harder than building a React Native app from scratch as others have done.
• Sharing data between native and React Native; Apollo sort of solves this.
• Designing and implementing designs cross platform is harder than per single platform.
• SSR with React Native Web with designs built on media queries is fun.
• The sheer number of tests started taking Travis down; we had to continually optimise.

The technical implementation

In the next article I take a look at the technical implementation, particularly the tools and ways of working.

We are open source

We open sourced the repo, a first for News UK. We auto-publish to public npm on merge to prove out the CI/CD culture with open source CI such as Travis and Coveralls. The backend has a threshold of 100% test coverage with the front-end currently running at ~96%.

newsuk/times-components

Special thanks

Special thanks to our Principal Engineers for (i) making this project work and (ii) contributing to and editing this article:

• Craig Bilner
• Andy Trevorah
• Joao Jeronimo