The Times Digital Transformation Programme
Part 3: The Technical Implementation
This programme is a major project to re-engineer the technology behind the digital versions of The Times and The Sunday Times. It’s transforming a disparate tech stack that evolved from the print editions of the newspaper to a new scalable, performant and rationalised architecture that can support future subscriptions growth and technical innovations.
This article is part three of four:
Part 1: Our business rationale and processes
Part 2: Choosing the technical approach
Part 3: The technical implementation
In this article, I describe the technical implementation of the digital transformation project.
How do you manage a lot of developers working on a smaller codebase?
Monorepo and Lerna
React Native and React Native Web follow the component approach, which is the status quo nowadays. Other projects like Babel, Angular, Ember, Meteor, Jest and many others develop all of their packages within a single repository, often called a ‘monorepo’. We adopted monorepo and added Lerna, which is mature and just kind of works out of the box. We have faced many challenges with transpiled artifacts and running tests across the repo. However, it handles Git tagging, npm publishing and allows CI/CD on master to merge.
Development tools
We standardised most of our development tools, so that we could stop our shiny new codebase fragmenting out with individual developer preferences like our old one did. Even more so, we adopted standard configurations for these tools to avoid ‘bike shedding’ (coined from some research that observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed, while neglecting the design of the power plant itself).
If our developers strongly feel that a tool is wrong out of the box, we encourage them to raise a pull request against the source project and have it flush down into our code that way. This was initially controversial but allowed us to get going faster.
Eslint
We enforce code quality with Eslint, using the Airbnb configurations straight out of the box.
Prettier
For standardised code prettiness and styling.
Travis/Bitrise
When you’re running tests cross platform you have three times as many tests. This is really hard with many pull requests, especially as our Travis builds had crept up to 45 minutes. The free tier got us off the ground, but we chose to pay for it (it’s good to give back). Our tests now run really quickly on beefy machines, bringing our PR feedback down to a matter of minutes or even seconds.
Storybook
Storybook is the cornerstone of everything for product management, the business, UX/UI, engineers, the data team; in fact any dog in the fight.
We extracted storybook from times-components to be storybook agnostic, opting instead for “showcase” files. A platform can then take these simple JSX files and turn them into a story of their choosing. In the repo itself they’re converted into typical React storybook stories, but for the expo app we can render them differently. This was largely due to man incompatibilities with storybook and the native devices/expo, however, is a nice abstraction regardless.
Coveralls
100% code coverage (and yes, we do have very nearly 100% code coverage) does not mean we have no bugs, but it does give us a healthy benchmark to hit. It encourages testing rigour and quality code reviews. It ensures that our code is executable and valid, and that the code is consistent across platforms.
TypeScript
We use TypeScript for static type-checking along with the latest ECMAScript features. We used Flow for most of the project, but switched quite recently for a whole bunch of reasons that would take another complete article to describe (maybe we’ll write that up soon).
Apollo
We went with Apollo because it gave us more flexibility than Relay but also a large community and excellent tooling around API enhancements and testing. Schema first development is also a good philosophy with separation between schema and resolvers; a nice touch.
Jest
Jest was chosen for the test framework because, again, it paired with the React ecosystem and developers could move across the stack with the same tools and benefits of snapshot testing. It was much maligned when it first came out but performance has gone through the roof in recent months/years and is now a decent testing tool.
Publish on merge (to the public npm registry)
All of our code is open source. It is very important to integrate components back into your platforms and unblock developers who are working on many components at the same time.
XNative repo
We try to keep as much data/logic out of our packages as possible, for example leaving the host platform to provide the analytics provider and GraphQL providers themselves. It turns out that not only can you share code across platforms through components but you can also share the “glue code” between the native code bases and the “dumb” components that reside in the component library.
Published artifacts
One of the more painful aspects we’re still working around is having a monorepo with transpiled code (TypeScript, Flow, es modules) that also works with Storybook, Jest and on the simulator/emulator. This is largely due to not being allowed more than one entry point in a package.json (ignoring webpack) and/or a clear separation at a package level for native/web code.
The entry point for a package is the dist folder which needs to be populated for the symlinks to pick up the updated code. We had to move from Metro to Haul to get this working in a relatively nice way. Unfortunately we still need to run Jest on the source code which also transpiles everything itself more than once across packages. On top of this we also want consumers to be fairly agnostic to consuming the packages therefore need to craft a react-native-web specific bundle for the web platform to consume, without needing to transpile/bundle every component with webpack and the necessary plugins etc.
How do you test across platforms?
Testing truly cross platform code is without doubt problematic. There’s DOM vs no DOM, Android/iOS quirks and more. You could just write unit tests and hope the rest just works or use shallow render and snapshot testing. We decided to do it properly. There are a few main testing approaches to consider.
Option 1: You don’t
You could not attempt to do cross platform testing and rely on end to end testing on each platform using Espresso, webdriver etc. Much like using three code bases to generate the output, this multiplies the effort needed to test everything yet divides the testing quality.
Option 2: You do it badly
You could follow the React Native documents and do some Jest snapshots of shallow rendered components. This does not give you true confidence that your output is correct or consistent.
Option 3: You try your best
We wrote jest-configurator to correctly test Android, iOS and Web with snapshots relevant to the platform. Jest is still iOS first, which we had to work around. Upgrading Jest with our setup also proved incredibly difficult.
Option 4: You create a whole team and throw everything at it
That’s what we decided to do. If we were going to go cross platform that had to apply to testing too. We set up a dedicated tools team, which has created Fructose (to run end to end tests in Simulator/Emulator), Dextrose (to perform visual snapshot regressions) and added Expo QR codes to pull requests to demo stories on real devices per feature branch. The React Native ecosystem is very young with respect to testing and we’re scratching the surface to produce robust cross-platform components. Everything we do (including these testing tools) is open source.
Lessons learned and key takeaways
In the last part of this series, I will look at the lessons learned so far and offer some key takeaways.
We are open source
We open sourced the repo, a first for News UK. We auto-publish to public npm on merge to prove out the CI/CD culture with open source CI such as Travis and Coveralls. The backend has a threshold of 100% test coverage with the front-end currently running at ~96%.
Special thanks
Special thanks to our Principal Engineers for (i) making this project work and (ii) contributing to and editing this article:
- Craig Bilner
- Andy Trevorah
- Joao Jeronimo
