Shipping Progressive Web Apps everywhere

For the past five years BBC Children’s & Education have been building interactive games and experiences with JavaScript and HTML5. This enables us to ship games to any platform with a modern web runtime. We work with agencies to commission new games and experiences for the many brands that we have.

We have around 300+ games, some of which are distributed via our five mobile apps on iOS and Android and others are embedded across www.bbc.co.uk.

Our 5 Apps within Children’s & Education

We optimise ourselves around being cross-platform. Designing our organisational structure to make bringing new content into our portfolio as painless as possible, and good value for money.

We maintain multiple in-house game engines. These enable agencies to build various different game mechanics effortlessly with varying fidelities, whilst building abstractions and delivery pipelines so the games can be delivered to multiple platforms rapidly.

Delivering these HTML5 games to the web is fairly straightforward. We fetch the content from S3 and pop it into the DOM, each game shares a common API interface that is configured at runtime, allowing the host platform to inject its own implementations of libraries and config. This allows our game developers to focus on building the game and not the underlying mechanics of how our web and app platforms work.

Games on the web can start utilising modern new browser features like Service Workers, Web Workers, and Progressive Web Apps to deliver more and more optimised experiences relatively easily.

However, delivering web games into app stores isn’t as straightforward as “popping it into an app shell”, though.

In a series of blog posts, the Children’s and Education team are going to outline how we rapidly develop web content for app stores and beyond using modern web technologies on a platform dubbed the “Universal App Platform”.

What came before?

Around five years ago, when the business was first facing the challenge of shipping HTML5 content onto iOS and Android stores, many of the options that existed were fairly immature. Tooling like PhoneGap and Cordova were only just materialising, and client-side libraries like React were not as popular at the time. Web standards were growing but there was no concept of a Progressive Web App yet and browser capabilities like Service Workers did not exist.

Back then, putting web content into apps was a hostile environment. Shipping onto app stores was typically considered a problem that could only be solved well by experienced native app developers.

This platform complexity, coupled with immature frameworks and the business requirement to ship web apps to app stores, led us to spinning up an internal engineering team to create a proprietary app wrapping framework. This framework would be capable of bundling HTML5 web content and generating iOS and Android binaries from that content.

We called this framework Pick’n’Mix.

Pick’n’Mix provided a heap of functionality to its embedded web content, including the ability to download and store remote content from a CDN to disk, UI overlays for app settings and download management, analytics events, push notification support, and many other features. This functionality was built from scratch and solely maintained by our internal native engineering team.

Fundamentally, Pick’n’Mix was a WebView/WKWebView with a JS “bridge” capable of passing messages between content running within the WebView to trigger certain actions on the web/native side. This bridge was a “standard” we maintain internally called the Games Messaging Interface (or GMI). This GMI ensures that games are interoperable between many platforms and abstracts away the nuances from game developers of having to worry about the runtime.

You might expect to see methods such as getConnectivity, setConnectivityCallback, download, onSettings, getSettings, setMotionSettings on the GMI.

A high-level look at the Architecture of Pick’n’Mix:

High res image: https://github.com/bbc/uap-architecture/blob/main/Pick%20n%20Mix%20Container%20Diagram.png

From a product perspective, our apps can be split into two distinctive parts:

1. The Menu system where users can browse the catalogue of games and experiences they are able to download and interact with
2. The Content items themselves

Content items are typically cross-platform and built in the browser and deployed to both the BBC website and Pick’n’Mix.

Menu systems, however, were only needed in the app deployable to enable discoverability of content. This meant that menus were built exclusively for Pick’n’Mix and as such we historically required the engineers building menu systems to develop, test, and deliver the menus as fully-fledged ‘Pick’n’Mix’ apps. A very closely coupled integration. This is an important nuance; we were asking engineers to deliver us HTML5 content to run exclusively on a proprietary platform.

To support this, we had to provide means for engineers to test and develop their menus within Pick’n’Mix itself, adding a “development harness” to the pile of proprietary things we’re having to build and maintain ourselves. This nuance cuts even deeper - though, by requiring HTML5 apps to be delivered to a proprietary set of standards, we were moving further and further away from new and emerging web standards. This meant we were unable to take advantage of fantastic resources like MDN, and fundamentally in a position where 80% of the underlying runtime had to be supported by us.

The web is an amazing thing; it can turbocharge your ability to get things done and ship content anywhere, but it’s flexible too, making it easy to box yourself into a corner of unsustainable bespoke implementations, before you know it your once esoteric requirement is a web standard.

Pick’n’Mix was monolithic, it was a semantically versioned framework that every app had to depend on and bundle itself into. Pick’n’Mix did not stipulate how apps should build themselves, none of our apps were running on the same version of Pick’n’Mix and each time an app needed to take a new version upgrades were laborious and expensive from a development and testing perspective.

Due to Pick’n’Mix’s un-opinionated stance on how things should be built, each app had to maintain its build pipelines and menus/games had to orchestrate their own interaction with the GMI, ultimately there was no sharing or composability between apps. On the surface, abstracting native functionality and business logic behind the GMI seemed sensible and good value for money, a single, well-defined API for fetching remote data, right?! Digging into this further though, it was clear that we’d simply offset the cost away from agencies having to build the native functionality (which is good in itself) to agencies having to integrate with an API that was just as complex and potentially more un-intuitive than if we did nothing at all.

This also led to us having significant amounts of duplication across our portfolio of apps. UX patterns implemented time and time again written imperatively with WebGL/Canvas, several integrations and duplicated business logic around how downloads should be handled under certain network conditions, inconsistent bundling tools and inconsistent config approaches across apps.

We used Wardley Mapping to understand where each of the technical components required to ship our apps on Pick’n’Mix sits in terms of the maturity (and thus cost) against the value they deliver to our audience.

A Wardley Map illustrating cost/maturity of all the aspects of our Pick’n’Mix framework

From this map, and our experience and knowledge of building hybrid web apps over the years, it was clear the majority of our time and money was being spent building our in-house app wrapping technology, and not on the app experiences themselves.

In 2020, the problem of wrapping HTML5 content for app stores is a well-solved one. Several open-source tools exist that can bundle HTML5 content into a native app, including PhoneGap, Cordova, Capacitor.js, Framework7, Ionic plus others. Progressive Web Apps (PWAs) and Trusted Web Apps are maturing as standardised ways of building portable, app like web experiences. As well as this, platforms are beginning to support PWAs/HTML5 apps as first-class deployment types, from Android via the Trusted Web Activity through to Amazon Fire TV sticks. Service Workers, Web Workers, and many other mature/newly emerging web standards are coming to fruition that makes building native-like apps on the web a reality.

It is clear that continuing to maintain our hybrid app wrapping framework in 2020 was unsustainable, so we decided to re-orientate ourselves around a series of principles to better inform a more sustainable technical approach to building experiences that can be shipped onto an app store.

Principles

• Embrace the technology — fundamentally our apps are websites, we should embrace that
• A single codebase for app and web — we’d got so caught up in where our code was deployed too, we were not optimising for the right things
• If it exists, we don’t reinvent it — The JavaScript ecosystem is extremely vibrant around the problem area we’re in, we should stand on the shoulders of giants
• It runs anywhere — we want to move away from maintaining proprietary tools and enable web developers to use the tools they are most familiar with
• There is no framework — the most important principle, we want to stand on the shoulders of giants and define a set of opinions that make building our experiences familiar to any developer who has worked on a modern web stack recently

Building a Universal App Platform

Once we set out our principles, we ran some experiments to answer a series of questions. How hard can we push the web today? How close to an app store ready app can we get with little to no native engineering? Could we move all natively built user interface components out to web components? Could we avoid building our own hybrid app framework full stop?

We learnt that as of today, shipping Progressive Web Apps to many platforms is becoming a reality. Whether it is Androids support for PWAs in Google Play via a Trusted Web Activity, or support for modern web standards across a range of smart devices from Smart TVs to Fire TV Sticks — it’s clear this is a rapidly changing and improving field.

Our unique and immersive user interfaces also means that we don’t need to match the native system look and feel. Having our own design system that kids are already familiar with, across all of our experiences, places us in a unique position where a single code implementation makes sense.

Left: Our new, cross platform settings menu for Web, iOS and Android. Right: Our old, natively build, platform specific settings menu.

Progressive Web Apps are yet to make an appearance across any part of the Apple ecosystem, but we know Safari has been adopting more and more modern web standards which we can use to our advantage as we develop this technical strategy.

We know, however, that there will be gaps in platform support. These gaps require us to have a flexible architecture that puts modern web standards at the heart of our engineering but retains the ability to patch in polyfills/native implementations where required.

From our research and development spikes, we settled on a 3-pronged technical strategy:

1. Modern web standards like Progressive Web Apps now give us the ability to build app quality experiences with just web technology — we should embrace the PWA as our core development environment and optimise for this as a priority
2. Emerging technology and open source tooling give us the capability of shipping compliant Progressive Web Apps to all app stores. We identified Ionic’s Capacitor.js to be the best choice to wrap compliant PWAs for app store distribution
3. A declarative, component-driven architecture will allow us to drive more re-use, increase consistency, and reduce platform-specific implementations across App + Web. We chose React.js to enable us to build up this architecture

PWAs + Capacitor.js + React = Universal App Platform.

This is a significant product and technical change for us. We’re shifting our focus away from “we build native apps” to “we build immersive experiences for any platform”. This new way of thinking enables us to build consistency into the platform as a whole whilst also reducing the cost of building our products, reducing cognitive load across all disciplines and making room for us to focus on new product propositions.

The Universal App Platform Architecture. The diagram illustrates the relationship between the Progressive Web App, the App shell for App Stores and how each are built and deployed respectively. High res image: https://github.com/bbc/uap-architecture/blob/main/container-diagram.png

Outcomes

• Re-built our entire proprietary app platform in less than a year whilst rebuilding all our menu experiences to have highly reusable, declarative front ends at the same time

Cross platform Universal App Platform components running in the web

• A single codebase composed of multiple apps and a cohesive set of components that can be reused, extended, and delivered across multiple platforms
• A single community of engineers collaborating, improving and developing a shared platform
• Standardised ways of working, linting rules, and testing approaches across multiple teams
• A shift away from imperative WebGL/Canvas programming to a DOM based component model through React.js for our Menu systems
• No further requirement to maintain proprietary development harnesses, testing frameworks, or documentation
• Our ability to power all of our five apps, plus our entire interactive embedding estate of 350+ games from a single codebase and a single set of cohesive components
• The ability to focus on new, product enhancing capabilities across the entire portfolio of Children’s and Education products

Re-mapping where we are today illustrates that through shifting heavily to off the shelf, open source and commodity driven ways of building software, we’ve been able to re-focus our time and money on building immersive experiences for our audiences. We’ve achieved his whilst eliminating significant amounts of duplication as well as having a modern, standardised technical stack and reaping all the benefits that come with this.

A Wardley Map illustrating cost/maturity of all the aspects of our Universal App Platform

The left screenshot below is our existing, bespoke, imperative Canvas/WebGL based Pick’n’Mix wrapped Playtime Island app, on the right is our new skinnable, component driven Progressive Web App wrapped in Capacitor.js producing the Playtime Island app.

Left: Playtime Island running within Pick’n’Mix build imperatively with Canvas/WebGL. Right: Playtime Island running as a component driven, Progressive Web App in Capacitor.js. Virtually no difference in look and feel.

Here is just a small flavour of how declarative, composable and intuitive our component driven architecture now is. This component displays an information banner across the app, transitioning onto the screen and animating images that sit either side of the banner text.

An example UAP React.js JSX component

What’s coming next?

As the Universal App Platform matures, we will be looking at new opportunities to deliver more compelling, personalised, and rich experiences to our audience, as well as understanding what future opportunities our Universal App Platform can support for products looking to be cross-platform.

In the short term, we will be launching all four CBeebies apps, migrating our game embedding technology and developing rich, personalised experiences to benefit both Web and Apps on the Universal App Platform. Ultimately decommissioning both Pick’n’Mix and our game embedding platform CAGE early 2021.

In the longer term, we’ll be using the opportunities this transformation has provided to develop our product propositions further, looking towards opportunities in personalisation, AR/VR, new platforms and more.

This has been a significant, strategic, technical and product level change for us. A tremendous number of people have been involved in making this change happen. Change at this scale is never easy, but working together is the key to success. Expect to see more details on this blog from teams working on the Universal App Platform over the coming weeks and months.

It really does run anywhere! Go Explore as a PWA running on a Tesla In Car Entertainment System.