No Monkey Business Static Progressive Web Apps (part 1)
or; How and why Interactive Investor uses decoupled Drupal, Gatsby.js, ReactJS and AWS to deliver rich content without making Google cry
Back in June 2017, on completion of our acquisition of the European business of TD Direct Investing (branded as TD Waterhouse), we, the new Interactive Investor (ii), were presented with a technology challenge:
How do we bring two technology stacks together, in a new modern stack, to benefit all, from our customers through to our developers?
This series of posts will cover the technology and challenges we faced relaunching our public site (free research, news and analysis, discussion and product marketing site) — focusing on “the front-end” User Interface (UI), UX and Content Management System (CMS) layers.
In this first post, I, Dominic Fallows — Apps Technical Lead (web, mobile and content) at ii, will go further into the challenges we faced and the solution that enabled us to deliver our modern UI architecture.
Part 1 — Table of Contents
Our challenge… is our opportunity!
How do we bring two technology stacks together, in a new modern stack, to benefit all, from developers through to our customers?
Our Product + Engineering teams believe in taking a user-centred and product-led approach. Our opportunity was to:
Modernise our solution, learning from the best and worst parts of our past technology stacks, whilst improving our experiences, using the latest technologies.
What makes a great UI?
There are many things that make up a great User Interface (UI) from a user, product and technology point of view. I have shared some of the qualities we identified and how we interpreted these to help form our UI architecture solution. From a technology perspective, a great UI should be:
- Intuitive — we need a platform and technology that would enable us to build a high-quality UI allowing us to deliver on strong UX strategy and design
- Attractive, clear and consistent — we need to be in full control of the UI templates and content production workflow allowing us to deliver on strong UI design and content strategy
- Responsive and adaptive — deliver the same great experience across all devices
- Fast and efficient — we need to create a modern web app and architecture that delivers great performance for users on their browsers and devices and also for search engines, like Google
What makes a great solution?
In this post, I’m going to focus on three aspects of what makes a great solution:
Of course, that includes UX and a great UI, but also the experiences of our teams. A whole selection of teams works on our solution, to name a few: Web App developers, Mobile App developers, API + Services developers, DevOps engineers, Analysts and Testers, Product teams, Marketing and Editorial content producers, etc.
Happy and engaged teams make great products.
We needed a stack that would be attractive to work with for our teams, current and future. There was a huge appetite from our teams to develop their skills on a modern stack made up of the latest technologies.
As a FinTech company, we operate in responsible ways to protect our customers. A great solution to us includes security, from Information Security (InfoSec) to DevSecOps — the philosophy of integrating security practices throughout the DevOps process.
A great solution enables people to find our product and services, from our free research, news and analysis, discussion platform through to our product sign-up journeys.
Shaping our solution
Following on from our discovery and analysis of all the aspects above (and many more requirements from across the business) we started to shape a solution:
- AWS Cloud Platform as our cloud computing provider
- Micro-service architecture to deliver our scalable business and data APIs
- Decoupled CMS to provide an enterprise content production experience
- Familiar UI language and frameworks considering developers across API + Services (Java) and App (web, mobile) teams
- An interactive, performant and scalable web app that is Search Engine Optimised (SEO)
UI language and frameworks for an interactive, performant and scalable web app that is Search Engine Optimised (SEO)
Working with web browsers is hard. There are many browsers actively used by our customers — like Internet Explorer, Chrome, Safari, Firefox — which all have unique ways of working. Added to this is the wide variety of devices that is now common-place — low-powered and high-powered smartphones and tablets, laptops and desktops — which all add on more complexity.
We needed a UI language and framework that would make working with web browsers efficient, allowing us to focus on our objectives (like UX/UI, Security and Discoverability — like SEO).
We needed to enable developers across teams to use their skills and work together on a large codebase. Our choice of UI language and framework should also enable our Java developers to up-skill in modern UI development.
After much research, and considering all of the popular web app options back in 2017, we selected the following for our UI language and framework:
- UI library: React
React enables developers to work with a virtual browser, which acts as an agent between the developer and the real browser, standardising the unique way each browser works with web apps. As the React team themselves say “React makes it painless to create interactive UIs”.
- UI framework: GatsbyJS (v0-v1)
We chose Drupal as our headless CMS, which we will cover in the next post in this series.
We were very happy we had found the perfect solution to deliver our objectives, so we got head-down and kicked off the development of our new public site in October 2017.
By the end of 2017, we were ready to launch a beta version of our public site. This beta site was small but helped us start to get customer opinion and to test our new web app with Google.
We started to release more pages to our beta public site like our CMS powered marketing pages and our Editorial news and analysis Articles. By mid-2018, we had several hundred marketing pages and upwards of 8000 news and analysis articles on our web app.
We already knew that we wanted to increase the SEO value of these pages, full of useful information about our products and our analysis and commentary. We used the power of Gatsby v0-v1 to create SEO optimised static versions of these pages.
All went well, and we were happy with our solution choices.
However, by Q3/Q4 2018 our organic traffic wasn’t recovering as fast as we thought it should and we started to experience problems with getting our pages indexed in Google. Some pages were taking months to appear, some didn’t appear at all. That was bad enough, but our scariest finding was that some pages started to get deindexed from Google’s search results.
What was going on?
After seeing positive signs that Google knew about all of our pages, even if they were taking time to include them in search results, we now saw our pages disappearing from the indexing process altogether.
To put it another way, Dynamic Rendering means delivering a different rendering and experience of the web app to users and Google. This concept was not new, it was something we had considered in our early planning days.
However, we felt there must be a better way to recover our SEO and indexing, whilst not reverting back to more traditional methods of web app development.
We were not comfortable with the idea of switching our output based on who was requesting it, that is fraught with complications (we’d need to detect, Google, Bing, Twitter, Facebook, LinkedIn etc — who are all search engines in their own right) with more moving parts to maintain.
At the time Gatsby (v0-v1) would not have been able to process the ~25,000 pages that we had selected for static versions, it simply required too much expensive cloud resource (GBs of RAM and lots of processor power) and took far too long to deploy (over 20 mins for just our CMS content alone, not including 24,000 research pages). Our content workflow requires us to publish, and therefore build and deploy, our web app many times a day.
Gatsby (v0-v1) was a powerful single-page app (SPA) and static-site generator, but we were asking too much of it, to be able to handle our hybrid static/dynamic pages as well.
Gatsby v2 to the rescue!
Static Progressive Web App and Gatsby v2
The JS ecosystem and Gatsby v2 started to develop the idea of Static Progressive Web Apps (SPWA).
A Static Progressive Web App (SPWA) is an evolution and hybrid of a:
- Static Site
A site made up of static web pages (sometimes called a flat site) that is delivered to the user exactly as stored.
- Single-Page Application (SPA)
- Progressive Web Application (PWA)
A PWA enhances SPAs with offline functionality, push notifications, and device hardware access traditionally available only to native applications. PWAs combine the power and flexibility of the web with the experience of a native application.
An SPWA meant that we could deliver the same great experience that is optimised for robots and humans alike.
Gatsby v2 was now a powerful SPWA generator. They had made huge improvements to its internal codebase. This gave us confidence that combining the concept of an SPWA and Gatsby v2 meant we could start to process large amounts of pages (~25,000) that used acceptable amounts of cloud resource and deploy times.
Summary and future
By November 2018 we had delivered our rebuilt site in Gatsby v2. We now statically generate ~25,000 web pages in approximately 5–6 mins. We are seeing fantastic SEO improvements, pages now get indexed in Google search results in hours and days (not weeks and months), update in minutes and hours and our organic traffic is ever increasing.
- We continue to learn new ways of optimising Gatsby V2 and our React application
- We are working on further ways to increase the efficiency of our SPWA
- We will keep on improving the Product, UX and UI for our customers
- We will continue to grow our modern stack
This is the first post in our series: No Monkey Business Static Progressive Web Apps or; How and why Interactive Investor uses decoupled Drupal, Gatsby.js, ReactJS and AWS to deliver rich content without making Google cry
In the next post, we will dig further into the Headless CMS (Drupal) architecture.
You can find our public site (free research, news and analysis, discussion and product marketing site) at https://www.ii.co.uk
Update 03/04/2019: with links to Elliot Ward post No Monkey Business Static Progressive Web Apps (part 2)