How we made our Serverless payments platform CMS driven
Diving into how we made our payments platform CMS driven while maintaining the ability to happily scale to 300 donations per second using Contentful and Webpack.
The Problem
During a cross directorate away day with our fundraising team, one of the frustrations that came out of our group was the process for creating and managing carts within our donation system.
For all of our campaigns, we create different carts to,
- Change imagery and money-buys (amounts and their descriptions) to reflect the campaign asks
- Track the activity and performance of the campaign
The old process was for a member of the fundraising team to submit a form to our team, we would then manually edit a JSON file which would then be put into a pull request, reviewed and would then go through our CI/CD pipeline. This was a massive pain for developers as usually required context switching, lots of back and forth between teams and always felt like the process could be simplified for all parties involved.
The Requirements
We had the following requirements from whatever solution we put in,
- Due to the scalability needs of our platform, the solution would need to be able to be statically compiled and not add extra dependencies.
- Editors would need to be able to preview their changes in near-real-time before being published into production.
- It would need to work in a local development environment.
- All changes would need to be automated and tested before hitting production.
How we achieved it
We use contentful to handle a load of our content requirements and are currently in the process of moving our main website over to it (with Gatbsy) from Drupal 8. It was a fairly obvious route to just set up a new space in contentful and plumb it in.
The first step was to define the content model and then migrate over the current cart ids from the JSON file. Contentful allows you to do this very quickly and is very intuitive for developers.
The next step was to alter our React Webpack configuration to pull the configuration from Contentful and build this into a replacement for the config.json file (We are huge fans of the way that Gatsby statically compiles, as is a perfect fit for our traffic profile).
We created a script to get the configuration from Contentful and then write the cart.json file ready for Webpack to build it. We then edited our applications start and build scripts to run the fetch script before a build or develop command.
The next step was to add some configuration to automatically re-fetch the carts and look for differences every 10 seconds.
We could now change a cart in Contentful and see it change automatically within the 10 seconds that we defined. The big question now was how could we do this in a preview environment that our content editors could easily use, seeing their changes immediately.
So, first thought was we could use EC2 running a node server. However, this goes against our philosophy of managing as few a servers as possible if we really don’t need to. We settled on a really simple (and free) solution in Heroku.
We created a Procfile in the base of our react application and then configured Heroku to automatically deploy off of our master branch from the Contentful preview API.
The end result was that we were able to edit our frontend in near-realtime prior to publishing content, giving editors immediate feedback.
After this, we updated our CI/CD pipeline to include the new environment variables and fixed all of our unit and integration tests.
We then needed to find a way to trigger our pipeline whenever content edits happened. We use Concourse and found this repository, that would check Contentful constantly and trigger the pipeline whenever there was an update. Unfortunately, it was looking at revisions and did not seem to be getting the latest updated content, so we forked it and made the changes that we needed in this repository to look at the latest updated data and use the timestamp to trigger to a build.
All we needed to do then was to add a webhook to Contentful to trigger the resource change and we were good to go.
We also needed to add another pipeline to backup our Contentful configuration and also to move content type configuration from a Contentful Staging Environment to its master environment. We did all this with a couple of Concourse tasks and the awesome Contentful CLI tool.
The Learnings
- Once upon a time, making something content managed was a big piece of work. It isn’t anymore.
- Content management as a service can easily be part of a Serverless application. It only took near enough a day and a half to implement this, much of which was pipeline integration.
- 5 minutes to production can be fine for CMS edits, as long as your stakeholders can see their changes live in a preview environment.
