Our first Redis Nose Day

We changed a few of the services that support our apps for Red Nose Day giving pages before the big day this year.

Image for post
Image for post

One of the more interesting processes was finding a good solution for all our caches and session data. In many cases, we found a Redis service worked well with our Cloud Foundry apps, especially when data has to be shared.

The move to Redis wasn’t entirely seamless, but it worked well for us on the night, and has taught us lots. This is a quick summary of what we’ve found so far.

A service within Cloud Foundry has great automatic benefits

We got some easy wins by using the Redis service that Pivotal kindly gave us use of.

  • There’s no service for us to deploy. Provisioning user-provided services for your Cloud Foundry apps is inevitably more work.

Shared services have caveats too

Similar behaviour from every instance has a flip side. Every Redis deployed in the standard way currently has the same default virtual machine size and set of configuration options — although these are both things which will become more flexible soon with improvements to the platform.

We know our traffic has huge spikes coming up to a campaign, so we take load testing pretty seriously. It was when we threw a few 10,000s of simulated donations at our new apps that we realised Redis was configured to keep everything forever, and didn’t handle running out of space as a result very well.

The bad part of this is we couldn’t easily change the service configuration for our Redis instances. We talked about a custom deployment of the service, but this was hard to do in time and would mean more unpredictable deviation from the standard service. We also couldn’t expect instances to always be long-lived, as they can get swapped around during platform maintenance. Again, this is something which will become more predictable soon with further PWS improvements.

Even before these platform changes, a great thing about Cloud Foundry is that adding apps is ‘cheap’, and this can make working around this kind of thing in the short term much simpler. We were easily able to put together a single-purpose utility app that updates our configuration whenever it doesn’t match the cache eviction policy we need, i.e. whenever the service instance is new.

Every 2 hours we just spin up this app and bind it to every Redis we care about. This way we know we’ll never go more than 2 hours without our cache evicting as we’d like. We wrote the bulk of this app in less than a day. You can see its source here to get an idea what’s involved in bootstrapping something like this — the key logic is only a few lines.

Finally, essential maintenance timing wasn’t fully under our control. Pivotal worked with us to avoid major disruption around Red Nose Day, but some disturbance from maintenance is hard to avoid with this kind of shared infrastructure. In addition to Pivotal’s other options like single-tenant Cloud Foundry on PWS-E, there are changes planned on PWS Redis that will also reduce unexpected downtime. So we’re confident we can find a reliable option within the Cloud Foundry ecosystem for Sport Relief 2018, where giving pages are likely to be used even more.

Coping without it

Maintenance on the Redis service did get us thinking more carefully about what we actually need from our Redis instances, and how we should react if they’re down. This got us a few technical wins and more resilience than we had even with our old infrastructure and memcache.

  • We now use more local caching (e.g. apcu) where it does the job and is faster than a shared cache.

Moving forward

Using Redis for this Red Nose Day has been a great step towards better understanding the relative benefits of Cloud Foundry native services in general vs. external ones. It’s important to think about how much control you want, and for future apps we’ll be thinking about our service requirements in detail, earlier.

We’re also excited that Pivotal Cloud Foundry Redis will soon be getting major improvements in configuration flexibility and how updates are deployed. This will mean increased reliability, and also make some of our workarounds redundant, allowing us to keep our apps simpler.

Having successfully used both types of service this time around, it looks like Cloud Foundry can give us the flexibility to keep making the right call for each one as we progress.

Comic Relief Technology

This is where we showcase the ways that we are use…

Comic Relief Engineering

Written by

This is the general user for comic relief engineering

Comic Relief Technology

This is where we showcase the ways that we are use technology to support the core aims of our organisation and how we do it.

Comic Relief Engineering

Written by

This is the general user for comic relief engineering

Comic Relief Technology

This is where we showcase the ways that we are use technology to support the core aims of our organisation and how we do it.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store