How we built a vaccine appointment solution in 4 weeks
In November 2020, seeing the critical role that GP Practices were going to have in delivering COVID-19 vaccinations, we recognised that by being used in 98%+ of GP Practices, we were in a unique position to help, and we set out to build a vaccination booking solution in 4 weeks.
That solution has now been used by millions of patients across the UK to book their vaccinations.
This was no easy feat and the success of the effort was built on five pillars, which I’ll summarise in this post. To dive deeper, into the pillars, accuRx and our background, and the future of the vaccine booking solution, check out Laurence’s tech nation talk at the bottom of the post.
🏛️ Plan for scale
We knew it was feasible for the solution to be adopted nationally and with the pace of vaccine roll out, we knew we needed to be able to scale to that level of load quickly with clinicians relying on us to have high availability.
We identified our database would be a significant bottleneck and so derisked this up front. Before we started building the application, we planned the likely database structures we would need and thought through likely user flows to check our structures would allow for efficient queries.
By using modern cloud infrastructure and database as a service, we had the ability to elastically scale the database to demand. We separated out the new appointment structures from our existing databases into a dedicated database, so we could pinpoint appointment related performance issues and scale independently.
We also made use of read only replicas, allowing us to more efficiently support workloads that don’t need to mutate the data.
🏛️ Build and release
We use git and GitHub to collaborate on code. We build and release using pipelines defined in YAML in Azure DevOps.
We’re passionate about the benefits of code review, not only for achieving high quality in the products we create but also for continuous improvement and learning. Check out these hints, tips and principles from our VP Engineering Ben. We see providing thorough, educational and friendly code reviews as a core part of our job as engineers.
Trunk based development helped us quickly respond to the changing requirements of the vaccination programme, making sure we were never slowing down clinicians that needed us to take into account changes to how the vaccine is administered and booked, and helped us reduce the risk in any one change through releasing multiple small changes many times a day. In trunk based development you collaborate on a single integration branch (trunk) using short lived branches. We evolved to this model from a more conventional main/develop model with longer lived integration branches and have really benefited from it as our team has scaled. When a commit lands in trunk, builds are kicked off, automatically released to our staging environment and can then straight away be promoted to production.
🏛️ Monitoring
We monitor our systems using Application Insights, Sentry, UptimeRobot and Slack for notifications.
For vaccine booking, we made the decision to embrace a low signal to noise ratio. We were happy to ignore many false positive alerts in order to make sure we didn’t miss something important. To achieve this, we used the route tagging feature in Sentry, allowing us to route vaccine booking related monitoring into dedicated Slack channels. The team working on vaccine booking got access to the more verbose/noisy logs and other teams working on the same components had a level of verbosity appropriate to their work.
We’ve found applying a swiss cheese model to monitoring works well for us, by monitoring the system at many different scales, we reduce the chance we’ll miss something important. For us that looks like monitoring behaviour of individual lines of code through log and exception tracking, in particular we take care to log about assumptions or invariants not holding, we monitor performance at a request level, looking out for higher than expected failure rates or response time and monitoring services as a whole, using health checks to quickly identify when a service is unhealthy.
🏛️ Learn and iterate
User feedback is everything.
We were lucky to have a six thousand strong Facebook group before we started vaccine booking. Together with accuBook specific WhatsApp groups, these platforms help us have a direct relationship with our users, and users with each other. We find the groups helpful for quickly identifying issues and opportunities in the product. Users find them useful for sharing tips, keeping up to date with the latest changes and feeding into product development.
Every product team at accuRx has a user researcher and we have a strong user research focus. Alongside gathering feedback from these groups, our user researchers use user interviews and usability testing to make sure our products are high quality and work well for users.
🏛️ Team
The final and the most critical pillar for building a vaccination booking solution in 4 weeks was the team. Whether directly driving forward accuBook or picking up other things to give the accuBook team the space to do so, the whole company came together.
Something we put a particular emphasis on at accuRx is the leadership of a product team and the relationship between Product Manager and Tech Lead. We think of the PM and TL as co-founders of a micro-company and really value the strength of their relationship. Through building great trust and psychological safety between each other, they provide a platform for their team to achieve great results. Check out Laurence’s Good Tech Lead/Bad Tech Lead blog for more of our thinking on this.
Providing a great support experience to our users is critical too. Our support team fielded over 22,000 conversations in the last 90 days, which was a huge increase reflecting the growing user base and complexity of the tasks we were enabling users to perform. We grow our support team from 5 to 12, a huge task in itself to achieve in a matter of weeks and this enabled us to scale to supporting the vaccine product as it rolled out across the UK. Go support team!
▶️ Tech Nation
Check out Laurence’s recent Tech Nation talk to dive deeper!
For an even deeper dive, into all things vaccine and accuRx’s journey over the last five years, tune into this Secret Leaders episode.
