How 500.000 people got a COVID-19 vaccine appointment
In late 2020 our company — innFactory — got a request to build a cloud software for COVID-19 testing in just 1 week. As a young software company, we thought “Sure — challenge accepted!” On the same day we asked all our team members if they were interested in a very special hackathon to build up the software for the COVID-19 test station based in Rosenheim. Most of the team instantly decided to join. 5 minutes before the deadline the software was online on Firebase hosting and all pods started to work fine. This was the beginning of our COVID-19 product story. Let me tell you how our software evolved and got to support the whole vaccination program, and which Google Cloud technologies we used to make it happen.
The evolution of our COVID-19 software
Everything started with a very basic software for COVID-19 that was used by a station in our hometown. Once it became clear that there would be a vaccine against COVID-19, we switched our focus to build up a vaccine management software. To gain knowledge about how a vaccination center works, we talked directly to the people who would use our tool. We had three counties that preordered our product right on the very first day, because the Bavarian State Government was not able to deliver a digital solution as fast as we could with Google Cloud Platform (GCP). Starting with the appointment management linked to available vaccines for the given day as well as the on-site process of the vaccination itself, finishing with scheduling a follow-up appointment and issuing a digital green vaccination certificate — we implemented everything you need to run a COVID-19 testing or vaccination center. Unfortunately, we did not take part in the public tender, so our software was replaced by a software called “BayIMCO” in Bavaria. Luckily some newspapers and TV programs featured our software before that and some well-known DAX companies became aware of our SaaS product when they screened the market for their company doctors. The first companies which ordered our product for their doctors were Munich RE and Linde followed by Rossman. Our product gained recognition after it had been proved that even under heavy load, the software we provided scaled perfectly because it was developed cloud native. Finally, Microsoft Germany bought it for all their German sites. Today we have carried out over 500,000 COVID-19 tests and vaccinations appointments and delivered many results to the patients.
The architecture behind our product
From the very first day, we hosted everything in the Google Cloud and used GitHub actions along with Terraform for our continuous deployment process. As we already had a lot of production-ready templates for React, Scala and Terraform, we were able to spin up the Cloud environment with a Kubernetes cluster very fast. Everything was cloud native right from the start and thanks to the automation with GitOps, our developers could concentrate on the software development .
All frontend websites are hosted on Firebase with a dedicated SSL certificate and every vaccination center has its own connected domain or subdomain like “company.impfzentrum.bayern”. The frontend chooses the correct backend endpoint for the given domain. Every company has its own Kubernetes namespace and pods are deployed multiple times by the GitHub action with different database connections to Postgres on Cloud SQL. We use one physical highly available Cloud SQL instance with lots of virtual databases. Before any appointment data is written, it gets AES256-encrypted inside of the pod with a unique secret per company. Even though looking for personal data is impossible, we developed this workaround to get GDPR compliant in Germany as we handle health data for our customers. Every company also got its own ingress to get separated from other customers. As we all know that COVID-19 is a contentious topic, we decided to secure every post request with reCAPTCHA and used Cloud Amour for advanced security. To send out our transactional emails we use a serverless function which calls the MailJet api together with a MJMLtemplate of the customer.
As we already mentioned, our software was also able to deliver vaccination tracking and test results. To do that, we invented a data format called “corsign” (CORona SIGNing). Corsign was an advanced JWT Token with some custom fields which were signed with all the relevant data. The Token was then encrypted with AES256. The encryption key was calculated out of the personal user data. Doing so we were able to send out the test result data without storing anything in our database. The users get their vaccine certificate or test results by decrypting the data from the mail with their personal data which no one else knows. The final pdf document is created with react pdf inside another serverless function. The decrypted corsign data is used to fill the pdf document with personal data and the associated QR code for the digital green certificate which is rendered from this data in memory. No database calls for the resulting documents are needed. For production, we use the Cloud Monitoring r toolset and an extension to the Play 2 Framework to trace every request with some useful logs. Uptime checks are performed for every custom domain and every ingress endpoint. While developing the software with 8 developers, we nearly released one or two versions per day to production. Now, when a new customer orders the software, we just have to add a new Terraform module which automatically deploys everything necessary.
From Cloud to Multicloud
When we won Microsoft Germany’s Partner of the Year Award for the product, we had to build up a multi-cloud deployment on Azure. At first we thought about forking the backend and frontend repository and build another terraform infrastructure. After initial considerations, we decided that we could extend the backend and frontend for the Azure services, and we just had to replace the infrastructure layer. In Azure we built up a second infrastructure based on AKS and Static Website and Azure Database. Finally, we replaced every single component with a similar PaaS Service on Azure. When the first appointments were available for booking, we had a peak of 12.000 requests per second for a few hundred available vaccines.
Conclusion
Without the useful PaaS services of Google Cloud, it wouldn’t have been possible for us to build and run a COVID-19 testing and vaccination management platform in such a short time. Within one week we were also able to build a multi-cloud product that’s been used by many clients and developed even further. And being able to help fight the pandemic, we managed to make the world a better place — one line of code at a time.
