Data security at Agoda: How we automate encryption key rotation
Introduction
Every data-driven organization often handles vast amounts of diverse data. However, several challenges come with it. One of the most common challenges is handling sensitive and confidential data.
Sensitive data, like Personal Identified Information (PII), must be protected per regulations such as GDPR. Unauthorized individuals should not be exposed to confidential data such as company internal statistics. So, what can we do to make us more confident in the security of our data? The answer is data encryption.
What is data encryption?
Data encryption is a way to prevent the leakage of sensitive data to cyber criminals by making the data unreadable for humans. If someone wants to decrypt the data, they need to have the encryption key to do it. This means the data we need to send on the internet can be read only by someone with the same encryption key.
The encryption key is like a password to access the data. If the encryption keys are stolen or cracked, our encrypted data will be vulnerable to a data breach.
Moreover, if our encryption key is deleted somehow, the encrypted data will be forever useless. This means managing the encryption keys should be our primary concern since we must protect them as much as we want to protect the data.
In this article, we will get into one of the best security practices to make our keys safe and secure the keys rotation.
Why do we need to rotate the encryption key, and why should we automate it?
Data privacy regulations such as Payment Card Industry Data Security Standard (PCI DSS), Sarbanes-Oxley Act (SOX compliance), or other privacy regulations require periodically rotating the data encryption key. So, if we have data that falls under these regulations, we need to rotate the key to maintain compliance.
Moreover, the National Institute of Standards and Technology (NIST) also recommends implementing key rotation. But that shouldn’t be the only reason for us to do so. We can do the key rotation even if our data doesn’t fall under any regulations. So, it is better to understand why we need the key rotation. Here are a few reasons;
- Rotating keys makes it more difficult for malicious individuals to crack them. Encryption can be cracked with enough time and resources, depending on the key bit size and algorithm used. By rotating the key, a hacker’s efforts will be rendered useless, and they will have to restart their attempts to crack it.
- It minimizes the impact from a compromised key, cryptanalysis, and brute force attacks.
- It maintains compliance with the regulation.
Suppose we have several hundreds of secrets that must be rotated periodically. Each key has a different expiration period, key sizes, encryption algorithms, storage locations, etc. If we need to update all the keys manually, the consequences might be that;
- It takes more effort proportionally to the number of keys.
- There might be potential human error.
- We might encounter more vulnerability since a new key might be stored in the local machine of someone who updates it.
To avoid this, we initiated the key rotation service project to automate this task.
How do we rotate the encryption key?
My team was recently tasked with working on a project involving data in transit, which involves transferring data across the network. For example, we needed to send a business data file to a third-party company outside our private network. To ensure better security, we implemented key rotation and initiated a key rotation service to automate this task.
The key rotation service is a config-based cron scheduler. It checks the expiration date of the keys every time the process is triggered. If it detects that the key will expire, it will generate a new one and store it in the secrets management system as the “Current Key.” Also, the Current key will be deprecated and become the new “Previous Key.”
The Previous Key is retained for use during the grace period, the time between generating the new key and its actual expiration. This is because the data sender may still encrypt using the previous key during this period. Thus, the Previous Key serves as a fallback option in case we are unable to decrypt the data using the new Current key.
The older Previous Key will be deleted permanently because storing every version of the keys could make our data more vulnerable to hackers.
Secret Management System
The Secrets Management System is the system that stores multiple digital secrets covered by an authentication layer. It also offers useful features such as web UI, token-based access control, etc.
There are several secret management systems, such as Spectral Secret Scanner and Confidant. They could be your choice for storing secrets with a nice web UI. We integrated the key rotation service with our organization’s secrets management system, HashiCorp Vault.
As mentioned above, each encryption key will have different policies. So, in our project, we could specify the policies of each key in the config file. Below is an example config of an encryption key we want to rotate.
We parameterize everything that could be different for each key in the config to make it scalable and suited for our use cases. Below are the details of these parameters.
- name: your key name.
- cron-schedule: cron expression for triggering the process.
- type: Specification we should use to generate this encryption key.
- pass-phrase-length: passphrase length for your key.
- expire-in-days: The interval before the next expiration date.
- grace-period: If the key rotation service fails to generate the new key on the expiration date, we won’t have a valid key on that day. So, we have this grace period to start generating a new key before the key expires.
- algorithm: such as RSA and AES.
- key-size: bit size related to the key and algorithm.
- vault-location: config to integrate with your Secret Management System. In our case, vault provides HTTP API to interact with the storage.
After the config of all the keys is prepared, we test and deploy the service to production and let it monitor and rotate the keys for us.
On top of that, we have some useful features to give us more convenience, such as
- Email notification — in case we need to notify someone within the organization or send the public key to the external party.
- Grafana dashboard — for monitoring each key rotation job details, such as the expiration date of each key. It also alerts if something goes wrong during the process and provides visibility on the service availability.
Conclusion
Automated key rotation provides multiple benefits, including preventing key compromise and minimizing potential data leaks. By implementing this approach, we not only enhance our security but also eliminate the need for manual key rotation, saving valuable time and reducing the risk of human error.
