Transforming Kubernetes Secret Management

The Evolution of Configuration and Secret Management

Configuration management has always been an important aspect of application deployments since inception, from the times of traditional IT infrastructures to the latest cutting-edge cloud platforms. These configurations are vital as they offer the flexibility to modify application behavior in various environments. For instance, adjusting log levels or altering database connections can significantly impact how an application functions in different environments.

However, some configurations are sensitive and demand protection. This includes safeguarding against external threats such as cyberattacks by bots and hackers, as well as internal risks like the accidental exposure of sensitive data to unauthorized personnel within an organization. The sensitivity of data, especially Personally Identifiable Information (PII), Protected Health Information (PHI), and other confidential information, necessitates stringent security measures. This need for security is further underscored by compliance frameworks such as HIPAA for healthcare, ISO standards for information security management, PCI-DSS for financial service providers, GDPR, and SOC-2 for service organizations.

The rise of container technologies and orchestrators like Kubernetes has revolutionized deployment processes, offering numerous benefits in terms of scalability and efficiency. However, this advancement has also introduced a new threat landscape. With AI-powered cyberattacks becoming more sophisticated, the management of secrets — sensitive data crucial for the operation of applications — becomes increasingly critical.

Kubernetes, despite being one of the most popular container orchestrators, has inherent challenges in secret management. While Role-Based Access Control (RBAC) can restrict access to secrets, these are often stored as base64-encoded strings, which are not truly secure and can be easily decoded with basic shell commands. This weakness highlights the importance of using external secret stores that offer enhanced security measures. There are a few more hidden challenges with native secret management in Kubernetes -

1. Scalability and Automation: As the number of microservices increases, managing secrets for each service manually becomes impractical.
2. Security: Sensitive data needs to be protected both at rest and in transit within the Kubernetes ecosystem.
3. Secret Rotation and Management: Frequent updating and management of secrets is required without causing service disruptions.

Prominent tools for managing these secrets include AWS Secret Manager, and HashiCorp Vault, along with options provided by other cloud platforms like Google Cloud’s Secret Manager and Azure Key Vault. Each of these tools offers unique features to securely manage and access secrets across distributed systems and cloud-native environments.

In this post, we will focus on using AWS Secret Manager to manage secret configurations in applications deployed on an AWS EKS Cluster. This choice allows us to maintain simplicity while illustrating a robust security mechanism, ensuring that sensitive configurations are securely managed and accessed in a cloud-native environment.

These hidden shortcomings of Kubernetes Native secret managing can be overcome by using AWS Secret Manager Service -

1. Limited Access Control: Kubernetes Secrets have limited access control, and access is managed through Kubernetes Role-Based Access Control (RBAC). This means that anyone with access to the Kubernetes cluster can potentially access the secrets stored in Kubernetes Secrets. In contrast, AWS Secrets Manager provides granular access control, allowing you to restrict access to specific secrets based on user roles.
2. Limited Secret Types: Kubernetes Secrets are limited to storing only small amounts of data, such as usernames and passwords. AWS Secrets Manager, on the other hand, can store a wide range of secrets, including database credentials, API keys, and certificates.
3. Lack of Encryption Options: Kubernetes Secrets are encrypted at rest, but you have limited control over the encryption mechanism. AWS Secrets Manager provides more advanced encryption options, including customer-managed keys and AWS Key Management Service (KMS) encryption.
4. Limited Auditing and Monitoring: Kubernetes Secrets does not provide extensive auditing and monitoring capabilities. AWS Secrets Manager provides detailed logging and auditing of secret access, helping you identify potential security threats.

Solving additional Challenges with External-Secrets Operator and Config-Reloader

There is always a scope to make things better, and the same applies here as well. The secrets are stored and managed in AWS Secret Manager services, protected by the strong capabilities of AWS IAM service. But still accessing these secrets in Kubernetes application deployments and keeping them in sync with changes require 2 additional tools -

1. External-Secrets Operator: Automates the synchronization of secrets from external sources like AWS Secret Manager or HashiCorp Vault, to Kubernetes, enhancing security and reducing manual errors.
2. Config-Reloader: Ensures that any changes in configurations or secrets are automatically reflected in the running applications without requiring manual restarts or redeployments.

Curious to know how all these tools and technologies work together to build a strong cloud security foundation? Let’s see an example.

Quick Hands-On Guide

Setting Up the Kubernetes Cluster

For those who don’t have an existing Kubernetes cluster, our GitHub repositories offer Terraform modules to create a production-grade VPC network and EKS cluster, complete with the necessary add-ons:

1. VPC Network: GitHub — Terraform Module to create an AWS VPC network with VPN and configure Peering b/w multiple VPCs
2. EKS Cluster: GitHub — Terraform module to create an Elastic Kubernetes Service(EKS) cluster, managed(CIS) node group and associated resources
3. EKS Add-ons: GitHub — Terraform module to bootstrap Elastic Kubernetes Service(EKS) cluster using Addons ( EKS add-ons ) and blueprints.

These battle-tested modules simplify setting up a robust Kubernetes environment ready to host your applications in Production environments.

Apply this manifest for creating a service account

Installing External-Secrets Operator and Config-Reloader

Assuming a Kubernetes cluster is already up and running, the next step is to install the External-Secrets Operator and Config-Reloader. These add-ons can be installed via Helm charts or using the YAML files provided in the repositories.

This will deploy the External-Secrets controller in your Kubernetes cluster. You can verify that the controller is running by checking the External-Secrets pod’s status

Creating and Using a Secret in AWS

1. Create an IAM policy and an IAM role to get access to the secret manager service

Make sure to change the AWS account ID and OIDC of your EKS cluster below.

Attach the IAM policy to the IAM role created above.

2. Create a Secret in AWS Secret Manager

For this example, we will create a secret that contains a variable and value. Here is an example AWS CLI command to create the secret.

Verify the secret was created in the AWS console.

Create a ClusterSecretStore in your EKS cluster to point to the AWS Secret Manager

3. Create an External Secret

The next step is to create an External Secret that will reference the AWS Secret we created in Step 1. We will use this Kubernetes Secret in our application’s deployment. Here is an example of Kubernetes manifest to create the Secret

4. Deploy a Sample Application:

For demonstration, deploy a simple application like an Nginx deployment in Kubernetes. This deployment should be configured to retrieve its secret value from AWS Secret Manager.

The next step is to deploy a sample application to use the Kubernetes Secret we created in Step 2. Here is the deployment manifest that uses the AWS secret

In the above manifest, we have defined an environment for using the secret value from aws secret manager. Also, we have added one annotation that will be responsible for updating the value inside the NGINX deployment using the config reloader, whenever there is a change in the AWS secret value.

5. Create a Service For sample nginx Deployment:

In this manifest, we are exposing nginx deployment using the service type Loadbalancer. Apply this manifest file for creating a service for nginx deployment.

Get the AWS loadbalancer endpoint.

6. Verify Dynamic Updates:

As of now, the secret named “secret-demo” in AWS Secret Manager holds a specific value “hello from secret version 1”

Upon accessing the Nginx deployment through the LoadBalancer endpoint, the current secret value is successfully displayed in the browser.

Let’s change the value of the secret manager from hello from secret version 1 to hello from secret version 2

AWS CLI command for changing secret value

Verify that it gets updated on the AWS console

Verify the change in the browser on the Loadbalancer endpoint.

Refreshing the index page of the Nginx application in the browser should now display the updated secret, showcasing the dynamic update capability enabled by Config-Reloader.

Conclusion

In conclusion, managing secrets securely is a critical aspect of any application deployment. AWS Secrets Manager provides an easy-to-use interface to manage your secrets, and integrating it with Kubernetes using External-Secrets and Config-Reloader simplifies the process of retrieving and updating secrets.

In this blog, we have covered the steps involved in using AWS Secrets Manager with Kubernetes External-Secrets and Config-Reloader. We started by creating an IAM policy and role that allows your Kubernetes cluster to access AWS Secrets Manager. We then created a Kubernetes Secret using External-Secrets that references an AWS Secrets Manager secret. Finally, we configured Config-Reloader to reload your application’s configuration automatically whenever your secrets or ConfigMaps change.

By following these steps, you can securely manage your secrets and keep your application configuration up-to-date without requiring any manual intervention. This not only saves time but also helps in ensuring the security of your application.

Therefore, it’s a best practice to use a combination of Kubernetes Secret Manager and other third-party solutions like AWS Secrets Manager to manage your secrets securely. This approach allows you to leverage the strengths of both solutions and helps you build a robust and secure secrets management system.