Planning a production ready kubernetes with fundamental Controllers & Operators — Part 2 — Secrets Management and Config-rotation

This post will continue presenting the idea of “additional controllers which do 80% of the job of making your kubernetes cluster production ready”

As I already mentioned in the introduction post, we are going to go through the steps needed to make a production grade kubernetes cluster. One of the main pain points with sensitive data is that we need to learn how to handle it … storing it in our cluster in base64 is a pragmatic way of dealing with sensitive data at runtime considering our application needs the content of the secret …

Let’s imagine you have a human secret manager that adds the sensitive data manually to the cluster — the word manually is quite problematic for production grade nonetheless how do we keep the content safe (users can still retrieve the secret resource) considering anyone can run the following command:

kubectl get secret <your-secret-name> -o json | jq '.data | map_values(@base64d)'

There are a few ways to handle this,

1. Harden your Role Based Access Control taking away the ability to access the secret resource

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: secret-reader-binding
  namespace: default
subjects:
- kind: User
  name: "specific-user"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: secret-reader
  apiGroup: rbac.authorization.k8s.io

2. The second would be replacing the human secrets manager with a SaaS based secrets-manager (could be in cloud and on premise depending on your clusters setup), as mentioned in the intro post I am a big fan of External Secrets Operator considering it integrated well with many backends and allows me to manage secrets at rest and in transport so let’s dig into using ESO, then we will discuss on how to apply configurations to our workload.

The External Secrets Operator (ESO) bridges these gaps by enabling more secure and dynamic secret handling. Here’s a detailed look at the functionality provided by the External Secrets Operator, including its key components and how they enhance security.

Functionality Provided by External Secrets Operator

The External Secrets Operator extends Kubernetes’ native secrets management capabilities by interfacing with external secret management systems such as AWS, Azure , GCP Secrets Managers or HashiCorp Vault, . It automates the creation, deletion, and updating of secrets in Kubernetes based on these external sources.

This approach allows secrets to be managed at the external provider, leveraging their more robust security measures such as encryption at rest and controlled access policies, the same RBAC we applied above to the secret also apply to the Custom Resources Introduced by the operator and can conceal not only the secret but it’s origin too …

Example of a Custom Resource Definition (CRD)

The External Secrets Operator introduces two primary CRD’s: the secretStore or clusterSecretStoreand the Externalsecret. The secretStore CRD defines the connection details to the external secret management system, including the provider type and authentication configuration. The Externalsecret CRD defines how individual secrets are synchronized into Kubernetes from the external systems.

Here’s a basic example of a `SecretStore` and an `ExternalSecret` CRD for interfacing with AWS Secrets Manager:

This snippet will be a bit different for gcp/azure/vault but the idea is the same:

# an exmaple of a secretStore using AWS CREDENTIALS (Not recommended)
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-secret
            key: accessKeyID
          secretAccessKeySecretRef:
            name: aws-secret
            key: secretAccessKey
---
# an exmaple using a service account with an attached iam Role 
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-clustersecretstore
spec:
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
      region: eu-central-1
      service: SecretsManager
---
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: database-password
spec:
  secretStoreRef:
  # 1st example using secretStore + credentials
  # kind: secretStore
  # name: aws-secrets
  # 2nd example using ClusterSecretStore + irsa (sa + iam role)
    kind: ClusterSecretStore
    name: my-clustersecretstore
  target:
    name: db-secret
  data:
    - secretKey: dbPassword
      remoteRef:
        key: /prod/db/password
        property: password

DALL-E | Externl Secrerts Operator

Components of External Secrets Operator

SecretStore or ClusterSecretStore: This CRD allows you to define where and how the secrets are stored externally. Each `SecretStore` can be specific to a cloud provider or a secret management system, defining the necessary authentication and configuration details to securely access the secrets.

ExternalSecret: This CRD represents the actual secret data that needs to be synchronized into Kubernetes. It references a `SecretStore` and specifies which secrets to fetch and how they should be presented within the Kubernetes cluster. The operator then continuously monitors these external secrets for any changes and updates the Kubernetes secrets accordingly.

Enhancing Security

The use of External Secrets Operator significantly enhances the security posture of applications by:

Centralizing Secret Management: Keeping all secrets in a dedicated external system that is purpose-built for secret management reduces the risk of accidental exposure.

Leveraging Advanced Security Features: External secret managers often provide advanced security controls, including detailed access policies, audit logs, and automatic secret rotation.

Reducing Attack Surface: By not storing secrets within the Kubernetes ley-value database, the External Secrets Operator minimizes the attack surface, as potential attackers would not find any sensitive data in the cluster itself.

Compliance and Best Practices: Using an external system for secrets management helps comply with security best practices and regulatory requirements that mandate strict handling and auditing of sensitive data.

In conclusion, the External Secrets Operator extends the basic secret management capabilities of Kubernetes by integrating with more secure and feature-rich external secret management services. This integration not only simplifies the management of secrets but also significantly boosts the security infrastructure of applications deployed in Kubernetes environments.

There is one flaw in the above, what happens when the secret content changes ? who updates / reloads the workload ? …

To enhance the discussion on configuration management in Kubernetes, it’s crucial to address the timing and automation of configuration updates.

A common challenge is controlling when new configurations are applied, as changes in Secrets or ConfigMaps do not automatically trigger updates for the applications using them.

Fortunately, this limitation can be effectively managed with the help of controllers like Stakater/Reloader.

This controller can monitor changes in ConfigMaps and Secrets and automatically initiate a rolling update to the relevant deployments, ensuring that applications always run with the latest configurations.

This capability complements tools like the External Secrets Operator, which syncs external secrets into Kubernetes, by ensuring that any updates to these secrets are promptly and safely rolled out to the applications. Thus, the integration of such controllers strengthens Kubernetes’ configuration management ecosystem, enabling more dynamic and responsive application environments.

Reloader | a controller which updates the workload there is a configuration update

Wait, what if we don’t want to apply / run a rolling update … Reloader like many other controllers in the cloud-native landscape work with annotations … taken directly from the reloader’s documentation:

For a Deployment called foo have a ConfigMap called foo. Then add this annotation* to your Deployment, where the default annotation can be changed with the --configmap-annotation flag:

metadata:
  annotations:
    configmap.reloader.stakater.com/reload: "foo"

And the same applies for Secrets for a Deployment called foo have a Secret called foo. Then add this annotation to your Deployment, where the default annotation can be changed with the --secret-annotation flag:

metadata:
  annotations:
    secret.reloader.stakater.com/reload: "foo"

This is where Kubernetes’s control loop comes into play — putting the 2 controllers ESO & Reloader to test, then we will have the following workflow which is purely event driven and fully automated thanks to these 2 controllers:

1. Secret update in the external secret manger will be picked up by the External Secrets Operator
2. ExternalSecret resource is updated and generats the secret
3. Reloader operator identifies the change event and triggers our Deployment to be updated …

To conclude, managing configuration especially sensetive information could be quite a challenge, nonetheless there are great cloud native options out there for secrets I can recall a handfull I worked with External Secrets Operator and Reloader are the choices we took for our infastructure and is considered one of our best practices.

Stay tuned for the next topic in the series discussing Ingress http traffic.
As always would love to get your thoughts on these topics, feel free to drop my a line / comment below.

Your Sincerely, Haggai Philip Zagury