From vulnerable to unhackable: secrets management in cloud-native environments

In the fast-paced world of cloud-native development, there’s a unique thrill in knowing that your infrastructure is robust, resilient, and… well, secure. But let’s be honest, secrets management in Kubernetes? It’s a bit like locking your front door with a piece of string. Sure, it might keep out the casual passerby, but a determined thief? They’ll waltz right in.

The day I learned Base64 isn’t enough

Let me take you back to a project that had me sweating bullets. I was deploying a critical application, one where security wasn’t just a priority — it was the priority. “Just use Kubernetes Secrets,” they said. And so I did. But something didn’t sit right. I couldn’t shake the feeling that I was missing something crucial.

And then it hit me — Base64 encoding. The so-called “security” that Kubernetes provides out of the box isn’t encryption at all. It’s like hiding your keys under the doormat and hoping no one lifts it. I knew then that I had to find a better way.

Better tools, better security: discovering IBM Secrets Manager, Vault API, and ESO

This realization sent me down a rabbit hole of research, and what I found changed everything. Tools like IBM Secrets Manager, the Vault API, and External Secrets Operator (ESO) weren’t just better — they were a game-changer. They offered the kind of robust encryption that keeps your secrets truly safe, and they integrated seamlessly with the existing infrastructure.

But here’s the thing: setting them up wasn’t just about following instructions. It was about understanding the risks, the pitfalls, and why these tools are essential for any serious cloud-native environment.

secrets management in cloud-native environments

When ESO came to the rescue

ESO (External Secrets Operator) was the hero of my story, swooping in to save the day with its ability to extend Kubernetes through Custom Resources. Picture this: you’re managing secrets from an external source, pulling them into your Kubernetes cluster without breaking a sweat. ESO makes this possible with two key components:

• ExternalSecret: It fetches and syncs secrets from external sources.
• SecretStore: This defines the connection between ESO and your external sources, detailing exactly how and where to access your secrets.

Here’s how you can set up ESO using two simple methods: the default CatalogSource or an internal registry.

Option 1: using default CatalogSource

1. Create CatalogSource:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: external-secrets-operator-catalog
  namespace: external-secrets
spec:
  sourceType: grpc
  image: ghcr.io/external-secrets/external-secrets-helm-operator-catalog:latest
  displayName: External Secrets Operator
  updateStrategy:
    registryPoll:
      interval: 30m

2. Create OperatorGroup: to ensure ESO operates across all namespaces, create an OperatorGroup that specifies an empty namespace, indicating a global scope:

apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: external-secrets
  namespace: external-secrets
spec:
  targetNamespaces:
    - ''

3. Create Subscription:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: external-secrets-operator
  namespace: external-secrets-subscription
spec:
  channel: stable
  installPlanApproval: Automatic
  name: external-secrets-operator
  source: external-secrets-operator-catalog #The catalog source from which the operator will be installed.
  sourceNamespace: external-secrets #The namespace where the catalog source is located.

Option 2: using internal registry

1. Mirror catalog to internal registry:

oc adm catalog mirror ghcr.io/external-secrets/external-secrets-helm-operator-catalog:latest <your-internal-registry>/<path>

2. Create CatalogSource for internal registry:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: external-secrets-operator-catalog
  namespace: external-secrets
spec:
  sourceType: grpc
  image: <your-internal-registry>/<path>:latest
  displayName: External Secrets Operator
  updateStrategy:
    registryPoll:
      interval: 30m

Repeat Step 2 and 3 from Option 1. Easy!

The dual API approach: flexibility meets security

We’ll use a dual API approach: IBM Secrets Manager API for setting up the stage, and a custom API that communicates with IBM Secrets Manager using the standard Vault API, providing a familiar interface for CLI commands and tools, while also managing synchronization with the External Secrets Operator (ESO).

Let’s play with Secrets Manager API

IBM Secrets Manager API builds on the HashiCorp Vault API but with the added power of IBM Cloud IAM Auth methods. With this, we created secret groups, generated API keys, and set IAM access policies.

Here’s a snapshot:

• Create secret groups: organize secrets into groups for easier management.

curl -X POST "https://<your-region>.secrets-manager.cloud.ibm.com/v2/secrets-groups" \
     -H "Authorization: Bearer <IAM_ACCESS_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"name": "example-secret-group"}'

• Create API keys: generate one API key for each secret group to control access securely.

curl -X POST "https://<your-region>.secrets-manager.cloud.ibm.com/v2/api-keys" \
     -H "Authorization: Bearer <IAM_ACCESS_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"name": "example-api-key", "secret_group_id": "<SECRET_GROUP_ID>"}'

• Set IAM Policies: limit the API key’s access to the group it’s assigned to via IAM policies.

curl -X POST "https://<your-region>.secrets-manager.cloud.ibm.com/v2/iam-policies" \
     -H "Authorization: Bearer <IAM_ACCESS_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"api_key_id": "<API_KEY_ID>", "secret_group_id": "<SECRET_GROUP_ID>", "policy": "read"}'

This setup lays the groundwork for managing secrets and prepares us for detailed CRUD (Create, Read, Update, Delete) operations using familiar tools.

Vault API: seamless integration

To leverage the familiarity of the Vault API while integrating with IBM Secrets Manager, we’ll develop a custom API. This custom API will act as a bridge, allowing you to perform CRUD operations using the Vault API format while interacting with IBM Secrets Manager.

Key functions of the custom API:

1. Create a KV Secret:

curl -X POST 'https://<your-instance-id>.<your-region>.secrets-manager.appdomain.cloud/v1/ibmcloud/kv/secrets' \
     -H 'Accept: application/json' \
     -H 'X-Vault-Token: <Vault-Token>' \
     -H 'Content-Type: application/json' \
     -d '{
         "name": "test-kv-secret",
         "description": "Extended description for my secret.",
         "payload": {
             "key1": "value1"
         },
         "labels": [
             "dev",
             "us-south"
         ]
     }'

• Read a KV Secret:

curl -L -X GET 'https://<your-instance-id>.<your-region>.secrets-manager.appdomain.cloud/v1/ibmcloud/kv/secrets/<secret-id-or-secret-name>' \
     -H 'Accept: application/json' \
     -H 'X-Vault-Token: <Vault-Token>'

• Update a KV Secret:

curl -X POST 'https://{instance_id}.{region}.secrets-manager.appdomain.cloud/v1/ibmcloud/kv/secrets/groups/{group_id}/{secret_id}/rotate' \
     -H 'Accept: application/json' \
     -H 'X-Vault-Token: {Vault-Token}' \
     -H 'Content-Type: application/json' \
     -d '{
         "payload": {
             "key": "new_value"
         }
     }'

• Delete a KV Secret:

curl -X DELETE "https://{instance_id}.{region}.secrets-manager.appdomain.cloud/v1/ibmcloud/arbitrary/secrets/groups/{group_id}/{secret_id}" \
     -H 'Accept: application/json' \
     -H 'X-Vault-Token: {Vault-Token}'

• Integration with ESO:

The custom API creates an External Secret object in the Kubernetes cluster when a new secret is created. For updates and deletions, it updates the annotations on the relevant External Secrets objects. This method allows ESO to detect and synchronize changes within the Kubernetes cluster.

CLI fun …

And for those CLI enthusiasts out there? We even built a tool that mirrors Vault’s commands, making it feel like home.

CLI example:

cvault read -format=json ibmcloud/arbitrary/secrets/groups/{secret-group}

In this example:

• cvault is the custom command-line tool.
• read is the command used to retrieve secrets.
• -format=json ensures the output is in JSON format.
• ibmcloud/arbitrary/secrets/groups/{secret-group} indicates the path to the secret group.

The custom CLI translates this command into necessary API calls, managing secret CRUD and ensuring synchronization with ESO.

Conclusion

In the end, it wasn’t just about setting up a system; it was about evolving from vulnerable to unhackable. With IBM Secrets Manager, the Vault API, and ESO, we’ve fortified what was once a critical weakness. Now, we can shift our focus back to what truly matters: driving innovation and growth

Connect with me

Got thoughts, questions, or just want to chat? I’d love to hear from you! Let’s connect and explore new ideas.

Happy Kuberneting! ⎈🚀