Keeping Secrets in an ASP.NET Kubernetes Application

In an earlier story titled Keeping Secrets in ASP.NET’s appsettings.json, I demonstrated how to keep secrets in appsecrets.json using Google Cloud Key Management Service (KMS). The resulting application is easy to maintain because secrets are encrypted and versioned with the source code, decryption keys are never distributed to developers or systems administrators, and secrets are decrypted at the last minute by the production server running the application. The full story contains many more details.

I had deployed the application to Google Compute Engine and Google App Engine. Recently, I decided to deploy the same application to Google Kubernetes Engine (GKE). Because keeping secrets is serious business, I wanted to follow security best practice and use a Google service account with minimal privileges. Service accounts are used for authentication and authorization. The default GKE service account had way more privileges than my application needed, and that’s a security risk.

Here’s a super-quickie introduction to service accounts. A conversation between my app and Google Cloud KMS looks like this:

To be able to decrypt appsecrets.json.encrypted on a GKE instance, I would have to install Google Cloud service account credentials on the GKE instance.

I found two ways to do this:

Method 1 installs service account credentials when a new GKE cluster or node pool is created. It’s quick and it’s easy, but it won’t work if the Kubernetes cluster is running elsewhere.

Method 2 installs a service account key as a Kubernetes secret. It works when a cluster is running inside or outside Google Cloud. Unlike method 1, it delivers the service account key to existing nodes, so no new nodes need to be created. It also allows different apps running on the same node to use different service accounts.

The most “Kubernetes way,” to deliver secrets to an application is to create a Kubernetes secret containing appsecrets.json. I plan to discuss this third method in a later post, but I will stick with method 1 and method 2 for this post because they follow the pattern described in Keeping Secrets in ASP.NET’s appsettings.json, and many people found that pattern useful. Plus, once I had Google service account credentials installed, I could easily call other useful Google Cloud APIs like Speech-to-Text and Vision.

Creating a Google Cloud Service Account

Both methods require a Google Cloud service account to authenticate my app, so I created a new service account with the gcloud command found in the Google Cloud SDK:

> gcloud iam service-accounts create my-service-account
Created service account [my-service-account].

Following the principle of least privilege, I gave the service account just enough permissions to pull Docker images from Google Cloud Container Registry and write debugging information:

Then, I gave the service account permission to decrypt appsecrets.json.encrypted:

My new service account was ready to go with the minimal permissions necessary to run apps on GKE and decrypt appsecrets.json.encrypted.

Method 1

When creating a new cluster on GKE, I used the --service-account flag to specify the service account I created above.

I could also have created a new node pool in an existing cluster and passed the same --service-account flag.

I deployed my application and exposed port 8080:

And I visited the application home page. It successfully decrypted the secret word!

Success!

Method 2

Method 1 is nice, but it only works when my app is running on GKE. How could I install Google Cloud service account credentials on Kubernetes nodes running elsewhere?

Fortunately, all Google Client API libraries inspect the environment variable GOOGLE_APPLICATION_CREDENTIALS. The environment variable GOOGLE_APPLICATION_CREDENTIALS stores a path to a service account key, so the first thing I needed was a service account key. I created the service account key with gcloud:

It’s tempting to pack the key into the Docker image, but this would be very insecure. Everyone with permission to read the Docker image would be able to decrypt appsecrets.json.encrypted. That’s bad.

Then I learned about Kubernetes secrets. A Kubernetes secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. That’s exactly what I needed.

I created a kubernetes secret and added key.json:

I tried deploying my application with the same kubectl command I used in method 1, but I forgot to set GOOGLE_APPLICATION_CREDENTIALS, so the application was unhealthy:

The container logs tell me why, and it’s exactly the permission error I expected:

It was time to set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the Kubernetes secret I created above. That required touching my deployment config in 3 different places. I edited the deployment with the command kubectl edit deployment social-auth-v3 and modified the deployment. Here’s the abbreviated yaml file, with my changes marked by comments.

I restarted my deployment with a rolling update, and finally, my app found the service account credentials and successfully decrypted appsecrets.json.

Success!

Conclusion

gcloud makes it easy to install service account credentials when you create a new Kubernetes cluster or node pool. Kubernetes secrets make it possible to install service account credentials no matter where your application is running.

The hardest part of this exercise was finding the right, minimal set of permissions to grant to the service account. Naturally, immediately after I had my app up and running, I found that exact information here. 😀

Full instructions for authenticating to Google Cloud Platform with service accounts are here. And the full application sample source code is here.