Kubernetes Engine access and audit

Adventures in Kubernetes

Published in

Google Cloud - Community

5 min readAug 10, 2018

I’ve been exploring ways to share access to a clusters with trusted external developers. These developers need more than pure Kubernetes access and will want to be able to ssh into Nodes too.

Useful Diversion: gcloud — account=[[ACCOUNT]]

I discovered this week that there’s a gcloud (for GCP) flag to specify the Google account to be used for the scope of the command. This feature is useful if you wish to be explicit about which account is being used (as I do with GCP projects and Kubernetes clusters, see story) but, it more easily permits switching accounts on-the-fly.

Example:

gcloud auth listACTIVE  ACCOUNT
*       my@gmail.com
        my@other.com
        my@google.comTo set the active account, run:
    $ gcloud config set account `ACCOUNT`

For convenience, I do have a default (active) gcloud account. This is the most recent gcloud auth login account but may also be set using gcloud config set account [ACCOUNT].

Until this week, I would perform this dance:

gcloud config set account my@google.com
gcloud some-command
gcloud config set account my@gmail.com
gcloud some-command

Now, I’ve learned that it’s possible (and far more elegant) to:

gcloud some-command --account=my@google.com
gcloud some-command --account=my@gmail.com

In order to test access for the developers, I’m using my Gmail account as the proxy.

Google Groups

It’s a good practice to use Google Groups for IAM role assignment in GCP projects. Generally (!) you would use Google Groups created on your organizational domain (e.g. acme.com) to create these groups. In my case, because the developers aren’t Googlers and because we’re (generally) not permitted to add non-@google.com accounts to Groups created on google.com, I’m going to use the public Google Groups domain (googlegroups.com).

Because my Gmail account creates the Group, it is automatically a member of the Group and I can immediately start testing. I will need to add Google accounts for the developers to this group for the developers to work. But (!) because I’m using a Group and assigning IAM roles to the Group, I can be assured that my Gmail account and their Google accounts will have functionally equivalent roles on the Project:

IAM

Here’s a snapshot of the Group’s permissions in the project. You may create this using the Cloud Console UI or, I’ll show you how to do this from the command-line. First we need to create the second role instance-ssh.

The first role is Kubernetes Engine Developer. This provides developer access to the cluster but, I also want the developers to be able to ssh into the cluster’s Nodes. The Nodes are Compute Engine VMs and there’s no out-of-the-box role that provides ssh only access. So, I created one using Custom Roles:

gcloud iam roles create instance_ssh \
--project=${PROJECT} 
--file=instance_ssh.role.yaml

and the YAML:

These 4 permissions permit accounts with this role to list the instances in a project and to ssh into them. The setMetadata permission supports gcloud compute ssh adding an account’s public key to the project’s metadata service. The accounts must also be able to use the Compute Engine service account (this is a slightly confusing albeit necessary feature; we treating a service account as a resource):

gcloud iam service-accounts add-iam-policy-binding \
${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \ 
--member=group:${GROUP}@googlegroups.com \
--role=roles/iam.serviceAccountUser \
--project=${PROJECT}

The Compute Engine service account is generated automatically and its name if formed from a combination of your GCP project’s number (not ID) and “-compute@developer.gserviceaccount.com”. It’s easiest to eyeball the Email address from Cloud Console:

Determining the project’s default Compute Engine service account

Kubernetes Engine Users

Be aware that, once authenticated by gcloud, Google accounts remain credentialed for use by the OS user. In my case, on my work machine (when I’m logged in), I can flip between the 3 Google accounts shown, simply by using the --account=[[ACCOUNT]] flag.

NB You must gcloud auth revoke [[ACCCOUNT]] to delete the cached credentials.

This fact is relevant to Kubernetes Engine too. Although Kubernetes maintains its own auth mechanisms, Kubernetes Engine supports Google’s OAuth and, once the user has gcloud container clusters get-credentials … and here’s the rub… *any* account that’s credentialed by gcloud and which has suitable permission to the cluster, may use it. Corrollary: it’s not just the account that was used by the get-credentials command.

If my@google.com created the cluster and got credentials, because I’d ACL’d my@gmail.com per the instructions above for the same project, the account that is the active account for gcloud, will be the account that kubectl (!) uses.

wai-wha!??

gcloud config set account my@gmail.com
kubectl apply --filename=${DEPLOYMENT-1} \
--namespace=${NAMESPACE} \
--context=${CONTEXT}
gcloud config set account my@google.com
kubectl apply --filename=${DEPLOYMENT-2} \
--namespace=${NAMESPACE} \
--context=${CONTEXT}

Will deploy the {DEPLOYMENT-1} as my@gmail.com and ${DEPLOYMENT-2} as my@google.com.

How can I confirm this? Logs.

Audit Logs

Nothing gets past the audit logger. What we want to do is determine who did what and when? It’s very easy:

PROJECT=[[YOUR-PROJECT]]
ACCOUNT=[[YOUR-ACCOUNT]]LOG="cloudaudit.googleapis.com%2Factivity"
FILTER="logName=\"projects/${PROJECT}/logs/${LOG}\" "\
"resource.type=\"k8s_cluster\""gcloud logging read "${FILTER}" \
--freshness=1h \
--project=${PROJECT} \
--account=${ACCOUNT} \
--format=json \
| jq --raw-output '.[].protoPayload.methodName' \
| sort \
| uniqio.k8s.apiextensions.v1beta1.customresourcedefinitions.create
io.k8s.apiextensions.v1beta1.customresourcedefinitions.patch
io.k8s.apiregistration.v1.apiservices.create
io.k8s.apps.v1.deployments.create
io.k8s.app.v1alpha1.applications.create
io.k8s.authorization.rbac.v1.clusterrolebindings.patch
io.k8s.batch.v1.jobs.create
io.k8s.certificates.v1beta1.certificatesigningrequests.delete
io.k8s.core.v1.configmaps.create
io.k8s.core.v1.configmaps.update
io.k8s.core.v1.endpoints.create
io.k8s.core.v1.endpoints.update
io.k8s.core.v1.namespaces.create
io.k8s.core.v1.persistentvolumeclaims.create
io.k8s.core.v1.persistentvolumeclaims.update
io.k8s.core.v1.persistentvolumes.create
io.k8s.core.v1.persistentvolumes.update
io.k8s.core.v1.pods.binding.create
io.k8s.core.v1.pods.create
io.k8s.core.v1.secrets.create
io.k8s.core.v1.serviceaccounts.create
io.k8s.core.v1.serviceaccounts.update
io.k8s.core.v1.services.create
io.k8s.extensions.v1beta1.deployments.create
io.k8s.extensions.v1beta1.deployments.patch
io.k8s.extensions.v1beta1.replicasets.create
io.k8s.storage.v1.storageclasses.create
io.k8s.storage.v1.storageclasses.patchgcloud logging read "${FILTER}" \
--freshness=1h \
--project=${PROJECT} \
--account=${ACCOUNT} \
--format=json \
| jq --raw-output '.[].protoPayload | { email:.authenticationInfo.principalEmail, method:.methodName }'gcloud logging read "${FILTER}" \
--freshness=1h \
--project=${PROJECT} \
--account=${ACCOUNT} \
--format=json \
| jq --raw-output '.[].protoPayload | .authenticationInfo.principalEmail + " " + .methodName ' \
| sort \
| uniqcluster-autoscaler io.k8s.core.v1.configmaps.update
cluster-autoscaler io.k8s.core.v1.endpoints.update
my@gmail.com io.k8s.apps.v1.deployments.create
my@gmail.comio.k8s.app.v1alpha1.applications.create
my@gmail.comio.k8s.batch.v1.jobs.create
my@gmail.comio.k8s.core.v1.configmaps.create
my@gmail.comio.k8s.core.v1.namespaces.create
my@gmail.comio.k8s.core.v1.persistentvolumeclaims.create
my@gmail.comio.k8s.core.v1.persistentvolumes.create
my@gmail.comio.k8s.core.v1.pods.create
my@gmail.comio.k8s.core.v1.services.create
my@gmail.comio.k8s.extensions.v1beta1.deployments.create
my@gmail.comio.k8s.storage.v1.storageclasses.create
my@gmail.comio.k8s.storage.v1.storageclasses.patch
system:apiserver io.k8s.apiregistration.v1.apiservices.create
system:kube-controller-manager io.k8s.core.v1.secrets.create
system:kube-controller-manager io.k8s.core.v1.serviceaccounts.update
system:kube-scheduler io.k8s.core.v1.pods.binding.create
...
system:unsecured io.k8s.core.v1.configmaps.update
system:unsecured io.k8s.extensions.v1beta1.deployments.patch

NB (Audit) Logs grow rapidly. It’s a good practice to limit the logs that you read by including the --freshness flag. Here set at one hour.

This is all documented for Kubernetes Engine, here:

https://cloud.google.com/kubernetes-engine/docs/how-to/audit-logging

jq

So, my interested was piqued as to how I could best transform the log data to get the methods by user:

gcloud logging read "${FILTER}" \
--freshness=1d \
--project=${PROJECT} \
--account=${ACCOUNT} \
--format=json \
| jq --raw-output '[.[].protoPayload | { email:.authenticationInfo.principalEmail, method:.methodName }]' \
| jq 'group_by(.email)' \
| jq 'map({"email": .[0].email, "method": map(.method) | unique})'

Prettier as a gist:

NB The jq commands may, of course, be composed into a single jq statement but, this represents the way I built up the pipeline, so I kept it.

Neat! It loses information (what deployment?) but… it shows the scope of the users’ methods.

Conclusion

In this story we’ve covered the useful account flag for gcloud. A brief detour through how Kubernetes Engine auth works and a soupcon of GCP IAM including custom roles, and lastly some audit logging goodness.

That’s all!