Kubernetes Engine: kubectl config

This has been documented elsewhere but I think it’s worthwhile to summarize kubectl and gcloud use of configs, their interaction, and how to switch between Kubernetes configs when you have multiple Kubernetes clusters.

I’m using Kubernetes Engine which makes configuration management much easier but — if you’re like me — this can result in complacency in letting the tools do their job without spending time to grok what’s going on.

Setup

Let’s assume you’ve created 2x Kubernetes Engine clusters and obtained credentials for both of them. You’ll be configured to use the cluster that was most recently authenticated:

gcloud container clusters create black ...
gcloud container clusters create white ...
gcloud container clusters get-credentials black ...
gcloud container clusters get-credentials white ...
kubectl get nodes
NAME                  STATUS    ROLES     AGE       VERSION
white-2378271e-qpd6   Ready     <none>    1h        v1.10.2-gke.3
white-5d71e034-njph   Ready     <none>    1h        v1.10.2-gke.3
white-f4567e1d-4rwd   Ready     <none>    1h        v1.10.2-gke.3

NB I’ve renamed NAME in the above to demonstrate the point.

We’re authenticated (currently) for our white cluster.

kubectl config

Let’s have a look at Kubernetes (Engine) configuration:

kubectl config view

The underlying configuration file is probably (!) located in:

~/.kube/config

Your config will resemble this file. I’ve redacted and renamed some elements for clarity.

There are 2 clusters, 2 contexts and 2 users here. There’s also a property current-context that points to the current context. One of the two names. In this case white.

One thing that may not be obvious is that, since I created both clusters (white and black) and I authenticated against both of them, I don’t need to have 2 different but identical users in my configuration. They both reflect the same user. I propose that you don’t this but for clarity, this configuration is identical to the above:

NB I’ve renamed one of the 2 users with my user name, deleted the duplicate and renamed the references in contexts with my user name.

Let’s drill into this auth-provider.

access-token

Grab the access-token from your kubectl config view. It should be of the form ya29.... Let’s call it ${ACCESS_TOKEN}.

NB The expiry in the results. If you try this command after that time, your token will have expired. If it has, make an arbitrary e.g.kubectl get nodes call to force a refresh of the token, rerun the kubectl config view and try again.

Then you may browse or curl:

curl https://www.googleapis.com/oauth2/v2/tokeninfo?access_token=${ACCESS_TOKEN}
{
 "issued_to": "12345678901.apps.googleusercontent.com",
 "audience": "12345678901.apps.googleusercontent.com",
 "user_id": "123456789012345678901",
 "scope": "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/appengine.admin https://www.googleapis.com/auth/compute https://www.googleapis.com/auth/plus.me",
 "expires_in": 1199,
 "email": "[[YOUR-EMAIL]]",
 "verified_email": true,
 "access_type": "offline"
}

NB email reflect the account that you are using with gcloud. If you review your gcloud configuration (either gcloud auth list or gcloud config list or, even more precisely, gcloud config get-value account), this should be your ACTIVE ACCOUNT name.

gcloud config

The auth-provider references gcloud. Specifically cmd-path, cmd-args and 2 properties expiry-key and token-key. cmd-path should match:

which gcloud
/.../google-cloud-sdk/bin/gcloud

cmd-args isn’t well-documented by Google (filed a bug :-)) but it is explained by one of Google’s Engineers (link):

gcloud config config-helper
configuration:
  active_configuration: default
  properties:
    core:
      account: [[YOUR-EMAIL]]
      disable_usage_reporting: 'False'
credential:
  access_token: [[ACCESS_TOKEN]]
  token_expiry: '2018-06-09T00:00:00Z'
sentinels:
  config_sentinel: /.../.config/gcloud/config_sentinel

or you can tack on --format=json and:

{
  "configuration": {
    "active_configuration": "default",
    "properties": {
      "core": {
        "account": "[[YOUR-EMAIL]]",
        "disable_usage_reporting": "False"
      }
    }
  },
  "credential": {
    "access_token": "[[ACCESS_TOKEN]]",
    "token_expiry": "2018-06-09T00:00:00Z"
  },
  "sentinels": {
    "config_sentinel": "/.../.config/gcloud/config_sentinel"
  }
}

When youkubectl, it uses the auth-provider configuration to use gcloud config config-helper to grab an access_token for your account so that it may access a cluster’s resources.

Let’s briefly detour through gcloud configurations to explain this.

gcloud config configurations

You’ll note in the above that gcloud has configurations too. I’ve used this facility less frequently but, for completeness…

gcloud config list
[core]
account = [[YOUR-EMAIL]]
disable_usage_reporting = False
Your active configuration is: [default]

NB [default]

Where does gcloud store its configuration? Probably:

ls -la ~/.config/gcloud

active_config
application_default_credentials.json
cache
config_sentinel
configurations

NB Edited to demonstrate the point

If you:

less ~/.config/gcloud/active_config
default

Which is a key to an entry in configurations prefix config_:

less ~/.config/gcloud/config_default
[core]
account = [[YOUR-EMAIL]]
[container]
[compute]

NB Your configuration may have more properties.

This should reflect what the result of gcloud config list.

You may (!) create multiple configurations to reflect different GCP configurations that you wish to use. For example:

gcloud config configurations list
NAME     IS_ACTIVE  ACCOUNT
default  True       [[YOUR-EMAIL]]
gcloud config configurations create henry
Created [henry].
Activated [henry].
gcloud config configurations list
NAME     IS_ACTIVE  ACCOUNT
default  True       [[YOUR-EMAIL]]
henry    True

Then, you can verify:

cat ~/.config/gcloud/active_config
henry
ls -l ~/.config/gcloud/configurations
config_default
config_henry

And, tidy up with:

gcloud config configurations activate default
gcloud config configurations delete henry
The following configurations will be deleted:
 - henry
Do you want to continue (Y/n)?  Y
Deleted [henry].

I find little use of gcloud configurations and instead tend to have multiple logins against the default configuration:

gcloud auth list
       Credentialed Accounts
ACTIVE  ACCOUNT
        [[EMAIL-#1]]
*       [[EMAIL-#2]]
        [[EMAIL-#3]]
To set the active account, run:
    $ gcloud config set account `ACCOUNT`

And, as shown above, switch between these using, e.g.gcloud config set account [[EMAIL-#3]].

kubectl config use-context

OK, we now understand how kubectl uses gcloud to auth against GCP. But, we’ve not yet addressed how to manage multiple kubectl configurations.

This is documented (link) but here’s my spin on it. Let’s return to our white and black clusters. As I mentioned, Kubernetes Engine will create unique names for your clusters,contexts and users. I’ve explained how you may (you don’t need to and be careful doing it) rename|consolidate your users. You may do the same thing for clusters and contexts too. Just be careful that you correctly rename all references to these in your config file.

If you break something, you may rerun gcloud container clusters get-credentials to re-authenticate against your cluster(s).

So, how do we switch between clusters? It’s actually straightforward. Assuming:

kubectl config get-contexts
CURRENT   NAME          CLUSTER AUTHINFO           NAMESPACE
*         white         white   dazwilkin
          black         black   dazwilkin

NB You’ll need to use whatever values of NAME are defined in your config.

You may:

kubectl config use-context black
Switched to context "black"
kubectl config use-context white
Switched to context "white"

And, if you want to make life simplier for yourself:

alias black="kubectl config use-context black"
black
Switch to context "black"

Conclusion

kubectl with Kubernetes Engine leverages Google Cloud Platform’s gcloud CLI to facilitate auth against clusters. This post summarizes how to switch between auth’d clusters without using gcloud repeatedly to get credentials.

Along the way, the post provides some handle-with-care instructions for simplifying a kubectl configuration, and a summary of how kubectl and gcloud manage configuration and where the data is stored.

Feedback always welcome!

That’s all.