How to setup an awesome shell experience for IBM Cloud Kubernetes Service

When I work with IBM Cloud Kubernetes Service (IKS), I connect to one of my clusters by downloading the cluster configuration and setting an environment variable in my terminal that is used by kubectl to authenticate to my cluster. This approach is useful because I can connect to multiple IKS clusters at the same time in separate terminal windows. However, when working with a single cluster, I’d like to have the ability to set the current context to a single cluster and have all shell windows use the same context. I would also like the ability to quickly switch between clusters across multiple shell windows. In this post I will describe the steps to configure a shell environment to dynamically set an IKS cluster context for use with the Kubernetes CLI.

UPDATED 05/10/2019 — The download script has been updated to utilize the new IKS global endpoint feature.

Setup dependent CLI tools

I use zsh as my terminal on mac. I will describe the steps from the point of view of zsh. You can adapt the steps below to other shells or you may install zsh as well (if you haven’t already). I recommend that you use Oh My Zsh. On mac you can simply execute the following in the terminal to install zsh.

brew update
brew install zsh zsh-completions

You will need to install both the IKS and Kubernetes CLI tools. The steps for installing these tools can be found within the IKS on-line documentation here.

The rest of the steps below assume that you have created one or more IKS clusters. If you have not then follow the these instructions to create an IKS cluster.

Download IKS cluster configurations

In order to use kubectl with an IKS kube cluster, you must download the kube configuration. The kube configuration will authenticate and authorize based on the user identity when accessing the clusters via kubectl.

From a zsh shell window, log into your IBM Cloud account. Enter the following command and follow the interactive prompts to log into IBM Cloud.

ibmcloud login

Once you are logged into IBM Cloud, you can now download your cluster configurations using the following commands.

Set the region where you have created your cluster(s). Execute the following command and select the region when prompted.

ibmcloud ks region-set

List your provisioned clusters.

ibmcloud ks clusters

Download each cluster’s configuration.

ibmcloud ks cluster-config <cluster-name>

Note, the cluster configuration is downloaded as a sub-directory with the same names as your cluster within the ~/.bluemix/plugins/container-service/clusters directory. If you delete a cluster, you will need to go to this directory and delete the corresponding configuration folder.

UPDATED — 05/10/2019

You will need to perform the steps above for each cluster within each region. Alternatively, you can execute this bash script that will loop through all of your clusters and download their configuration files in one step. You can pass in a -d option to the script to clean up (delete) existing config files to ensure orphaned cluster configs are removed.
Note, you can only execute this script within a shell session that you have logged into IBM Cloud.

You can execute the gist file as a single shell command if you wish.

bash <(curl -s https://gist.githubusercontent.com/dcberg/468818e6e72f769cd607d3f047707cfe/raw/e966a9da6450e0414146a82d3642210226410f18/iks-download-cluster-configs.sh)

Or if you would like to clean-up existing cluster configs in one command, you can pass the -d option.

bash <(curl -s https://gist.githubusercontent.com/dcberg/468818e6e72f769cd607d3f047707cfe/raw/e966a9da6450e0414146a82d3642210226410f18/iks-download-cluster-configs.sh) -d

Configure merged kube config

We will use the kube merged config approach to expose each of the IKS cluster configurations that were downloaded to kubectl. Download the following shell script to a local directory.

IMPORTANT: Be sure to update the username variable on line 12 to use your IBM ID which was used earlier to log into ibmcloud. This is necessary to unique the identities for each cluster.

To use the merged config approach we will need to set the KUBECONFIG environment variable in each shell window. We will want to automate the setting of the KUBECONFIG environment variable to the merged kube config value. This is easily done by adjusting the zsh configuration file. You can edit the .zshrc file from the terminal or open in your favorite text editor.

vi $HOME/.zshrc

Add the following line to the .zshrc file and save. Be sure to use the directory where you placed the iks-merged-config.sh file that you downloaded earlier.

export KUBECONFIG=$(<CHANGE TO DOWNLOADED DIR>/iks-merged-config.sh)

You will need a local kube config file to ensure that the current context is set in the local config file without disrupting the IKS config files. This allows you to use the merged kube config as well as a single KUBECONFIG environment variable to a specific cluster. This gives you the best of both worlds by having a shell window locked to a single cluster config as well as several shells using a shared cluster context.

If you do not have a local kube config file then you can use the following snippet to create a local config file saved as~/.kube/config.

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://fakeserver
  name: cluster.fake
contexts:
- context:
    cluster: cluster.fake
    namespace: kube-system
    user: admin
  name: cluster.fake-context
current-context: cluster.fake
kind: Config
preferences: {}
users: []

Validate things are working

Now let’s validate that everything is working. First, open a new shell window to validate that the .zshrc file is properly calling the iks-merged-config.sh file to set the KUBECONFIG environment variable to the merged kube config value.

Run the following command to view a merged kube configuration.

kubectl config view

This will output the complete (merged) configuration. You should see all of your IKS cluster contexts in the output.

View all of the contexts. These are the cluster contexts that you can switch between

kubectl config get-contexts

You should see an output similar to the following (obviously you will see different names). The * represents the current context.

CURRENT   NAME                   CLUSTER        AUTHINFO                      NAMESPACE
*         cluster.fake-context   cluster.fake   admin                         kube-system
          dan-icp                dan-icp        dan-icp-danberg@us.ibm.com    default
          dan-wdc7               dan-wdc7       dan-wdc7-danberg@us.ibm.com   default
          dcb1                   dcb1           dcb1-danberg@us.ibm.com       default

Let’s set a different context using one of the names above.

kubectl config use-context <context name>

Now you can connect to the specific cluster and view the kube resources. Let’s confirm by listing the nodes for the cluster.

kubectl get node

You should get a response similar to the following.

NAME           STATUS    ROLES     AGE       VERSION
10.187.24.15   Ready     <none>    5d        v1.9.10+IKS
10.187.24.41   Ready     <none>    73d       v1.9.8-2+af27ab4b096122
10.187.24.5    Ready     <none>    5d        v1.9.10+IKS
10.187.24.6    Ready     <none>    73d       v1.9.8-2+af27ab4b096122
10.187.24.62   Ready     <none>    73d       v1.9.8-2+af27ab4b096122

Open a second shell window and run the same command again. You should get the same results.

kubectl get node

Every new shell will automatically have the KUBECONFIG environment variable set to the merged kube config value and they will share the current context.

Additional awesome shell tools

Here are two more awesome shell tools that are useful when using kubectl with multiple cluster contexts.

Adjust shell to show current cluster and namespace

The first issue I hit when using this new capability to switch kube between kube clusters is that it wasn’t easy to know which cluster and namespace I was using when the shell window was open. This can be remediated by installing the kube-ps1 plugin to zsh.

brew update
brew install kube-ps1

You will need to edit your .zshrc file again.

vi $HOME/.zshrc

Add the following text to the file and save (see the github link above for other install options).

source "/usr/local/opt/kube-ps1/share/kube-ps1.sh"
PS1='$(kube_ps1)'$PS1

Note, the kube-ps1 page has alternative install options if the above steps do not work for you.

When you open a new shell window you will see that the shell prompt is updated to show the current context and namespace as well as a nifty kube icon.

Add tools to set context and namespace

Another great tool is kubectx which is used to easily set the current cluster context and kube namespace.

Install in mac using the following command (other install options available in the link above).

brew update
brew install kubectx

Now you can easily set the current cluster context with the following command.

kubectx <cluster name>

Auto-completion is supported so you don’t need to know the names of all of your clusters (super nice). Changing the cluster context in one shell window changes it in all other shells unless you have locked a shell to one specific kube config file.

To set the current namespace across all shells use the following command. Again auto-completion is enabled which shows the available list of namespaces for the current context.

kubens <namespace>

Conclusion

I hope you enjoy your new shell experience when working with kubernetes clusters and IKS as much as I have. Please let me know if you have your own favorite shell tools for kubernetes.

ENJOY!!!