Effectively Managing Kubernetes Access from the Terminal
Customizing Your Terminal With Kubectx and Kube-ps1
There are lots of ways to manage access to a Kubernetes cluster. None of them are difficult, but what if you have to manage multiple users and contexts on a single cluster? That becomes a little challenging. To add to that, I’m sure many of us are managing multiple clusters, and some of us probably multiple users and contexts on multiple clusters. Managing all of this can become quite cumbersome and really eat into our time.
I’ll be covering how I personally decided to give myself a better experience managing my Kubernetes clusters. I’ll cover the highs, the lows, and everything in between, all based on my personal preferences.
Research, Break, Fix, Repeat
In my work at Capital One, I interact with around three clusters and potentially three users and contexts per cluster on a daily basis. Eventually changing environment variables and running long kubectl commands ate away at my patience. Naturally, being the sysadmin that I am, I started to write some simple bash functions to put in my .bashrc. This worked for a while, but it wasn’t perfect. I had to run multiple commands to change kubeconfigs and contexts/users. This satisfied me for a little bit, but I wanted a better solution since it still struck a nerve every time it didn’t quite work as expected.
Eventually I told myself I was going to spend an afternoon (ended up being more like a day+) to put a good solution in place for my needs. I mean, if I spent who knows how long writing Ansible playbooks to manage all my dot files and package installs, this should be easy right?
Spoiler… it wasn’t.
After researching tools to manage Kubernetes access I decided on kubectx; it seemed to fit my needs. It manages namespace switching, is brew installable, and has 4k+ GitHub stars. As I was going through the README, like any good user of somebody else’s project, I noticed a reference to another tool which I quickly decided to also implement: kube-ps1. This is a nifty little application that allows you to add your Kubernetes cluster and namespace information to your PS1 line (your terminal command prompt). This has significantly fewer stars, but I’m a sucker for tinkering with my terminal, happy day for me.
Now I needed to figure out what to do about my actual access problems. I have three clusters, three users per cluster, and two different authentication methods. Two of the users are not named uniquely between clusters, but the clusters are named uniquely. Mapping credentials to users should have been straight forward.
Spoiler… it wasn’t.
I first started down the official path looking at the Kubernetes docs for managing multiple clusters. This got me probably 80% of the way there, but it didn’t have an answer for identically named users across clusters. I kept breaking my kubeconfig because the context didn’t know where to look for the user authentication.
Frustration ensued. Furious Google searching ensued. Failure set in.
As I started reaching the end of my line and going down with the ship, about to call my attempt a failure and resign myself to disappointment, I pulled out a last ditch effort and asked in the MagicCityTech#kubernetes Slack channel (my Birmingham buddies) if anyone had figured this out. I got one reply, the same guy that answers all the Kubernetes questions, and it’s a link which solved everything for me.
Getting it All to Work
The first thing I did was retrieve all my kubeconfigs and service account tokens out of my gopass repository; if you don’t have gopass you’re seriously missing out. From there I started building out my “master” kubeconfig, which I would later use for all of my accesses. It ended up looking a little something like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <data>
server: <server>
name: dev-ue1
- cluster:
certificate-authority-data: <data>
server: <server>
name: qa-ue1
- cluster:
certificate-authority-data: <data>
server: <server>
name: qa-uw2
contexts:
- context:
cluster: dev-ue1
namespace: kube-system
user: dev-ue1-admin
name: dev-ue1-admin
- context:
cluster: dev-ue1
namespace: admin-services
user: dev-ue1-admin-services
name: dev-ue1-admin-services
- context:
cluster: dev-ue1
namespace: user
user: dev-ue1-user
name: dev-ue1-user
- context:
cluster: qa-ue1
namespace: kube-system
user: qa-ue1-admin
name: qa-ue1-admin
- context:
cluster: qa-ue1
namespace: admin-services
user: qa-ue1-admin-services
name: qa-ue1-admin-services
- context:
cluster: qa-uw2
namespace: kube-system
user: qa-uw2-admin
name: qa-uw2-admin
- context:
cluster: qa-uw2
namespace: admin-services
user: qa-uw2-admin-services
name: qa-uw2-admin-services
kind: Config
preferences: {}
users:
- name: dev-ue1-admin
user:
client-certificate-data: <client certificate>
client-key-data: <client key>
- name: dev-ue1-admin-services
user:
token: <token>
- name: dev-ue1-user
user:
token: <token>
- name: qa-ue1-admin
user:
client-certificate-data: <client certificate>
client-key-data: <client key>
- name: qa-ue1-admin-services
user:
token: <token>
- name: qa-uw2-admin
user:
client-certificate-data: <client certificate>
client-key-data: <client key>
- name: qa-uw2-admin-services
user:
token: <token>This kubeconfig allows me to use all of my contexts and users across all of my clusters via a single configuration file. This also allows me to access each of the clusters using the non-uniquely named service account admin-services. I can do that because in the kubeconfig I have uniquely named all of my users in my context. I can then look up the specific access token per user and use the certificate-authority-data for the specific cluster to authenticate. Without having the certificate-authority-data in the cluster block, this will not work. Now that the hard and complicated part is done, and stored securely in gopass, I can move on to actually making it useable.
The first thing I did was add a line to my .bashrc to export the KUBECONFIG environment variable to my newly created kubeconfig. I then brew installed kubectx and kube-ps1, and started tweaking my PS1 line and how things were displayed. Naturally, the stock kube-ps1 output wasn’t what I wanted, so it was time to start modifying. I should note that if you’re already using a tool to manage your PS1 line you might need to do something a little different to get this working.
My PS1 line was already pretty long; I have helpers for git status and branches, as well as Terraform workspaces. So when I saw that kube-ps1 displayed my entire context and namespace it just wasn’t going to work. Luckily, I’m a tmux user (thanks again to MagicCityTech for getting me on the tmux train) and have access to a nifty little status line at the bottom of my terminal. I decided I want to display the cluster and user in my tmux status line, and only the namespace in my PS1 line.
The custom config ensues.
First, I turned off the silly Kubernetes symbol kube-ps1 uses, removed the prefix and suffix, and removed the namespace text color configuration (I have my own color preferences). This was all easily done by reading through the README and setting some environment variables in my .bashrc.
KUBE_PS1_SYMBOL_ENABLE=false
KUBE_PS1_PREFIX=""
KUBE_PS1_SUFFIX=""
KUBE_PS1_NS_COLOR=""export \
KUBE_PS1_SYMBOL_ENABLE \
KUBE_PS1_PREFIX \
KUBE_PS1_SUFFIX \
KUBE_PS1_NS_COLOR
Next I needed to write a custom function to get only the information I wanted and set my colors. In my .bashrc I wrote a simple function and executed it as part of my PS1 line.
kube_ns() {
purple='\033[0;35m'
nocolor='\033[0m'
kube_ps1=$(kube_ps1)
namespace=$(echo ${kube_ps1} | cut -d ':' -f 2)
echo -e "${nocolor}(${purple}${namespace}${nocolor})"
}PS1="\$(kube_ps1) \[\033[32m\]\u\[\033[0m\] \[\033[36m\]\w\[\033[0m\]\$(git_prompt_info '\[\033[34m\]%b\[\033[0m\]') \$(terraform_workspace_info '\[\033[95m\]%b\[\033[0m\]')\n❯ "
Now that I had that in place, and a visually appealing terminal, I tried using kubectx.
Spoiler… it worked.
You can see that kube-ps1 is populating the (namespace) part of my command prompt which is exactly what I want. You can also see how easy it is to switch, not only context, but also clusters, with a single command.
kubectx also lets you change namespaces using kubens. This command will list all the namespaces and then change to whichever one you specify.
Notice that my command prompt updates when I change namespaces but my context remains the same.
Remember when I said I was a tmux user? Well that’s additional configuration which was way simpler. In my .tmux.conf file I had to configure my tmux environment to know about my KUBECONFIG env var and my status-left-length and status-left; I use the right side to display the date. I keep all my configs in ${HOME}/.config.
set-environment -g KUBECONFIG ${HOME}/.config/kube-master.configset-option -g status-left-length 100
set-option -g status-left "#[fg=red]#(kubectl config current-context | cut -d '-' -f 1–2)#[fg=white]#(echo :)#[fg=blue]#(kubectl config current-context | cut -d '-' -f 3–4)"
Adjust your cut command as needed based on your context naming convention.
Those three lines give me the exact information I want in the bottom left corner of my terminal. It also will update as you change clusters/contexts/users with kubectx, just as you would expect.
Was It Worth It
As for whether you should you do this, the answer is up to you and depends on how much you want to invest in your local environment. This wasn’t required to do my job well, but it did allow me do it more efficiently. I invest a lot into my local config because I enjoy having a finely tuned development environment that fits all of my needs. If you are the same way, then this is something that might be for you as well.
Related:
DISCLOSURE STATEMENT: © 2019 Capital One. Opinions are those of the individual author. Unless noted otherwise in this post, Capital One is not affiliated with, nor endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are property of their respective owners.
