Kubernetes container debugging

There’s a raft of container guidance (mostly ignored for convenience) that applies equally to Kubernetes. I need to do a better job in not running containers as root, but I do a decent job (particularly with Golang) in running (dumb-init and) one process per container.

However, when I’m developing Deployments, these frequently don’t work first time. How then to debug the issues?

Hang on, if it’s a container that works locally isn’t there some assurance that it will also work everywhere? Yes, but…No container is an island…Or… It worked until I tried to divide by zero…

The issue is that running containers (like functions) depend on runtime configuration (data) and this is presented differently between environments.

In this case, the issue is that, for the container in question, volume references that include symbolic links do not (currently) work. This wasn’t surfaced when deploying the container locally under Docker because a simple host volume mount of file-files was used.

When deployed to Kubernetes, using volume mounts of Secrets (containing the wallet files), the issue was surfaced. This is because Secrets (and ConfigMaps) presents files using symbolic links. Secrets or ConfigMaps provide a best-practice for presenting file-based content to containers (Pods) on Kubernetes.

Walk-through

I’m going to use a recent example deploying Ethereum to Kubernetes. This is a practical example that also includes a 3rd-party container (with I believe an underlying issue).

It took me some time to identify the issue because, running the container locally (!) did not present a problem.

The commands are provided to show ‘working’ and help explain the problem. You need not run these commands yourself.

mkdir -p /tmp/keystore

This will work:

docker run \
--rm \
--interactive \
--tty \
--volume=/tmp/keystore:/keystore \
ethereum/client-go \
  --rinkeby
  --keystore=/keystore
  console

It runs geth — the definitive Ethereum Golang implementation — as test node on the ‘rinkeby’ network *using* /keystore to store wallet file(s). The host directory /tmp/keystore is mapped on the container as /keystore.

The container produces lots of logging but there’ll be a console in there somewhere and we can create a new Ethereum account persisted as a wallet file in the /keystore directory:

> personal
{
  listAccounts: [],
  listWallets: [],...
> personal.newAccount("some-random-password")
"0x208507fd69e7ccf2af33f1ddc55e2738751c9aad"
> personal
{
  listAccounts: ["0x208507fd69e7ccf2af33f1ddc55e2738751c9aad"],
  listWallets: [{
      accounts: [{...}],
      status: "Locked",
      url: "keystore:///keystore/UTC--2018-05-25T16-32-28.467584997Z--208507fd69e7ccf2af33f1ddc55e2738751c9aad"
  }],

NB The Ethereum account begins 0x2085. It is stored in a Wallet file of the form UTC — [[data-created]] — [[account].

Exiting the container, the wallet is now created in /tmp/keystore/. More importantly, because this is persisted on the host filing system, if I rerun the container with the same volume mounts, the account remains:

ls /tmp/keystore/
UTC--2018-05-25T16-32-28.467584997Z--208507fd69e7ccf2af33f1ddc55e2738751c9aad

OK. Straightforward and problem-free thus far.

Now I’m going to show you the error as the repro. I discovered the error only after I deployed this working configuration to Kubernetes. It then took me some time to realize the problem, that it wasn’t a Kubernetes issue and create this repro without Kubernetes.

The following creates a symbolic link (with the same-name) in the directory /tmp/lnkeystore to the underlying file in the /tmp/keystore directory. Apart from the different directory names, the only difference between the 2 wallet files is that we’ve created a file that’s a symbolic link to the known-working wallet file:

ACCOUNT="208507fd69e7ccf2af33f1ddc55e2738751c9aad"
WALLET="UTC--2018-05-25T16-32-28.467584997Z--${ACCOUNT}"
mkdir -p /tmp/lnkeystore
ln -s \
  /tmp/keystore/${WALLET} \
  /tmp/lnkeystore

But, if I now run the container using the /tmp/lnkeystore instead of the /tmp/keystore, the account will not be present in the container. This is because of the error which appears to be that, if the wallet file is a symbolic link, the container doesn’t recognize it:

docker run \
--rm \
--interactive \
--tty \--volume=/tmp/lnkeystore:/keystore \
ethereum/client-go \
  --rinkeby
  --keystore=/keystore
  console

NB The only change is in — volume=/tmp/lnkeystore.

And, if I run personal in the container, I expect the account (and wallet) to exist but they aren’t seen by the container:

> personal
{
  listAccounts: [],
  listWallets: [],

Kubernetes debugging

So, how did I discover this issue in Kubernetes and how did I debug it?

I discovered the issue as described above: if the Ethereum wallet file(s) is a symbolic link, the account is not discovered. After deploying to Kubernetes, I expected to be able to reuse an existing account and wallet. But, this didn’t work. the deployment did not find my account (wallet).

Using a 3rd-party container image (ethereum/client-go) discourages (but does not stop) using or changing the container image to debug|fix issues. Changing the container image to debug is a common approach and is helpful. In this case, the image uses Alpine and Alpine provides a minimal operating system that we could use to investigate the problem. I’ll show that approach briefly.

Here’s the profile for ethereum/client-go. You can see that the conveniently published Dockerfile shows that the runtime image is built from Alpine:

https://hub.docker.com/r/ethereum/client-go/~/dockerfile/

However, it’s a good practice to *not* include an operating system (not even one as constrained as Alpine) and to only include the process we intend to run (perhaps coupled with e.g. dump-init). What can we do in this case? There’s no formal solution to this problem but there’s a hack that works well. I’ll show that approach too.

Containers that include debug tools

If the container you’d like to debug includes a shell and some debugging tools, the process is easier. If it’s easier why don’t all images include these?

The principle of doing one thing well encourages us to not include components that we *may* need and focus on the component(s) that we must have. More components also results in a broader attack surface and requires more maintenance.

First, determine what shell and tools you have in your container. If you created the image, you will know this. If you’re using a 3rd-party image, often the authors document the companion Dockerfile.

In the example we’re using here, we know (from above) that we have Alpine. Alpine’s shell is called ash and it includes a bunch of Linux command-line tools that we can use.

First, identify your Pod. In Kubernetes Pods aggregate containers. Much (!) of the time, Pods run single containers. The following applies to my Ethereum Deployment but I’ll explain as I go.

I have a Namespace called “ethereum” and I have a single Deployment that results in a single Pod running a container called “ethereum”:

kubectl get pods --namespace=ethereum
NAME                        READY     STATUS    RESTARTS   AGE
ethereum-7f967cbbb4-ch22q   2/2       Running   0          9m

We need the (Pod) name: ethereum-7f967cbbb4-ch22q

I know (because I created it) that the Pod contains one container and the container’s name is (also) ethereum. We know that it’s running ash and so we can:

kubectl exec ethereum-7f967cbbb4-ch22q \
--stdin \
--tty \
--container=ethereum \
--namespace=ethereum \
-- ash

And this returns a shell prompt and so — as below — can for example list the content of the volume mount called /keystore:

/ #
/ # ls /keystore
UTC--2018-05-25T16-32-28.467584997Z--208507fd69e7ccf2af33f1ddc55e2738751c9aad

And, we know that we should have a geth process running in this container too:

/ # ps aux
PID   USER     TIME   COMMAND
    1 root       0:47 geth --rinkeby --keystore=/cache ...
   23 root       0:00 ash
   30 root       0:00 ps aux

So, this is useful.

And, still within the shell, you can get a console against that geth process with the following:

/ # geth attach http://localhost:8545 console
Welcome to the Geth JavaScript console!
instance: Geth/v1.8.9-unstable-d6ed2f67/linux-amd64/go1.10.2
coinbase: 0x3df64eff7ae6bde185de128a97bc0a9539427453
at block: 2346567 (Fri, 25 May 2018 17:16:13 UTC)
 modules: eth:1.0 net:1.0 personal:1.0 rpc:1.0 web3:1.0
> personal
{
  listAccounts: [],
  listWallets: [],...

There’s our error :-(

After fixing the issue, the same behavior will return:

/ # geth attach http://localhost:8545 console
Welcome to the Geth JavaScript console!
instance: Geth/v1.8.9-unstable-d6ed2f67/linux-amd64/go1.10.2
coinbase: 0x3df64eff7ae6bde185de128a97bc0a9539427453
at block: 2346567 (Fri, 25 May 2018 17:16:13 UTC)
 modules: eth:1.0 net:1.0 personal:1.0 rpc:1.0 web3:1.0
> personal
{
  listAccounts: ["0x208507fd69e7ccf2af33f1ddc55e2738751c9aad"],
  listWallets: [{
      accounts: [{...}],
      status: "Locked",
      url: "keystore:///cache/UTC--2018-05-25T16-32-28.467584997Z--208507fd69e7ccf2af33f1ddc55e2738751c9aad"
  }],
  deriveAccount: function(),

NB listWallets has a property url that includes a path that includes the \cache trick.

If your containers have shells and/or debug tooling, this approach works just fine.

But, it should become decreasing common because it breaks one (or more) tenet(s) of good container image design. So…

Containers that don’t include debug tools

This approach permits us to keep our containers minimal but to run sidecar debugging containers if we need them. I think that’s a good compromise.

To be clear, I recommend that you *don’t* run sidecar containers all the time. This may become a thing but — as-is — this falls into the trap outlined above. The hack (!) is to *only* deploy the debug container when you need it.

But with Kubernetes, this is trickier because you need to inject (!?) the Debug container into a Pod alongside the container you wish to debug.

Deployments to the rescue.

You should be using Deployments to deploy resources to Kubernetes. You should be using spec files to describe these Deployments. You can *always* create a spec file for any Kubernetes resource if you need to:

kubectl get [RESOURCE]/[NAME] --namespace=[NAMESPACE] --output=yaml

So, given a Deployment of the form:

kind: Deployment
...
spec:
  containers:
  - name: x
    ...
  - name: y
    ...

You could (!) edit it to look like this and then Apply it:

kind: Deployment
...
spec:
  containers:
  - name: x
    ...
  - name: y
    ...
  - name: debug
    image: alpine
    command: ["ash","-c","[[MY DEBUGGING COMMANDS]]"]

NB You’re not limited to use alpine. Because you’re designating the container image, you may use which image you wish: ubuntu, busybox, most-excellent-debugging — tools, etc.

What does this do? It adds another container to the Pod called debug that runs some predefined debugging commands|script.

If you don’t know in advance what debugging commands you want, you can use a second trick which is to hold the debug container busy-doing-nothing:

kind: Deployment
...
spec:
  containers:
  - name: x
    ...
  - name: y
    ...
  - name: debug
    image: alpine
    command: ["ash","-c","while true; do sleep 60s; done"]

This avoids the debug container terminating which causes Kubernetes anxiety as it perpetually tries to recreate the container ;-) It results in a Pod that contains a container called debug that you may kubectl exec into to debug.

You can see debug running in my Ethereum Deployment here. At the bottom of the screenshot under “Containers” is a container called “debug” from the “alpine” image:

You may access this container as before:

kubectl exec ethereum-7f967cbbb4-ch22q \
--stdin \
--tty \
--container=ethereum \
--namespace=ethereum \
-- ash

If — as in my case — you need to access the volume mounts that are being used by your container-being-debugged, you simply replicate the volumeMounts on the debug container:

containers:
- name: debug
  image: alpine
  command: ["ash","-c","while true; do sleep 60s; done"]
  volumeMounts:
  - name: keystore
    mountPath: /keystore
  - name: cache
    mountPath: /cache      
- name: ethereum
  image: ethereum/client-go
  args: [...]
   volumeMounts:
   - name: datadir
     mountPath: /datadir
   - name: keystore
     mountPath: /keystore
   - name: cache
     mountPath: /cache
volumes:
- name: datadir
  persistentVolumeClaim:
  claimName: datadir
- name: keystore
  secret:
    secretName: keystore
- name: cach
  emptyDir: {}

Conceptually Kubernetes Pod is like localhost. And, any volumes that are defined for the Pod (as in this case) are accessible to be mounted into any of the its containers. Neat! So this works perfectly for us.

When you’re done debugging the issue, revert the Deployment spec by removing the debug container and re-apply the Deployment to your cluster.

Here’s the (new) Pod. Kubernetes deleted the previous Pod with the new Deployment. This no longer containers the debug container:

Resolving the Issue

Completing the debugging example, I was confounded by the Deployment not working. I quickly determined that I was unable to access the Ethereum account (and wallet) when deployed to Kubernetes using Secrets.

I tried using a ConfigMap instead of a Secret. Same issue. This is because ConfigMaps and Secrets both present volume-mounted files as symbolic links. They do this to provide the dynamism that is required by Kubernetes. So there’s no avoiding the use of links.

The last resort was to create a Persistent Disk and attach this “old school” to the Pod. This is a poor choice for many reasons. First, it requires manual creation, formatting, mounting etc. of the disk. Second, Persistent Disk is only now becoming Regional (and then only 2 replicas; has been Zonal) and this reduces Kubernetes flexibility — particularly with Regional Clusters — in binding a zone-based disk to a region-floating Pod.

The Peristent Disk works :-(

It works because the wallet files I copied into it aren’t symbolic links.

Then I created the hack-trick that I’ve added to my original post. Kubernetes supports a dynamic volume type called emptyDir. What — I wondered — would happen if I were to copy the dynamically managed account (wallet) *from* the Secret into an emptyDir? Would I again have a file-file (not a link)? This works!

So, it’s a hack and a work-around but I now use an init container to copy the wallet file from the Secret into an emptyDir *before* the Ethereum container starts. The Ethereum container is configured to look for the wallet in the emtpyDir volume rather than the Secret volume.

All’s good-ish.

Aside: Container-Optimized OS

Google’s Container-Optimized OS (COS) employs a similar principle. COS is a minimal operating system (!). One challenge this creates is that, if you deploy a bare-bones container to COS, you don’t have any tools available to help with debugging.

COS provides CoreOS toolbox:

https://cloud.google.com/container-optimized-os/docs/how-to/toolbox

CoreOS does not publish a container image for toolbox but Google does provide an image (via Google Container Registry [GCR]) and it also includes the Google Cloud SDK (aka gcloud).

Google’s CoreOS toolbox image is: gcr.io/google-containers/toolbox

You may — if you’d like — leverage CoreOS toolbox in your Kubernetes debugging described previously, simply reference the GCR image and swap ash for bash:

kind: Deployment
...
spec:
  containers:
  - name: x
    ...
  - name: y
    ...
  - name: debug
    image: gcr.io/google-containers/toolbox
    command: ["bash","-c","while true; do sleep 60s; done"]

Feedback always welcome.

That’s all!