Simultaneous Command Execution for K8s with kubectl

I was working with Apache Druid and the great team at Imply on a PoC for a company. Part of the PoC, having said Apache Druid, is to have a sub-second query response time for all of the queries.

Knowing Apache Druid for some time now, I know this is not a big problem for this system. It was designed to be able to handle such query latency on top of being infinitely scalable. Let’s just say, Druid is a very capable system.

We continued working on the PoC, loaded ~100B rows into the system which is around 3.5TB of data, and defined the queries we would run the tests with. Lo and behold, most queries ran in sub-second query latency, but there were some outliers that took 2 or even 2.2 seconds.

How to Investigate

Trying to figure out why those specific queries took so long, the people at Imply suggested the following: Why not create flame graphs for further investigation?

My response: “Sure, Absolutely. I’m down… What’s a flame graph?”

This is a Flame Graph

Evidently, there is a utility called Swiss Java Knife (SJK), which is a command line tool for JVM diagnostic, troubleshooting, and profiling. You can read more about what, where, how, and why on their GitHub page, specifically the section on Flame Graphs, and while this is a SUPERCOOL library, it is out of the scope of this post.

Go check it out!!!

The Problem

The Druid cluster used in this PoC has about 15 Nodes that need monitoring and Flame Graph execution. This means the following:

1. I have to copy the sjk jar file to each one of these nodes
2. I need to run a command activating the sjk simultaneously on each of those nodes in order to capture the data while running a druid query.

No biggy. But life has a way of making it more interesting: I cannot use any external tool for synchronized command execution. Don‘t ask why, such is life.

There was no way in hell to do any of this manually, even the copying part. I am way too lazy for something like this (don’t tell my clients).

Turn Lemons Into Lemonade

So I had to figure out a way to do this with nothing but: kubectl.

kubectl supports copying files to and from K8s nodes by way of the “kubectl cp” command. It also supports executing commands on K8s nodes using the “kubectl exec” command. Making these commands perform simultaneously is just a matter of running all commands in the background using the “&” (ampersand) character.

So I’m at the point to prepare a Linux command which is comprised of 15 command lines like this:

kubectl cp ./sjk.jar pod1:/opt/tmp & \
kubectl cp ./sjk.jar pod2:/opt/tmp & \
kubectl cp ./sjk.jar pod3:/opt/tmp & \
...

And executing a command would look like this

kubectl exec pod1 -- bash -c "java -jar /opt/tmp/sjk.jar" & \
kubectl exec pod2 -- bash -c "java -jar /opt/tmp/sjk.jar" & \
kubectl exec pod3 -- bash -c "java -jar /opt/tmp/sjk.jar" & \
...

This works, but it is not efficient. For the following reasons:

• The name of the nodes can change at random
• the source and dest may change
• the command can change

This called for simple bash scripts:

Copying to and from

Copy the following code into a file calling it ‘mkcopy’ and make sure the file is executable.

#/bin/bash

podgrep=$1
src=$2
dest=$3

for pod in $(kubectl get pods -A -o name | grep -E "$podgrep"  | awk -F'/' '{print $2}'); do
    tmp_src=${src/\@pod/$pod}
    tmp_dest=${dest/\@pod/$pod}
    kubectl -n druid cp  ${src} ${dest} &
done

Please note I am using ‘@pod’ as a placeholder for the podname which will be replaced by the script

Using the script will look as follows:

mkcopy 'druid-imply-data' ./sjk.jar @pod:/opt/tmp
mkcopy 'druid-imply-data' @pod:/opt/tmp/results.html ./results_folder

Executing simple commands

Executing simultaneous remote commands would be done with the following script. Copy the code into a file calling it ‘mkrun’ and make sure the file is executable.

#/bin/bash

podgrep=$1
cmd=$2

for pod in $(kubectl get pods -A -o name | grep -E "$podgrep" | awk -F'/' '{print $2}'); do
    kubectl -n druid exec $pod -- bash -c "$cmd" &
done

Using the script will look as follows:

mkrun 'druid-imply-data' 'java -jar sjk.jar'

Conclusion

I certainly did not invent the wheel here, and I am sure that many readers of this post, those who made it this far, will say “of course, what’s new here?”. Well, nothing much. I just thought someone else might enjoy the script I wrote for myself and posting it so it may benefit others.

As I wrote in the past: if I can save someone else some time and effort, I’ve achieved the reason for this post.