Bash hacks gcloud, kubectl, jq etc.

An emporium of bash tricks for optimizing these CLIs

Daz Wilkin
Nov 29, 2017 · 7 min read

I’ve uncovered and been presented with some most-excellent bash and other CLI hacks for optimizing my experience with gcloud and kubectl. I’m going to list them here in the hopes that others find them useful.

This page is for me (and folks like me) who know there must be a CLI “way” but have not seen one, or forget the correct solution through infrequent use.

This will be an ongoing post… I’ll update whenever I use a command and think it may be of interest.

Bash

I used this recently as a way to pick an endpoint at random to issue curl commands against them:

https://stackoverflow.com/a/2388555

So, this is most useful when the PATHS aren’t as easily generated but, for purposes of example:

PATHS=("s1" "s2" "s3" "s4")
for TEST in {1..10}
do
SELECTED_PATH=${PATHS[$RANDOM % ${#PATHS[@]}]}
VALUE=$((RANDOM%10))
curl --silent http://echo.jsontest.com/${SELECTED_PATH}/${VALUE}
done

gcloud

List “my” (those I’m a member of…) projects but only the project ID, only those where the projectId includes ‘dazwilkin’ and only those where the projectId begins ‘dazwilkin’:

gcloud projects list \
--format='value(projectId)'

One challenge with format and filtering is that each CLI has its own way. An alternative is to format the output as JSON or YAML and then use a general-purpose tool to post-process the results with grep or jq (and see jqplay):

gcloud projects list --format='value(projectId)' \
| grep '^dazwilkin'

It’s common to want to iterate over gcloud resource IDs or names. With the ability to filter projects and get the Project ID:

for PROJECT in $(gcloud projects list --format='value(projectId)')
do
gcloud iam service-accounts list \
--filter='value(email)' \
--project=${PROJECT}
done

I create projects by the day and by the theme and dispose of them promptly once done. Occasionally I like to leave VMs hanging around for a few days but to ensure they’re all shut down

PROJECT=[[YOUR-PROJECT-ID]]
INSTANCES=$(gcloud compute instances list \
--project=$PROJECT \
--format="value(name)")
gcloud compute instances stop ${INSTANCES} --project=${PROJECT}

The first command grabs the list of instances across the project (all regions and zones) and assigns this to INSTANCES. The second command shuts them all down. You can merge the two commands if you prefer.

My colleague sought a way to filter “members” from IAM policies. Another colleague (Thanks Glenn!) provided a way to do this with gcloud:

gcloud projects get-iam-policy ${PROJECT} \
--flatten=bindings \
--filter=bindings.role:roles/editor \
--format="value(bindings.members)"

NB For an explanation of ‘flatten’, ‘filter’ and ‘format’ see the Cloud SDK documentation here and another colleague’s blog post here.

A quick explanation of what’s going on. “flatten” flag bursts the policy into multiple records, one for each element in the binding array. This corresponds to one per role type. “filter” then isolates the record for “roles/editor”. “format” then returns the binding.members (for this role).

Here’s a way to do this using jq.

This will enumerate the members for $ROLE in $PROJECT:

ROLE="roles/owner" # for example

NB the “raw-output” flag used here strips strings of enclosing quotes.

More complex but useful, this one extends the above filter by filtering the “user” members and returning the member details without the “user:” prefix:

FILTER=".bindings[] | select (.role==\"${ROLE}\") | .members[] | select (. | startswith(\"user:\")) | ltrimstr(\"user:\")"

NB You may, of course, replace “user” with other types of members (“group”, “domain”, “serviceAccount”).

Ironically, no sooner did I learn about the gcloud “flatten” flag, than I had an opportunity to use it again. A customer engineer asked how he could enumerate the Node Images for all clusters for all projects. He wanted to confirm that they’re no longer using the now-deprecated container-vm image.

Here’s one way to do this:

Chatting with one of our customers about overcoming a known limitation of the Cloud Logging (Console) filtering, leads me to the next hack. It’s not currently possible to compare numeric values in Cloud Console Logs because many numeric values are recorded as strings.

4.7 ≥ 11.0 !! Thanks Gabriel

A solution is to use gcloud to grab log output as JSON and then pump it into jq for some magic:

PROJECT="[YOUR-PROJECT-ID]"
ROOT="projects/${PROJECT}/logs"
LOG="${ROOT}/appengine.googleapis.com%2Fnginx.request"

There’s a lot of bash and strings here which makes it more gnarly than we’d like but…. after setting environment variables, the filter string is constructed for, in this case, an App Engine (Flex) app, grabbing only the Nginx request log for today’s (UTC) entries.

Then, jq grabs two keys from the results, the value of httpRequest.requestUrl (which becomes ‘url’) and the value of jsonPayload.latencySeconds (a string called ‘latency’). This resulting projection ({url, latency}) is filtered to remove any latency values that are null and these results are filtered to convert the non-null string to a number and then pull only those values greater than 0.5 second.

Results in:

{
"url": "/favicon.ico",
"latency": "0.613"
}
{
"url": "/",
"latency": "0.583"
}

So, I learned that it *is* possible to use Cloud Logging filters with “Duration” and “Timestamp” values. Even though presented as Strings, it is possible to compare these numerically. A nuance to the above filter is that the request logs entries include HTTP Request metadata as well as the JSON payload data that we used above. The HTTP Request metadata includes a latency String that’s actually a Duration. Confused, yet? Read more here. Because it is a Duration value, we can filter like this:

gcloud logging read  "$FILTER httpRequest.latency > \"0.5s\"" \
--project=$PROJECT \
--format=json

We’re using a different log value (httpRequest.latency rather than jsonPayload.latencySeconds) but, because httpRequest.latency is a treated as a “Duration”, we can lose the jq post-filtering and augment the gcloud filter to include the refinement: httpRequest.latency > “0.5s”.

Another, this time counting App Engine health checks (both flavors: Legacy and Updated) for the given MODULE, VERSION for the time period AFTER to BEFORE:

LOG="{$ROOT}/appengine.googleapis.com%2Fnginx.health_check"

Results in:

XXXX 200,/liveness_check
YYYY 200,/readiness_check

Or:

ZZZZ 200,/_ah/health

kubectl

As mentioned above, switching between CLIs (e.g. gcloud and kubectl) means switching between filtering and format ways. kubectl uses ‘output’ to achieve both filtering and formatting. I find kubectl output easier to filter and format than gcloud but it’s still mostly trial-and-error. I start by outputting the command as JSON:

kubectl get pods --format=json

The API is nicely consistent and multi-valued results are always in an ‘items’ array and many (all?) have a ‘metadata’ section and a ‘name’. Unfortunately, the query language used by jq is not JSONPath and so conversion is needed. kubectl also support Golang’s template package but this appears used less frequently recently and JSONPath is more common.

kubectl get pods \
--format=json \
| jq .items[].metadata.name

A common (because it’s very useful) pattern that you see with Kubernetes is nesting kubectl commands with the inner command filtering|formatting output for the outer command, viz:

kubectl port-forward $(\
kubectl get pod \
--selector=app=prometheus \
--namespace=istio-system \
--output=jsonpath='{.items[0].metadata.name}'\
) \
--namespace=istio-system \
9090:9090 &

Kubectl has a useful port forwarding command that requires a port mapping (here 9090:9090) and the name of the pod. In this case, to determine the name of the pod, we need to identify the pod that’s labeled (app:prometheus). There should only be 1 pod matching the criteria so the items[0] is mostly turn the result array (with one item) into a singular result.

Another common pattern with kubectl commands is wanting to grab the IP address of a load-balancer for a known service:

SERVICE=[YOUR-SERVICE]

I learned today that there’s a neat ‘custom columns’ format too. So, I’m playing with Brendan Burns’ clever Metaparticle project and, using it, I deployed a service comprising 15 pods running on a regional-cluster. One problem is that the default output=wide format is too wide, yields the ‘node’ name that I’m seeking but includes some columns that I don’t need. I want the pod name and its node’s name, simple:

kubectl get pods \
--output=custom-columns=Name:.metadata.name,Node:.spec.nodeName

As with most of these commands, unless you’re very familiar with the output structure, creating what you need is a process of trial. In this case, I called the command first using the generic YAML output, identified the columns I wanted and then referenced these in the custom-columns spec:

kubectl get pods --output=yaml

You create a service and expose it (beyond the cluster) as a NodePort. Possibly because you’re going to reference the NodePort through an L7 Load-balancer. You may need to also configure firewall rules to expose the cluster’s nodes which now all expose this port. How to quickly test that it’s working as expected? First, determine the NodePort:

SERVICE=[YOUR-SERVICE-NAME]

Then identify one of the cluster’s nodes. All the cluster’s nodes expose the NodePort so any of them is acceptable. Let’s grab one at random. The kubectl command lists the cluster’s nodes’ names (prefixed with “node/”). Cut the prefixing “node/” from the result, sort the results randomly and grab the first:

INSTANCE=$(kubectl get nodes --output=name \
| cut --delimiter="/" --fields=2 \
| sort --random-sort \
| head --lines=1)

Now we combine everything and use gcloud to ssh into the randomly chosen cluster node and port-forward the NodePort to localhost.

PROJECT=[YOUR-PROJECT-ID]

Then, depending on what’s backing the NodePort, you can access it from your localhost. If it were a simple HTTP service, then you may now:

curl localhost:${NODEPORT}

Feedback is *always* welcome.

More to come…

Google Cloud - Community

Google Cloud community articles and blogs

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.

Daz Wilkin

Written by

Google Cloud - Community

A collection of technical articles and blogs published or curated by Google Cloud Developer Advocates. The views expressed are those of the authors and don't necessarily reflect those of Google.