A tour of Kubernetes focused developer tools — part 1

Debugging applications in K8s with Squash

Photo by Joshua Aragon on Unsplash

I recently had a discussion around Kubernetes developer tooling and the feature set such tools should support for development and (as importantly) debugging. This is a rapidly evolving area I’ve meant to deep dive into for some time, ever since Telepresence became CNCF incubated. Unfortunately even after a year of deploying on Kubernetes in production, our team hadn’t really adopted a nice developer tooling experience; developers still used Helm chart deployments for testing their changes — a side-effect of needing strong governance while sharing a dev cluster. There were no immediate pains, so not a lot of gains in wading through the entire ecosystem, trying to figure out the best fit toolset.

I’d come across Draft of course, and even earlier Forge when I was just getting into K8s (both of them now dead for all intents). And I’d heard of Skaffold and meant to compare it with Helm, etc. I also recently tried out the Okteto tutorials and chanced upon Garden. So I had done some decent toe dipping to test out the water and had lots of bookmarks saved for later but the large variety of options felt a little bit daunting. There are about a 100 blogs on this topic, with a 100 different opinions. And the overlap with deployment models doesn’t help things — should you use YAML templates, Jsonnet, ksonnet, etc.? Sidebar — yaml templating sucks!

Thankfully smarter people have nicely consolidated this stuff as in this excellent talk:

This talk finally got me into the pool (as ideas often do at 2 am), and also lagging well behind in my blog output, I drew up a list of tools to blog about. I intend to delve into 1 tool every couple of days for as long as I could keep it up.

1. Squash — remote debug applications in pods
2. Stern — stream logs from pods/containers
3. ksync — file syncing
4. kubefwd —bulk service port-forwarding
5. gitkube — GitOps with K8s
6. Telepresence — local development against a cluster (a proper CNCF project)
7. Garden — in-cluster builds
8. Okteto — in-cluster builds
9. Tilt (I know nothing about it)
10. Kustomize — I’m aware this is not actually for local development
11. Helm 3 ( just want to get a sense of the new version, which is the default now)
12. Skaffold — popular project from Google, more for deployment workflow
13. Google’s Cloud code plugin for K8s — I’ve had this installed for 6 months and not used it
14. Quarkus — though this stretches the definition of dev tooling
15. AWS Cloud Development Kit for Kubernetes — fairly new, based on AWS Cloud Development Kit (which I know nothing about either)
16. alexellis/arkade just seems to be a fancy CLI
17. Ballerina — long pending on my list

I considered including Metaparticle, but it’s website seems to be down, which doesn’t bode well.

Most of these are not new of course. The way this ecosystem evolves, some of them may be close to going away. Some may be too new and unstable, and others may figuratively suck. Let’s find out! The intent is to try out different ways of deploying and debugging services in a local minikube cluster on MacOS and hopefully form some useful opinions. As a Java or Go developer, which tooling would best augment my standard dev setup?

Onwards!

Squash

Squash Docs

This is from the Solo.io folks who are behind Gloo — an Envoy powered API gateway (and a whole other blog series)

Setup

Installation was easy on MacOS with Homebrew

brew install solo-io/tap/squashctl

I spun up my local minikube cluster

➜ ~ kubectl version
Client Version: version.Info{Major:”1", Minor:”15", GitVersion:”v1.15.5", GitCommit:”20c265fef0741dd71a66480e35bd69f18351daea”, GitTreeState:”clean”, BuildDate:”2019–10–15T19:16:51Z”, GoVersion:”go1.12.10", Compiler:”gc”, Platform:”darwin/amd64"}
Server Version: version.Info{Major:”1", Minor:”12", GitVersion:”v1.12.5", GitCommit:”51dd616cdd25d6ee22c83a858773b607328a18ec”, GitTreeState:”clean”, BuildDate:”2019–01–16T18:14:49Z”, GoVersion:”go1.10.7", Compiler:”gc”, Platform:”linux/amd64"}

You can deploy the standard sample demo bundled in the squash codebase with the simple command

squashctl deploy demo

This deploys sample calculator microservices — 1 for display logic, and 1 for the actual calculation.

solo-io/squash

You get a bunch of easy prompts to deploy the demo services, along with the option to deploy either the Java or Go versions of example-service2.

Squash config file not found. Writing default config to ~/.squash/config.yaml.
? Select a namespace for service 1. default
? Select a namespace for service 2. default
? Choose a demo microservice to deploy go-java
deploying go-java sample microservice
Deployed demo app example-service1 in namespace default
Deployed demo app example-service2-java in namespace default

By default there is a calculator UI service written in Go (example-service1) and an API to do the calculation written in either Go or Java (example-service2). The standard Quick Start (on the Squash home page) asks you to run the Java service.

Demo services deployed and running on minikube

Debugging

By default Squash supports the Go dlv debugger and the Java jdb debugger which can be run from the command line. I tried it out, but really preferred getting it working with editor integration.

First we expose example-service1 so we can interact with the UI.

kubectl port-forward service/example-service1 8080:80
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

For VS Code integration, there is a VS Code Squash extension https://marketplace.visualstudio.com/items?itemName=ilevine.squash

You have to set the path to squashctl in VS Code Settings. You also need to specify a source code mapping in case the service was built and deployed by someone else (needed for remote debugging)

Follow the instructions in the Quick Start to enable debugging of this deployed demo.

Squash Docs

When you start it up, this puts VS Code in Remote mode and as per the instructions “will spin up a debugger Pod on your Kubernetes cluster”

This actually spins up a squash-debugger namespace and runs a pod in it.

squash-debugger namespace and pod

You can then set a break-point in your code and when you hit the endpoint with some traffic (in this case a simple calculation operation), it pauses execution at the debug point set in your VS Code.

Remote debugging Go with dlv in VS Code

As a bonus, this was my first time ever debugging code in VS Code. I’ve tended to use it solely as a fancy editor in the past.

Sidebar — as a relative newbie to Go, it was a WTH moment to me that it’s just that simple to attach a remote debugger to your Go program. You don’t have to build/run the program in separate debug mode like in C++ and Java.

There is a nice technical overview published by Solo at

Squash Docs

Essentially every debug session creates a new ‘Plank’ pod which runs on the same node as the target process and shares the host’s PID namespace, thus allowing it to see and connect to all processes on that node. These ‘Plank’ pods are spawned by the main ‘Squash’ pod. The whole process honors RBAC permissions setup in the cluster.

Squash defines it’s own CRD.

➜ example git:(master) kubectl get crd
NAME CREATED AT
debugattachments.squash.solo.io 2020–03–20T11:16:14Z
➜ example git:(master) cat ~/.squash/config.yaml
# Squash configuration file
# The specification can be found at https://squash.solo.io
secure_mode: false
verbose: true
log_commands: false
createdby: squash-initialization

For multi-user clusters which would be the default usage mode, Squash should run in Secure Mode, which can be bootstrapped via the following command

squashctl deploy squash

The debuggers supported are Java, Go dlv, and gdb, with some legacy support for Python and Nodejs at some point, which seems to have been in a process of evolution, but I couldn’t find any latest updates on the same.

So the Go debugging worked. Not being a primarily VS Code user I next turned my attention to Java. For Java applications, a basic requirement is to start the Java process inside the container with remote debugger enabled. Once you start your application in debug mode, the rest of the things proceed as usual.

Current Status

As of writing this, the last released version was v0.5.19 released in Sep 2019 with the initial release sometime in 2017. The project doesn’t appear to be dead, but certainly seems to be coasting along with the latest commit sometime in the last couple of months. They were supposed to get a lot of new support out in 2019 but it doesn’t appear to have happened.

It helps that it has a fairly restrictive use case — simply get remote debugging working on a cluster pod, which it more or less seems to have achieved a while back.

As a JVM developer for me the biggest restriction is the lack of Intellij support. I love VS Code, but it’s never going to be my primary IDE. There was some work done on Intellij integration but it seems to have been abandoned.

solo-io/squash-intellij

There was some support for Envoy via a specific Squash filter, and there was a roadmap planned for Istio but it seems to have gone the same way.

Squash - envoy tag-v1.10.0 documentation

Overall though, if I simply had to debug my Go service running in a dev cluster, I would definitely consider Squash. It’s nice and easy to get going, and has a fairly low learning curve — I went through the entire documentation in ~2 hrs and everything just worked. With one caveat — command line debugging just failed for me. If I had a dev team struggling with debugging issues, I might strongly consider setting them up with Squash on a shared dev cluster — right after I finished convincing them to abandon Intellij and switch over to VSCode.

I know I decided to deploy an existing project from scratch, but with this it didn’t really feel necessary. I mostly got the gist of it, and if needed can figure out how to use it and how it works for my own applications.

Next time — something much simpler for pod log tracing.

wercker/stern