Debugging Go application inside Kubernetes from IDE

After writing How to develop Go gRPC microservices and deploy in Kubernetes, I received many queries about how to debug applications inside Kubernetes.

Deploying container, service or secret configuration is super easy in Kubernetes. But what about debugging? As a developer, it is always useful to be able to debug application with your favorite IDE.

Problem

When your application only works with Kubernetes API, you can simply launch your application from IDE and connect it to the remote Kubernetes API. But when your application needs to connect to other systems which are only available inside the Kubernetes cluster then this solution will not work.

Repository

For this article I will use kubernetes-go-grpc repository. You can find more details and deployment instructions in this article How to develop Go gRPC microservices and deploy in Kubernetes. As this article is focused on debugging solution, I am not going to describe about the project.

Solution

At the time generating binary go compiler can change the sequence of operations, add code, remove code or apply transformations. So, it is harder to map Go line to the optimized output. But during debugging you need to map or stop execution at a specific Go line. For this reason, you need to generate binary without optimization. You can disable optimization by passing flag to the compiler using gcflags. Execute below command to generate binary without optimization

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -gcflags "all=-N -l" -o ./app

Here -N will disable optimization and -l disable inlining. To remote you need to start a headless Delve server on the machine.

dlv --listen=:40000 --headless=true --api-version=2 --accept-multiclient exec ./app

Then connect to that server from your IDE (in this case Goalnd).

Dockerfile

To containerize your application use below dockerfile named as Dockerfile.debug

FROM golang

RUN go get -u github.com/go-delve/delve/cmd/dlv
RUN mkdir app
WORKDIR /app
COPY app .
EXPOSE 40000
EXPOSE 8080

ENTRYPOINT ["/go/bin/dlv", "--listen=:40000", "--headless=true", "--api-version=2", "exec", "./app"]

Here download Delve for debugging, copy your debug enable binary, expose 8080 port as api is running on this port and 40000 port as Delve server will start on this port. To build your docker image execute

docker build -f Dockerfile.debug -t api-debug .

To use your local docker repository and deploy in kubernetes execute below command

eval $(minikube docker-env) \
&& kubectl run --rm -i debug-api-deploy --image=api-debug --image-pull-policy=Never

To know your pod name execute

kubectl get pods

and find your api pod name. You pod name should be something like debug-api-deploy-867bd55f4d-gzh5c. Now you need to forward port 8080 and 40000 so that you can connect your pod remotely.

kubectl port-forward <YOUR_POD_NAME> 40000:40000
kubectl port-forward <YOUR_POD_NAME> 8080:8080

Configure the IDE

In this article you will use Goland but it will work similarly with other IDE. You need to create a Remote Debug and point it to localhost:40000 since you have activated the port forward.

From that point, you are able to set Breakpoint in our local IDE, and it will communicate with the Delvedebugger deployed in the Kubernetes cluster to allows to debug your application.

Finally

The application will wait for your IDE to connect to it before starting execution of code, that way you are able to start debugging from start. Normally, the Pod will stop in the state completed each time you stop debugging on IDE side or loose connection.

Happy coding …. :)