Technical introduction to Squash
Powerful debugging capabilities are essential for development of robust complex applications. Strong debugging tools are available for traditional application development, and these are often integrated with modern IDEs, providing an effective development experience. These tools, however, are not available for development of distributed applications, hindering the development of complex microservices apps.
This is why we created Squash. Our goal is to bring the power of modern popular debuggers to developers of microservices apps that run on container orchestration platforms. Squash bridges between the orchestration platform (without changing it) and the IDE. Users are free to choose which containers, pods, services or images they are interested in debugging, and are allowed to set breakpoints in their code, monitor values of their variables on the fly, step through the code while jumping between microservices, and change these values during run time.
Squash is built to be easily extensible, allowing — and encouraging — adding support for more platforms, debuggers and IDEs.
Check out Squash on GitHub.
Debugging with Squash
To show Squash in action, we consider a microservices application running on Kubernetes, and use Squash extension for VSCode.
To get started, we choose Squash: debug container command from the IDE command palette prompt.
Squash then prompts us to choose which running pod we’d like to debug (notice that Squash had already obtained the list of available pods from Kubernetes).
After we choose the relevant pod, Squash provides us with a list of the containers that run in that pod, and waits for us to select the one we would like to debug.
Now that Squash knows what we want to debug, we should also tell it what debugger we’d like to use.
If we wanted to, we could now open a new VS Code window and repeat these steps multiple times, and debug several microservices at once (as done in the demo below).
Once we’re connected to Squash server, the status bar turn orange. We are all set to start debugging!
We can now use all the cool features of the debugger: live debugging, setting breakpoint, stepping through code, viewing the values of variables of interest and modifying them for troubleshooting, and more. We can also jump between microservices together with the application, as seen in this demo:
What happens behind the scenes?
First, some Squash Key concepts:
Squash server holds the information about the breakpoints for each application, orchestrates and controls the squash clients. Squash server deploys and runs on Kubernetes
Squash client deploys as daemon set on Kubernetes node. The client wraps as docker container, and also contains the binary of the debuggers.
Squash User Interface squash uses IDEs as its user interface. After installing the Squash extension, Squash commands are available in the IDE command palette.
Squash set of commands:
- debug container
Extension: When a user chooses this option, the extension communicates with kubectl and presents the user with the running pods. Once the user chooses a pod, the extension presents the containers currently running in the pod. When the user chooses a container, the extension makes a call to Squash server with the information of what to debug and waits for Squash server to supply the port for connecting to the debugger server. Finally, the extension connects to the debug server and transfers control to the native debug extension.
Squash server: When the extension makes an api call to squash server, the server connects to the relevant squash client and sends the information of the pod and container that should be attached to a debugger, and the set of breakpoints. The server then waits for the reply that tells it which port to connect.
Squash client: Upon getting a request from the server, the client communicates with CRI to obtain the container host pid, allowing the debugger to attach. The client then runs the debugger, attaches it to the process in the container, and sets the application breakpoints. Squash client return the debug server address to Squash server to allow the VS Code debugger client to connect directly to the native debug server.
- debug service
Extension: When a user chooses this option, the extension communicates with kubectl and presents the user with the existing services. Once the user chooses a service, the extension presents the current images used in that service.
When the user chooses the image, the extension makes a call to Squash server with the information of what to debug, and waits on Squash server to supply the port to connect to the debugger server. The extension then connects the debug server and transfers control to the native debug extension.
Squash server: When the extension makes an api call to squash server, the server starts to watch the service for the creation of a new container. When a container with the specified image is created, the server notifies the relevant squash client with the pod and container that should be attached to the debugger, and with the set of breakpoints. The server then waits for a reply that tells it which port to connect. The server sends the first reply back to the extension. The server only allows a single active session at a time, and returns an error to clients trying to add more sessions.
Squash client: Upon getting a request from the server, the client communicates with CRI to convert container to host pid, allowing the debugger to attach. The client then runs the debugger, attaches it to the process (container), and sets the application breakpoints.
When a debug event happens (either a breakpoint hit or program crash) the Squash client that manages the container where the event happened returns the debug server address to Squash server, allowing the VS Code debugger to connect. Squash also instruct the others squash clients, that manage other containers in the service, to detach the debuggers.
- resume debug session
Resumes the session between the squash server and the IDE. This command should be used whenever the user that initiated a debug service command had lost the session with Squash server (by closing the IDE or for any other reason), and is interested in resuming the session. An indication that the session had been resumed can be found in the status bar: “⏱ Waiting for Debug Session”
- release debug session
Release the session with the squash server to the IDE. This command is used when the user who initiated a debug service command no longer wants to wait for debug session. This command does not detach the debugger from the process, but releases the session with the squash server. This command can be called from command palette, or simply by clicking the “⏱ Waiting for Debug Session” indication in the status bar.
- stop debug service
The IDE extension instructs Squash server to stop debugging the service and release the session.
- Toggle service breakpoint
When debugging a service with Squash, user can set Squash breakpoints  from the command palette in the IDE.
After initiating the debug service command, Squash client attaches the debugger to all the containers (created by the requested image) in the service, and waits for a debug event to happen. A debug event is either an unmanagement exception or a Squash breakpoint being hit. The Squash client then returns the debug server address to Squash server, and instructs the others squash clients that manage other containers in the service to detach the debuggers. Once the port returns to Squash server and the debugger client connects to the debugger server the IDE breakpoint become active.
To us, debugging has always been a big pain in developing microservices applications. We believe that Squash can mitigate this pain, and hope it will prove useful to others in the community.
We invite everyone to send us comments and suggestions, and join us in bringing Squash to more debuggers, platforms and IDEs.