Debugging Golang Appengine module with Visual Studio Code

Introduction

I have been playing with Appengine for a couple of years now, initially using Java and GWT (back in 2012). Three years ago, I discovered Go and could not resist to build some Appengine modules. To write Go, I use to be an Atom user; I recently switch to vscode. This editor comes with a Golang extension and can use Delve to debug Go code.

In this article I would like to share my experience about how to debug an Appengine module in local using vscode. I will appreciate any feedback and proposition of other approach to achieve same kind of task.

Setup

Assumption

VSCode and its Golang extension are installed on your system. Also “git” is available on your system.

Note that I tested this setup on Arch Linux and Ubuntu with Go 1.6.1

Proposed configuration

Create a folder to host our installation and project. In this article it will be known as basefolder. Avoid using symblink in your basefolder: this would lead to some problems when vscode will try to use go tooling (issue here).

mkdir -p ~/tmp/dbgGoAppengine
export BASEFOLDER=~/tmp/dbgGoAppengine

Appengine is using its own version of Go. Today (July 2016) the version of the SDK is 1.9.38 and can be downloaded here. Unzip the appgengine archive inside your basefolder.

cd $BASEFOLDER
# !!! following link is for linux; update it for you OS
wget https://storage.googleapis.com/appengine-sdks/featured/go_appengine_sdk_linux_amd64-1.9.40.zip
unzip go_appengine_sdk_linux_amd64-1.9.40.zip

What is the Golang version exposed by the SDK?

cd $BASEFOLDER/go_appengine/goroot; export GOROOT=`pwd`
bin/goapp version
>>> go version go1.6.1 (appengine-1.9.38) linux/amd64

Note 2 things here:
First, the version of Golang contains an unusually string: (appengine-1.9.38)
Second, the go binary of the SDK is named “goapp” and not “go”. If we setup the GOROOT here, vscode will complain that it cannot find “go” command under GOROOT. A simple link will do the trick here:

cd $GOROOT/bin; ln -s goapp go

The Golang extension of vscode use a large set of tools. Let’s install them in our new environment.
First let’s set your GOPATH:

export GOPATH=$BASEFOLDER/go; mkdir -p $GOPATH/src

Now we can install the tools as proposed in the Golang extension of vscode:

go get -u -v github.com/nsf/gocode
go get -u -v github.com/rogpeppe/godef
go get -u -v github.com/golang/lint/golint
go get -u -v github.com/lukehoban/go-outline
go get -u -v sourcegraph.com/sqs/goreturns
go get -u -v golang.org/x/tools/cmd/gorename
go get -u -v github.com/tpng/gopkgs
go get -u -v github.com/newhook/go-symbols
go get -u -v golang.org/x/tools/cmd/guru

Now everything is ready to run a module under appengine.

Launching Appengine module from vscode

Starting vscode

To have the best development experience we must take care of setting 3 variables in the environment before starting vsode: GOROOT, GOPATH, PATH.

Note that to have vscode and golang tools working correctly I had to set the GOROOT and the GOPATH in the environment and also in the vscode project preferences.

In order to ease that setup, I have create a starter project that you can clone.

cd $GOPATH/src
git clone https://github.com/dbenque/GoAppengineVSCodeStarter.git

This project contains:

• A launcher script “launch.vscode.sh” to setup the environment and update the vscode workspace’s preferences.
• An appengine module: minimal code example.
• Configuration files for vscode settings, tasks and debugger (folder .vscode)

At that stage you are ready to launch the project. The launcher script must be run from your project directory (workspace directory for vscode) and needs the variable BASEFOLDER to be defined.

Go to you project directory and:

> ./launch.vscode.sh

or if you BASEFOLDER is not set:

> BASEFOLDER=~/tmp/dbgGoAppengine ./launch.vscode.sh

Now you should be under vscode, if you open a “.go” file, you can access symbol definition by moving the mouse over (also use CTRL key). Note that you can browse you project and the appengine SDK symbols.

Launching Appengine module

The starter project defines one task “.vscode/task.json” to launch the appengine module. To launch the “dummy” module: press F1 to open vscode palette, then type and select “Run Task”, finally select the task “server dummy module only”. Later on, to terminate the task: press F1 and type “Terminate Running Task”.

Ensure that the Output is visible (CTRL+Shift+H) and that the output is set on “Tasks” (combobox on the right of the panel). In the output you should see the local Appengine output:

Later on, you may want to launch a different module, or several modules simultaneously; for that you will have to modify the task definition (F1 + “Configure Task Runnner”) stored in the file “.vscode/task.json”:

You can check that the module is responding correctly by browsing the following URL:

http://localhost:8080/dummy/

Debugging your module under vscode

Patched Delve

We are going to use the Delve debugger. Till recently Delve was not supporting Appengine. Now that @derekparker has merged my PR#566, it should be ok.

Before that any attempt to attach Delve debugger to an Appengine module would result in such error:

could not attach to pid 28078: Could not parse version number: “go1.6.1 (appengine-1.9.38)”

Appengine processes

In the previous chapter when we have launched the task, the Appengine was started and the module was deployed. If you do a “pstree” you will notice that vscode created the following process (purged view):

The 3 process highlighted in blue are the one associated to Appengine. The one running the module is “/tmp/tmpg3svT5appengine-go-bin/_go_app”.

So we need to attach Delve to that process and we are done! Yes we are for a couple of seconds ;-). There are 2 things to take into account before going further and have a good debugging experience:
1- Once you break into your module, Appengine notices a high latency on the service and start another instance of the module.
2- Each time you will make a change in your GOPATH, Appengine rebuilds the module and deploys a new instance with latest code changes.

Controlling number of instances

Having only one instance of our module running at a time is required. So we are sure to debug the good one (this only one). To achieve that we have to configure our module with “Manual scaling” (documentation: here and here)

In the module configuration file, check that the scaling policy (line 6–7):

Of course remember to remove these lines once your are done with your debugging and before deploying your code to Appengine. (Remove or replace with another scaling definition).

Continuously reattach Delve

To tackle the point 2, the proposed solution is to monitor the processes spawned by Appengine and attach automatically Delve server to the good instance. You can build your own script for that, or reuse my project “delveAppengine”:

go get -u -v github.com/dbenque/delveAppengine

Note that this work for Linux. Some adaptations are require for Mac and Windows. Thanks to Cedric Lamoriniere for Mac adaptations. Any pull request for Windows support is welcome.

Launch the “delveAppengine”:

$GOPATH/bin/delveAppengine

At that stage if you have an Appengine module running, you should see a log like this:

2016/06/07 23:27:51 debugger.go:54: attaching to pid 27836

If you have no log, you probably need to start an Appengine module (see previous chapter “Launching Appengine module”)

If you have an extra log with the following error:

Could not attach to pid 27836: set proc/sys/kernel/yama/ptrace_scope to 0

(On linux) You will have to modify your configuration and edit the file “proc/sys/kernel/yama/ptrace_scope” so that it contains the value “0” (zero).

echo 0|sudo tee /proc/sys/kernel/yama/ptrace_scope

For more details: https://www.kernel.org/doc/Documentation/security/Yama.txt
Once the change is done, modify a source file of the module, save: the module will be redeployed, and this time the debugger should be able to attach to the process.

Once the debugger is correctly attached; you can modify a “.go” file in your Appengine module. Appengine that is monitoring your project and the GOPATH will notice that; it will rebuild and redploy another instance of the module:

On the “delveAppengine” side you should see new logs:

> delveAppengine 
2016/06/07 23:27:51 debugger.go:54: attaching to pid 27836
2016/06/07 23:30:54 main.go:104: Old server still listening.
2016/06/07 23:30:55 debugger.go:54: attaching to pid 30760

Options

delveAppengine have some options and default values that you can use for customizations:

> delveAppengine --help
Usage of delveAppengine:
  -delay int
        Time delay in seconds between each appengine process scan (default 3)
  -key string
        Magic key to identify a specific module bianry (default is empty string)
  -port int
        Port used by the Delve server (default 2345)

Zombies

When Appengine launches the new process, the old one is still attached to the debugger. This result in a zombie process. Of course if you repeat the operation several times you will end up in an army of zombies… The system removes them as soon as you stop “delveAppengine” or the Appengine task under vscode.

Files organization

Because appengine copies the source file that are under the module directory before building the module, we won’t be able to place any breakpoint into these files: the debugger would not recognize the file path.

For other files, appengine is going to use the sources found under the GOPATH and GOROOT. So the files under GOPATH and GOROOT are eligible to hold breakpoints, the file path will be recognized in the symbol table of the debugger.

Thus I recommend the following file structure for your project:

projectRoot
   |- modules
   |   |- myModule1
   |   |    |- init.go        // router pattern to handler
   |   |    |- myModule1.yaml // configuration for the module
   |   |- myModule2
   |        |- init.go        // router pattern to handler
   |        |- myModule2.yaml // configuration for the module
   |    
   |- pkg
   |   |- myModule1
   |   |    |- *.go           // sources, subfolders(packages)
   |   |                      // with handlers and business code
   |   |- myModule2
   |   |    |- *.go           // sources, subfolders(packages)
                              // with handlers and business code

Keep the code under “modules” as small as possible. Only the code under “pkg” can be seen by the debugger.

Debugging

The starter project comes with a configuration for the debugger. It defines a remote debugging, using port 2345; this port is the used by our running debugger delveAppengine.

File “.vscode/laucnh.json”:

{
 “version”: “0.2.0”,
 “configurations”: [
 {
 “name”: “Remote”,
 “type”: “go”,
 “request”: “launch”,
 “mode”: “remote”,
 “remotePath”: “${workspaceRoot}”,
 “port”: 2345,
 “host”: “127.0.0.1”,
 “program”: “${workspaceRoot}”,
 “env”: {},
 “args”: []
 }
 ]
}

Open the source file “pkg/handler.go”. Switch to the debug view of vscode (Ctrl+Shift+D). Place a breakpoint in the function “Handler” by clicking in the margin or using F9.

Press F5 to connect to the debugger. Select the “remote” configuration.

In the window running the delveAppengine you should see new logs like:

2016/07/14 20:05:51 debugger.go:235: created breakpoint: ......
2016/07/14 20:05:51 debugger.go:390: continuing

Browse again the url of your module: http://localhost:8080/dummy/
The debugger should interrupt the execution at your breakpoint. Now you can use vscode interface to interact with the debugger:

It is better to place the breakpoint (F9) before running the debugger (F5) or while the program is interrupted. If not you may end up with an “unverified breakpoint” represented by a grey dot in vscode. In that case you will have to restart the debugger (Ctrl+Shift+F5) for it to be taken into account.

If you notice that breakpoint definitions in vscode is not aligned with the remote debugger, stop the debugging session (Shift+F5), modify and save one file in the project to trig a redeployment. Place again your breakpoints and start a new debugging session (F5).

In worst cases stop the debugging session (Shift+F5), also stop “delveAppengine” then restart it and finally open a new debugging session (F5).

Multiple modules

If your application is made of several modules, several “_go_app” processes are launched. We need to help “delveAppengine” to find the good one. For that we can use the option “-key” and give a string that only exit in the targetted module for example, a symbol:

$GOPATH/bin/delveAppengine \
       -key=”GoAppengineVSCodeStarter/pkg/dummy.Handler”

You can also target a string definition. For example you can modify your module init function like this:

func init() {
 http.HandleFunc(“/”, dummy.Handler)
 log.Println(“Init dummy module”)
}

and then pass “Init dummy module” as a key to “delveAppengine”

$GOPATH/bin/delveAppengine \
       -key=”Init dummy module”

Not that it is only possible to debug one module at a time. Even if it is possible to attach a “delveAppengine” to each module (on different port), vscode can only run one debugging session at a time.

Note: you can still debug multiple modules at the same time if you open one vscode per module.

All commands

Here is a recap of the full list of commands to setup the environment. Before that you must install vscode and its Golang extension.

#Creation base directory
mkdir -p ~/tmp/dbgGoAppengine
export BASEFOLDER=~/tmp/dbgGoAppengine
cd $BASEFOLDER
#Download Appengine
# !!! following link is for linux; update it for you OS
wget https://storage.googleapis.com/appengine-sdks/featured/go_appengine_sdk_linux_amd64-1.9.40.zip
unzip go_appengine_sdk_linux_amd64-1.9.40.zip
#Setup Go environment
cd $BASEFOLDER/go_appengine/goroot; export GOROOT=`pwd`
cd $GOROOT/bin; ln -s goapp go
export GOPATH=$BASEFOLDER/go; mkdir -p $GOPATH/src
#Install go tools
go get -u -v github.com/nsf/gocode
go get -u -v github.com/rogpeppe/godef
go get -u -v github.com/golang/lint/golint
go get -u -v github.com/lukehoban/go-outline
go get -u -v sourcegraph.com/sqs/goreturns
go get -u -v golang.org/x/tools/cmd/gorename
go get -u -v github.com/tpng/gopkgs
go get -u -v github.com/newhook/go-symbols
go get -u -v golang.org/x/tools/cmd/guru
#Clone startup project and launch vscode for that project
cd $GOPATH/src
git clone https://github.com/dbenque/GoAppengineVSCodeStarter.git
cd $GOPATH/src/GoAppengineVSCodeStarter/
BASEFOLDER=~/tmp/dbgGoAppengine ./launch.vscode.sh
#Get the debugger and launch it
go get -u -v github.com/dbenque/delveAppengine
$GOPATH/bin/delveAppengine