Remotely debugging .NET in AWS Lambda (with Breakpoints)

As a .NET developer, debugging should be something you’re very familiar with and use every day. Having a debugger just an F5-press away is easy to take for granted. At least, it is until you temporarily switch to another language or deploy to an environment like AWS Lambda. Put your hand up if you’ve ever littered code with Console.WriteLine (or similar) to work out what's wrong…

Let’s talk about debugging .NET AWS Lambda functions. There are some good ways to do this locally, such as:

  • Adding a Main method and running your function as a console app.
  • Using the .NET Mock Lambda Test Tool from AWS to bootstrap your function and provide it with input.
  • Using lambci/docker-lambda run your function inside a Docker container based on the AWS Lambda filesystem.

These are great, but there’s nothing like the real thing. Especially when you’re only experiencing a problem after you deploy. In that case, you want remote debugging.

Debug Adapter Protocol

Tools like Visual Studio and Visual Studio Code use something called the Debug Adapter Protocol to communicate with the .NET Core debugger.

The idea behind the Debug Adapter Protocol is to standardize an abstract protocol for how a development tool communicates with concrete debuggers.

Before the Debug Adapter Protocol (DAP), each tool needed to implement support for each debugger. This took significant effort and result in a lot of duplication and waste.

The DAP is an abstract protocol that defines request, response, and event messages to do common debug tasks such as setting breakpoints, stepping through code, and evaluating expressions.

Since it’s unrealistic to expect every debugger to support this protocol, debug adapters can be created to act as intermediaries between the development tools and debuggers.

Development tool extensions, like OmniSharp for Visual Studio Code, add the small amount of extra code needed to support debugging additional languages.

Remote Debugging

Messages in the Debug Adapter Protocol consist of HTTP-like headers separated by \r\n, and a JSON object separated from the headers by \r\n.

The client sends an “initialize” request to start the debug session.

Put simply, Request messages are written to a debug adapter’s standard input, while responses and events are read from its standard output.

Microsoft provides “offroad” instructions for remote debugging using plink (the command line interface to PuTTY) and SSH. They show how to configure plink to SSH to a remote machine, start vsdbg there (the .NET Core debugger), and pipe standard I/O back and forth through the SSH connection to vsdbg.

This is really cool because the development tool doesn’t even need to know that it's talking to vsdbg running on another machine.

vsdbg acts as its own debug adapter when starts with --interpreter=vscode. This is a legacy name from when the DAP was just a part of Vsual Studio Code.

Making it work with Lambda Functions

Even if Lambda containers had an SSH server installed, they don’t have an IP address you can connect to. There’s also no way to tell vsdbg to open an outbound connection. I don’t know of any development tools that support that anyway.

These problems have at least one solution. I got a proof of concept working and named it LambdaRemoteDebug. The diagram below roughly shows what’s happening. Don’t worry if it doesn’t make much sense because I’ll go into more detail.

LambdaRemoteDebug is a NuGet package you install in your Lambda function. When your function runs, it starts itself in a separate process to avoid being paused by the debugger. That separate process makes an outbound TCP connection to a broker (more on that next) and starts the vsdbg process. Finally, it relays TCP and vsdbg I/O back and forth.

The broker is LambdaRemoteDebug.Tools, which is a .NET Core Global Tool (command line tools installed via NuGet packages). It accepts an inbound TCP connection from the Lambda as well as from the Client. plink is used to connect to the client with the broker.

When both a client and function connect, the broker relays traffic between the two. The broker can run on the same computer as your client or anywhere else, such as an EC2 instance. It only needs to accept incoming TCP connections on the ports you configure.

All this results in an indirect connection between the client and vsdbg which works just fine for debugging.

There are some limitations

You’re probably expecting me to tell you that there’s some fundamental flaw in this, but there’s not - it works well!

However, version 1.0.0 of LambdaRemoteDebug is very much a proof of concept. It’s as simple as possible to get it working for real-world debugging, and there are some limitations as a result, including:

  • The client must connect first.
  • You must restart the broker if you stop debugging before a Lambda connects.
  • The next Lambda execution after you stop debugging will fail.
  • It can only be used for one client and Lambda at a time (you can run multiple copies of the broker to work around this today).
  • It requires plink (easy to remove, but I think this is a very common tool).

All of these have solutions in my head, but I wanted to get this out early and hopefully get some feedback on whether it's useful.

Tutorial

Want to try it out? It’s really quite easy (and could be even easier in future). I’ve put Windows-specific instructions below, but you could adapt it to suit. Please get in contact with me in the comments or on Twitter if you try it!

Installation

1. Ensure you have “plink.exe”

There’s a good chance you already have plink.exe if you installed PuTTY via the “Windows Installer” so have a look for it. On my machine, it’s located at C:\Program Files\PuTTY\plink.exe.

If you don’t have it already, you can download all the PuTTY utilities, or just plink.exe, from the official website.

Either way, note the location of plink.exe because you’ll need it later.

2. Modify your project file (if needed)

2.1. Target framework
The NuGet package you’re going to install currently targets netcoreapp2.1, so you also need to, using <TargetFramework>netcoreapp2.1</TargetFramework>.

The Serverless aws-csharp template also targets netcoreapp2.1, but you may be targetting something else such as netstandard2.0.

2.2. Portable PDB
.NET Core introduced a new symbol file (PDB) format — portable PDBs. Unlike traditional PDBs which are Windows-only, portable PDBs can be created and read on all platforms. Lambda functions run on Amazon Linux, so you must generate portable PDBs.

Portable PDBs are the default for .NET Core projects using .csproj files, so you probably don’t need to do anything. If you’re using a project.json, have a read of these instructions.

For legacy reasons, the C# compiler option (and hence the name of the msbuild/project.json flags) to generate Windows PDBs is ‘full’. However, this should NOT imply that Windows-only PDBs have more information than Portable PDBs.

3. Install LambdaRemoteDebug

This NuGet package lets your code indicate that it wants to attach a remote debugger. You can install the LambdaRemoteDebug NuGet package via the GUI or the command line using dotnet add package LambdaRemoteDebug.

4. Modify your handler

This is the only code change you need. Perhaps the need for this can be removed in the future, though it does provide some nice flexibility.
At the point where you want to attach to a remote debugger (usually at the start of your handler method), add a call toLambdaRemoteDebug.Attach();

5. Create a launch.json file

This file tells the client everything it needs to know to communicate with a debug adapter.

It’s the same regardless of whether you’re using Visual Studio or Visual Studio Code. However, if you’re using Code, it needs to go in the .vscode directory to be picked up automatically.

There’s a template below, but you’ll need to change one or two things:

5.1. configurations.pipelineTransport.pipeProgram
plink.exe is in my path, so "plink" is enough. You can put a full path here if needed, but don’t forget to escape \ using \\.

5.2. configurations.pipelineTransport.pipeArgs
Choose a port for your client to connect to your broker on and replace 21425 (or use that since it’s a nice number). Note the number you choose, this will be your <client port>.

languageMappings and exceptionCategoryMappings are just Micorosft-provided UUIDs. Nothing else in configurations should be changed.
processId is 2 because your dotnet process is the second process to start.

6. Install LambdaRemoteDebug.Tools
This is a .NET Core Global Tool. These are command line tools installed via NuGet packages. LambdaRemoteDebug.Tools contains the broker.

Install — dotnet tool install --global LambdaRemoteDebug.Tools
Uninstall — 
dotnet tool uninstall --global LambdaRemoteDebug.Tools

After installing, you should be able to run lrdbg without any arguments as a test.

7. Create a Lambda Layer
Instead of downloading vsdbg repeatedly, it’s better to package it into a Lambda Layer for fast reuse. You can use mine in eu-west-1, or you can create your own.

7.1. Use mine
arn:aws:lambda:eu-west-1:157642868069:layer:lambda-remote-debug:1
This is a public layer built from vsdbg version 16.0.11220.2 commit:a70b65d83261a401b0727b6a63dadc38dc28b76d.

This is convenient for getting started quickly, but I suggest you make your own in case it suddenly disappears.

7.2. Create one yourself
It’s best to do this on an Amazon Linux machine. Below I’m using EC2.

  • Create an EC2 instance with the .NET Core 2.1 with Amazon Linux 2 AMI.
  • Install vsdbg: wget https://aka.ms/getvsdbgsh -O — 2>/dev/null | sudo /bin/sh /dev/stdin -v vs2017u5 -l /opt/vsdbg
  • Install zip — apt install zip -y
  • Create a .zip file — cd /opt && zip -r ~/layer.zip vsdbg && cd ~
    You can do this with another tool, but just make sure the vsdbg directory is at the top level of the .zip file.
  • Create a new Layer — Use the AWS Console or CLI/API. The configuration I used is below. Note the Lambda Layer Version ARN for later.

Debugging

After following the Installation steps, you’re ready to debug.

1. Start the Broker

You already have a <client port> from step 5.2, so you only need to choose a <lambda port>. You’ll configure Lambda functions to connect to later.

Usage — lrdbg broker <client port> <lambda port>
Example — lrdbg broker 21425 32564

It will say it’s waiting for a client connection.

2. Lambda: Add the layer

You can do this via the CLI/API or follow the GUI. Either way, use the Lambda Layer version ARN from step 7.

3. Lambda: Configure environment variables

There are two variables you need to set on your function. Without these, LambdaRemoteDebug.Attach() won’t do anything but log a message.

  • LAMBDA_REMOTE_DEBUG_IP — The IP address of the broker.
  • LAMBDA_REMOTE_DEBUG_PORT — The <lambda port> of the broker.

4. Connect a Client

4.1. Visual Studio Code
Open your project folder and switch to the Debug view using the left menu, View > Debug, or by pressing Ctrl-Shift-D.

Ensure AWS Lambda is selected in the dropdown (that’s the name of the configuration in launch.json). If it’s not there, ensure that launch.json is in the .vscode directory.

4.2. Visual Studio
This is a bit more clunky but works basically the same. I don’t know of a less manual way to start debugging. It also sucks you have to use the full path to launch.json. Anyway…

Open your project/solution, then open the Command Window using View > Other Windows > Command Window or by pressing Ctrl+W, A.

The command to start debugging is: DebugAdapterHost.Launch /LaunchJson:”C:\path\to\launch.json” /ConfigurationName:”AWS Lambda”

Visual Studio will freeze until you execute your Lambda function. After a couple of seconds, a popup to cancel will appear, though. This could be fixed in a future version.

5. Trigger your Lambda and troubleshoot problems

You can directly or indirectly trigger your function any way you normally would.

If it works, but your breakpoints aren’t hit or stepping through code is strange, you probably built using the Release configuration instead of Debug.

If it doesn’t work, check the CloudWatch Logs. The most common problem you’ll encounter is the function being unable to connect to the broker. Below are some connectivity troubleshooting tips.

  • Check the environment variables contain the right IP and port.
  • If you’re connecting over the internet, ensure inbound connections are routed to the broker (you probably need to configure a NAT on your router).
  • Ensure there isn’t a firewall, security groups, or ACL blocking inbound connections on your ports.
  • If your function runs in a VPC, ensure it has a NAT Gateway, NAT Instance, or Internet Gateway attached.
  • If you’re using a VPN into AWS, ensure you’re using the correct IP address.

Feel free to get in touch if you find any other problems. I’m sure they exist.


Wrapping it up

The code is available on GitHub, along with an example project in which you just need to change the environment variables and layer in serverless.yml. You can then deploy it with build && serverless deploy.

If this project is useful to you, please let me know!


For more like this, please follow me on Medium and Twitter.