Remotely debugging .NET in AWS Lambda (with Breakpoints)
As a .NET developer, debugging should be something you’re very familiar with and use every day. Having a debugger just an F5-press away is easy to take for granted. At least, it is until you temporarily switch to another language or deploy to an environment like AWS Lambda. Put your hand up if you’ve ever littered code with
Console.WriteLine (or similar) to work out what's wrong…
Let’s talk about debugging .NET AWS Lambda functions. There are some good ways to do this locally, such as:
- Adding a
Mainmethod and running your function as a console app.
- Using the .NET Mock Lambda Test Tool from AWS to bootstrap your function and provide it with input.
- Using lambci/docker-lambda run your function inside a Docker container based on the AWS Lambda filesystem.
These are great, but there’s nothing like the real thing. Especially when you’re only experiencing a problem after you deploy. In that case, you want remote debugging.
Debug Adapter Protocol
The idea behind the Debug Adapter Protocol is to standardize an abstract protocol for how a development tool communicates with concrete debuggers.
Before the Debug Adapter Protocol (DAP), each tool needed to implement support for each debugger. This took significant effort and result in a lot of duplication and waste.
The DAP is an abstract protocol that defines request, response, and event messages to do common debug tasks such as setting breakpoints, stepping through code, and evaluating expressions.
Since it’s unrealistic to expect every debugger to support this protocol, debug adapters can be created to act as intermediaries between the development tools and debuggers.
Development tool extensions, like OmniSharp for Visual Studio Code, add the small amount of extra code needed to support debugging additional languages.
Messages in the Debug Adapter Protocol consist of HTTP-like headers separated by
\r\n, and a JSON object separated from the headers by
Microsoft provides “offroad” instructions for remote debugging using
plink (the command line interface to PuTTY) and SSH. They show how to configure
plink to SSH to a remote machine, start
vsdbg there (the .NET Core debugger), and pipe standard I/O back and forth through the SSH connection to
This is really cool because the development tool doesn’t even need to know that it's talking to
vsdbg running on another machine.
vsdbgacts as its own debug adapter when starts with
--interpreter=vscode. This is a legacy name from when the DAP was just a part of Vsual Studio Code.
Making it work with Lambda Functions
Even if Lambda containers had an SSH server installed, they don’t have an IP address you can connect to. There’s also no way to tell
vsdbg to open an outbound connection. I don’t know of any development tools that support that anyway.
These problems have at least one solution. I got a proof of concept working and named it LambdaRemoteDebug. The diagram below roughly shows what’s happening. Don’t worry if it doesn’t make much sense because I’ll go into more detail.
LambdaRemoteDebug is a NuGet package you install in your Lambda function. When your function runs, it starts itself in a separate process to avoid being paused by the debugger. That separate process makes an outbound TCP connection to a broker (more on that next) and starts the
vsdbg process. Finally, it relays TCP and vsdbg I/O back and forth.
The broker is LambdaRemoteDebug.Tools, which is a .NET Core Global Tool (command line tools installed via NuGet packages). It accepts an inbound TCP connection from the Lambda as well as from the Client.
plink is used to connect to the client with the broker.
When both a client and function connect, the broker relays traffic between the two. The broker can run on the same computer as your client or anywhere else, such as an EC2 instance. It only needs to accept incoming TCP connections on the ports you configure.
All this results in an indirect connection between the client and vsdbg which works just fine for debugging.
There are some limitations
You’re probably expecting me to tell you that there’s some fundamental flaw in this, but there’s not - it works well!
1.0.0 of LambdaRemoteDebug is very much a proof of concept. It’s as simple as possible to get it working for real-world debugging, and there are some limitations as a result, including:
- The client must connect first.
- You must restart the broker if you stop debugging before a Lambda connects.
- The next Lambda execution after you stop debugging will fail.
- It can only be used for one client and Lambda at a time (you can run multiple copies of the broker to work around this today).
- It requires plink (easy to remove, but I think this is a very common tool).
All of these have solutions in my head, but I wanted to get this out early and hopefully get some feedback on whether it's useful.
Want to try it out? It’s really quite easy (and could be even easier in future). I’ve put Windows-specific instructions below, but you could adapt it to suit. Please get in contact with me in the comments or on Twitter if you try it!
1. Ensure you have “plink.exe”
There’s a good chance you already have
plink.exe if you installed PuTTY via the “Windows Installer” so have a look for it. On my machine, it’s located at
If you don’t have it already, you can download all the PuTTY utilities, or just
plink.exe, from the official website.
Either way, note the location of
plink.exe because you’ll need it later.
2. Modify your project file (if needed)
2.1. Target framework
The NuGet package you’re going to install currently targets
netcoreapp2.1, so you also need to, using
aws-csharptemplate also targets
netcoreapp2.1, but you may be targetting something else such as
2.2. Portable PDB
.NET Core introduced a new symbol file (PDB) format — portable PDBs. Unlike traditional PDBs which are Windows-only, portable PDBs can be created and read on all platforms. Lambda functions run on Amazon Linux, so you must generate portable PDBs.
Portable PDBs are the default for .NET Core projects using .csproj files, so you probably don’t need to do anything. If you’re using a project.json, have a read of these instructions.
For legacy reasons, the C# compiler option (and hence the name of the msbuild/project.json flags) to generate Windows PDBs is ‘full’. However, this should NOT imply that Windows-only PDBs have more information than Portable PDBs.
3. Install LambdaRemoteDebug
This NuGet package lets your code indicate that it wants to attach a remote debugger. You can install the LambdaRemoteDebug NuGet package via the GUI or the command line using
dotnet add package LambdaRemoteDebug.
4. Modify your handler
This is the only code change you need. Perhaps the need for this can be removed in the future, though it does provide some nice flexibility.
At the point where you want to attach to a remote debugger (usually at the start of your handler method), add a call to
5. Create a launch.json file
This file tells the client everything it needs to know to communicate with a debug adapter.
It’s the same regardless of whether you’re using Visual Studio or Visual Studio Code. However, if you’re using Code, it needs to go in the
.vscodedirectory to be picked up automatically.
There’s a template below, but you’ll need to change one or two things:
plink.exe is in my path, so
"plink" is enough. You can put a full path here if needed, but don’t forget to escape
Choose a port for your client to connect to your broker on and replace
21425 (or use that since it’s a nice number). Note the number you choose, this will be your
exceptionCategoryMappingsare just Micorosft-provided UUIDs. Nothing else in
configurationsshould be changed.
2because your dotnet process is the second process to start.
6. Install LambdaRemoteDebug.Tools
This is a .NET Core Global Tool. These are command line tools installed via NuGet packages. LambdaRemoteDebug.Tools contains the broker.
dotnet tool install --global LambdaRemoteDebug.Tools
dotnet tool uninstall --global LambdaRemoteDebug.Tools
After installing, you should be able to run
lrdbg without any arguments as a test.
7. Create a Lambda Layer
Instead of downloading
vsdbg repeatedly, it’s better to package it into a Lambda Layer for fast reuse. You can use mine in
eu-west-1, or you can create your own.
7.1. Use mine
arn:aws:lambda:eu-west-1:157642868069:layer:lambda-remote-debug:1This is a public layer built from vsdbg version
This is convenient for getting started quickly, but I suggest you make your own in case it suddenly disappears.
7.2. Create one yourself
It’s best to do this on an Amazon Linux machine. Below I’m using EC2.
- Create an EC2 instance with the
.NET Core 2.1 with Amazon Linux 2AMI.
- Install vsdbg:
wget https://aka.ms/getvsdbgsh -O — 2>/dev/null | sudo /bin/sh /dev/stdin -v vs2017u5 -l /opt/vsdbg
- Install zip —
apt install zip -y
- Create a .zip file —
cd /opt && zip -r ~/layer.zip vsdbg && cd ~
You can do this with another tool, but just make sure the vsdbg directory is at the top level of the .zip file.
- Create a new Layer — Use the AWS Console or CLI/API. The configuration I used is below. Note the Lambda Layer Version ARN for later.
After following the Installation steps, you’re ready to debug.
1. Start the Broker
You already have a
<client port> from step 5.2, so you only need to choose a
<lambda port>. You’ll configure Lambda functions to connect to later.
lrdbg broker <client port> <lambda port>Example —
lrdbg broker 21425 32564
It will say it’s waiting for a client connection.
2. Lambda: Add the layer
You can do this via the CLI/API or follow the GUI. Either way, use the Lambda Layer version ARN from step 7.
3. Lambda: Configure environment variables
There are two variables you need to set on your function. Without these,
LambdaRemoteDebug.Attach() won’t do anything but log a message.
LAMBDA_REMOTE_DEBUG_IP— The IP address of the broker.
<lambda port>of the broker.
4. Connect a Client
4.1. Visual Studio Code
Open your project folder and switch to the Debug view using the left menu,
View > Debug, or by pressing
AWS Lambda is selected in the dropdown (that’s the
name of the configuration in
launch.json). If it’s not there, ensure that
launch.json is in the
4.2. Visual Studio
This is a bit more clunky but works basically the same. I don’t know of a less manual way to start debugging. It also sucks you have to use the full path to
Open your project/solution, then open the Command Window using
View > Other Windows > Command Window or by pressing
The command to start debugging is:
DebugAdapterHost.Launch /LaunchJson:”C:\path\to\launch.json” /ConfigurationName:”AWS Lambda”
Visual Studio will freeze until you execute your Lambda function. After a couple of seconds, a popup to cancel will appear, though. This could be fixed in a future version.
5. Trigger your Lambda and troubleshoot problems
You can directly or indirectly trigger your function any way you normally would.
If it works, but your breakpoints aren’t hit or stepping through code is strange, you probably built using the Release configuration instead of Debug.
If it doesn’t work, check the CloudWatch Logs. The most common problem you’ll encounter is the function being unable to connect to the broker. Below are some connectivity troubleshooting tips.
- Check the environment variables contain the right IP and port.
- If you’re connecting over the internet, ensure inbound connections are routed to the broker (you probably need to configure a NAT on your router).
- Ensure there isn’t a firewall, security groups, or ACL blocking inbound connections on your ports.
- If your function runs in a VPC, ensure it has a NAT Gateway, NAT Instance, or Internet Gateway attached.
- If you’re using a VPN into AWS, ensure you’re using the correct IP address.
Feel free to get in touch if you find any other problems. I’m sure they exist.
Wrapping it up
The code is available on GitHub, along with an example project in which you just need to change the environment variables and layer in
serverless.yml. You can then deploy it with
build && serverless deploy.
If this project is useful to you, please let me know!