Finally proof that serverless has servers?
I spoke at dotScale in Paris last week about some stuff I’ve been working on to learn more about Function-as-a-Service (FaaS) performance.
While my talk was mostly about cold starts, function warming, and understanding the internal architecture of FaaS, lots of people were curious how I was able to SSH into a running AWS Lambda function. This post describes how to turn Lambda functions into a short-lived Linux servers with some help from Go’s SSH crypto libraries.
Adventures in missing dependencies
Many popular FaaS services are built on Linux containers—Azure seems to use IIS voodoo I don’t understand. From an serverless app developer perspective, you write a handler function in a supported language that’s bundled with its dependencies. When a predefined event like an HTTP request occurs, the handler function is invoked inside the container and your code runs (or has an error, or times out).
Because the environment is just a container, you’re allowed to execute binaries using standard language features like exec() in node.js or python subprocess. This is also how open-source projects like Apex let developers run Go in AWS Lambda.
With a 12-line lambda function, I tried running the sshd process on a non-privileged port with a host key and configuration I bundled with my the function. This was a bad idea for several reasons:
- AWS Lambda’s container environment is missing several libraries that the ssh daemon requires.
- AWS Lambda functions do not seem to allow any inbound port access (yes, even if running in a VPC with a security group that explicitly allows it).
- Exotic ssh configuration files are hard to write and filled with weird options.
- There’s no easy way to automatically determine the external IP address of a Lambda function.
It was time for some SSH tunneling and copy-and-paste Golang code from Github.
Writing an SSH server that creates a tunnel in Go
Go has some nice high-level SSH libraries for creating servers, clients, and tunnels. Unfortunately, my first copy-and-paste attempt had a dependency on an OS feature (unix pseudoterminals) that did not exist in the AWS Lambda environment. Digging around a bit more in the Go documentation, I found some libraries designed for interacting with terminals that didn’t have a dependency on the pty package.
Cobbling it all together, I had a simple go service that:
- Runs an SSH server capable of executing commands in bash on a non-privileged port.
- Creates a tunnel to a remote host (i.e. my laptop via ngrok or an EC2 host running inside my VPC) and requests a port to be opened that forwards to the SSH server running inside the Lambda function.
I then wrote a Lambda function that:
- Is configured with a 5-minute timeout (the maximum allowed value).
- Executes the Go SSH binary I built for 64-bit Linux (an under-appreciated feature of Go development on macs).
- Used Lambda environment variables to connect to a specific host and port number (i.e. my laptop via a jump proxy).
With this in place, using the tunnel I can access my go SSH server and run shell commands from another host using a standard SSH client until the Lambda function times out:
This isn’t a good idea (but it was fun to build)
I don’t think creating SSH tunnels and running random servers inside AWS Lambda is a particularly good way to take advantage of serverless, but it was fun to learn more about the execution environment by building it. Couple other possibilities I’m thinking of now that there’s a proof on concept:
- Running other types of servers in Lambda that allow inbound connections through tunneling.
- Integration with a key-management services for SSH authentication (KMS or Vault perhaps?)
- Support for concurrent sessions or integration with a TCP load balancer.
The code is up on Github (project name: faassh). Pull requests and questions welcome.