Run Netflix BLESS SSH Certificate Authority in AWS Lambda

Issue Short Lived User Certs for SSH Access

I found the following video posted on my company’s intranet the other week and really enjoyed Russell Lewis’ talk on how Netflix are using SSH Certificates and the dangers of mismanaging your SSH keys.

I’d never really had much exposure to SSH Certificates, aside from my own personal reading about at home about a year ago.

As a cloud user at home and at work, in particular Amazon Web Services (AWS), I was intrigued about trying out Netflix’s BLESS tool in AWS Lambda, to issue users with temporary (short lived) CA signed SSH Certificates to authenticate SSH access to servers.

I spent some time looking into BLESS and what it can do, and setup up a mock environment of my own in AWS to see what it’s all about; and thought I’d log my initial findings here in a small post for others to read.

Prerequisites

1. AWS Account
2. KMS Key
3. Python Lambda Function

You will need an account in AWS account with suitable permissions to set this up. Netflix recommend you have a completely isolated AWS account with just BLESS running inside to keep it away from other concerns. As this is just a mock setup, I have used the same account for both BLESS and other services.

You will also need a KMS key in the AWS Region(s) you want to operate in. As KMS is regional, you’ll have to create one in each region you want to use BLESS in. It will be needed later to encrypt your CA Certificate Key Pair Passphrase. The BLESS config file bless_deploy.cfg caters for multiple regions by being about to declare a base64 encoded string for each region.

$ export BLESS_REGION=eu-west-2
$ aws kms create-alias \
            --region $BLESS_REGION \
            --alias-name alias/bless \
            --target-key-id \
$(aws kms create-key --region $BLESS_REGION | jq -r .KeyMetadata.KeyId)

You will also need to setup an AWS Lambda function, to upload our deployment artifact to, with permissions to access your KMS key. It will be python runtime and for this post’s example the handler will be bless_lambda_user.lambda_handler_user

Setting up your Environment

Download the BLESS Code

The official repository lives at Netflix/bless on GitHub.

Netflix/bless

However in my initial setup, I was hit with errors, and later found there was an open Pull Request to address a bug with the latest version of marshmallow ; The PR is still open at time of writing.

As a quick temporary step, I cloned the official repository and added NeolithEra’s repo as another git remote and checked out their branch to work on.

$ git clone git@github.com:Netflix/bless.git

$ git remote add NeolithEra git@github.com:NeolithEra/bless.git
$ git fetch NeolithEra
$ git checkout NeolithEra/master

Setup Python venv and Download Dependancies

The repository includes a Makefile to help users get setup quickly in a few simple commands.

Start by creating and activating a virtual environment to work from:

$ python3 -m venv venv
$ source venv/bin/activate

pip install all the code dependencies:

$ make develop

Run the tests:

$ make test

Hopefully all the tests pass for you…

Package up Dependancies

AWS Lambda requires the runtime code and its dependancies to be uploaded together in a single artifact. Run the lambda-deps make command to save all your python dependancies in a local directory.

$ make lambda-deps

Behind the scenes the Makefile is spinning up a Docker container of amazonlinux:2 and pip installing all the requirements into the aws_lambda_libs folder.

Setting up your Certificate Authority (CA)

Generate a CA Certificate Key Pair

Use ssh-keygen to generate a new key pair for you, and be sure to add a strong passphrase to protect it with.

We’ll make use of KMS later to encrypt the passphrase and allow only our lambda function to decrypt it on the fly when required.

$ ssh-keygen -t rsa -b 4096 -m PEM -f cas -C "SSH CA Key"

Encrypt your CA Key Pair Passphrase with AWS KMS Keys

Get the KMS keyID for your region and use the AWS Cli to encrypt you CA passphrase that will be later used in your BLESS config file.

It needs to be base64 encoded in the config file.

$ aws kms encrypt \
    --key-id $(aws kms describe-key --region eu-west-2 --key-id alias/bless | jq -r .KeyMetadata.KeyId) \
    --plaintext Password123 \
    --region eu-west-2 | jq -r .CiphertextBlob | base64

Configure, Package up & Deploy your Lambda Function

Create a directory for your BLESS config and key that will be bundled in the Lambda artifact.

There’s a sample configuration that you can use and edit for your setup available in the git repository.

Copy your config and private CA key into the configs directory.

$ mkdir lambda_configs
$ cp bless/config/bless_deploy_example.cfg lambda_configs/bless_deploy.cfg 
$ cp cas lambda_configs/cas.priv

Update the Bless CA section with your CA Passphrase and Filename

$ vim lambda_configs/bless_deploy.cfg

The region in the variable below is the same one(s) used to encrypt your passphrase with KMS above. If you have encrypted the passphrase with multiple KMS keys in multiple regions, you’ll need to add a variable for each.

[Bless CA]
eu-wast-2_password = <base64 encoded encrypted passphrase>
ca_private_key_file = cas.priv

Bundle up your Lambda Artifact

$ make publish

$ ls -l publish
total 32024
drwxr-xr-x  50 keirwhitlock  staff      1600 24 Dec 19:36 bless_lambda
-rw-r--r--   1 keirwhitlock  staff  16165660 24 Dec 19:36 bless_lambda.zip

Upload to AWS Lambda

Upload the resulting bless_lambda.zip to your AWS Lambda function.

1. Open the AWS Lambda console.
2. Choose Functions on the navigation pane, and then open your function.
3. In the Function code section, expand the Code entry type drop-down list, and then choose Upload a .ZIP file.
4. Choose Upload, and then select your .zip file.
5. Choose Save.

Server Side Configuration

Launch an instance to test your BLESS signed certs from. Any linux OS should do. I used Amazon Linux 2 AMI purely because it was top in the list.

Trust the CA Key on the Server

Now you need to setup your server(s) to trust your CA certificate and if you wish, restrict with users can log in.

Add the following line to your sshd_config :

TrustedUserCAKeys /etc/ssh/cas.pub

Then scp a copy your CA’s public key to the server to /etc/ssh.

Be sure to restrict it to root privileges & restart sshd when done.

$ systemctl restart sshd.service

Restrict which users can log on

I created an additional local user to test with other than the default ec2-user

On the server create a file in /etc/ssh with the names of all the users allowed to log in via this method

$ echo keirtest > /etc/ssh/auth_principals

.. and refer to it in the /etc/ssh/sshd_config with the AuthorizedPrincipalsFile parameter:

AuthorizedPrincipalsFile /etc/ssh/auth_principals

Once added, restart sshd

$ systemctl restart sshd.service

Testing your BLESS Setup

The Netflix team have included a bless_client.py script to allow you to test your lambda function is working correctly and generate your signed certificate from your local SSH key.

If you watched the video I included at the start, you’ll see that Netflix use a Bastion host for their users to first connect to, and then jump from there to their desired destination, using BLESS to issue a short lived cert at that point.

As such the client script is expecting some details on where you are coming from and going to, so that BLESS can log this information in AWS CloudWatch logs for auditing purposes.

Get your local public ssh key signed by your BLESS lambda function:

$ ./bless_client.py region lambda_function_name bastion_user bastion_user_ip remote_usernames bastion_ips bastion_command <id_rsa.pub to sign> <output id_rsa-cert.pub>

View your signed certificates details:

$ ssh-keygen -L -f id_rsa-cert.pub

Now try and connect to your test instance via SSH, passing both your pub key and the signed pub key:

$ ssh <REMOTE USERNAME>@<REMOTE IP> -i <output id_rsa-cert.pub> -i <id_rsa.pub to sign>

Conclusion

I’ve only touched on what you can do with BLESS here in this post, but I’d encourage anyone interested in securing your systems in this manner to watch Russell’s YouTube video & give it a go.

Next I’m going to explore some of the other abilities SSH Certificates & BLESS can give you. Such as:

1. Create a bastion host to jump from and log the session details.
2. Setup host certificates; so you never get that unknown host warning on valid servers.
3. Setup log auditing and alarms for improper SSH use.

Further Reading

1. Creating SSH Certificates - Red Hat Docs
2. Netflix BLESS README.md
3. Blog by Mike Milone advocating the use of SSH Certificates