Open Sourcing Our SSH Toolchain

Eduardo Lopez
Dec 3, 2018 · 5 min read

Managing entries in an authorized_keys file for each person who has access on each server is quite painful, especially in modern cloud environments where infrastructure is increasingly dynamic. Unfortunately, what this means is that teams often default to having a single long-lived key shared by multiple people. This makes non-repudiation, rotation, or revocation hard or even impossible. The longer these static keys exist, the more likely it is they will be compromised — along with all the infrastructure they were supposed to protect. All these factors amount to significant risk.

To mitigate these risks, there are several approaches you can take. Use a configuration management system, such as Chef or Ansible, to manage authorized_keys for each user on each server. This means that keys are long-lived and every time a key needs to be added, rotated, or revoked, an ops team must push updates to the entire fleet of servers. Another approach is to lean on cloud providers, but sometimes all you can do is set a single SSH key per machine.

Instead, at CZI we decided to use SSH certificates to solve for these issues. Today we’re open sourcing blessclient, terraform-provider-bless, and a BLESS terraform module; a set of tools that simplify working with SSH certificates.

SSH certificates are an extension recently added to OpenSSH that allow Certificate Authorities (CAs) to sign public keys. When a CA signs a key, it returns a certificate specifying constraints such as time based or principal validity. Users can then present this certificate alongside their public key to the server for authentication. If the server trusts the CA and all the certificate constraints are satisfied, SSH connections are allowed to proceed as normal.

Some nice properties of SSH certificates:

  • Access to servers can be dynamically granted with a time-based expiration.
  • Servers do not need any prior knowledge of a user’s public key; they only need to trust the CA.
  • Servers do not need any special software installed; only slight modifications to sshd configuration are needed.
  • Stateless; multi region availability and geographic co-location are possible with little work.

Because of these, our team can quickly onboard and offboard users. Individuals can rotate their keys whenever they want to without ever having to engage infrastructure or security teams. We can issue individual, short lived certificates to mitigate the risk of private key leakage while ensuring rich individual accountability.


Netflix’s BLESS is an SSH Certificate Authority (CA) that runs as an AWS Lambda function. Users send their public key to the lambda where the Certificate Authority signs it and returns an SSH certificate. Users can then use this certificate to gain access to servers that trust the corresponding CA. It is important to note that the CA becomes the root of trust for SSH. Servers need only to trust the CA and need no prior knowledge of users’ keys. This also means that protecting the CA private key is incredibly important — a compromised CA could grant arbitrary access to those who possess its private key.

There are several great guides that explain how to setup BLESS. If you take a moment to skim through these, you’ll notice that setting up a successful BLESS deployment requires thorough knowledge of AWS Lambda, KMS, and IAM. Even then, you’ll probably spend hours digging through CloudWatch logs (and who likes doing that).

If you’ve been following our blog, you might know that we use fogg and Terraform modules to manage all our infrastructure. Automation and Open Source are two ways our team can scale its impact across all our partner organizations. We therefore set out to create a Terraform module that would automate the deployment of BLESS and associated resources.

There are two main challenges when creating a self-contained BLESS Terraform module:

  • Preventing your CA private key from leaking into the Terraform state store.
  • Packaging a configured BLESS lambda that accepts configuration input from Terraform.

As mentioned above, protecting the CA private key is incredibly important. Ideally nothing but BLESS lambda should have direct access to the CA private key. If the private key were to leak into the Terraform remote store, those with access to the state store would be able to directly use the CA key to mint SSH certificates and gain arbitrary access to servers.

To solve these, we created terraform-provider-bless. This custom provider abstracts away the complexity of generating the CA and encrypting it with KMS; all without leaking any sensitive material to the Terraform state store. It also accepts input from Terraform, creates the BLESS configuration file, and generates a zip archive that Terraform can upload to AWS Lambda.

Using our module and terraform-provider-bless, a BLESS deployment might look like:

// configure terraform-provider-bless
provider "bless" {
region = "<aws_region>"
profile = "<my_profile>"

// BLESS and associated resources
module "bless" {
source = ""

// resource tagging
project = "${var.project}"
service = "bless"
env = "prod"
owner = "${var.owner}"

// arn of users authorized to use BLESS
authorized_users = [...]


Now that we have BLESS deployed, how do people interact with it? For this part of the puzzle, we turned to python-blessclient. python-blessclient is a great project open sourced by Lyft that handles certificate negotiation with the BLESS lambda. As we started ramping up usage across other teams within CZI, it quickly became apparent that distributing the python-blessclient configuration and managing the required python environments is cumbersome. Especially without relying on sophisticated laptop configuration management tools.

To solve these, we decided to port python-blessclient to Go. There are several reasons why Go is a good fit for this project:

  • Go allows you to generate statically linked binaries; users can just drop the executable in their PATH and start using blessclient. No need to download, compile, or install extra dependencies.
  • Go allows you to cross-compile binaries; we can easily publish binaries for Mac and Linux.
  • Go standard library has good ssh support; we can leverage these to write more nuanced logic around ssh agent, certificates, and keys.

The result of this work is blessclient. Through blessclient you can transparently negotiate ephemeral SSH certificates with BLESS.

What’s next

We have successfully been using blessclient at CZI for the last couple of months. Open source is one of the core tenets of our infrastructure team and we will continue making improvements to blessclient and associated tools. We hope that you find blessclient useful. Contributions are always welcome — please don’t hesitate to open an issue or send a pull request!

CZI Technology

The Technology team at Chan Zuckerberg Initiative

Eduardo Lopez

Written by

CZI Technology

The Technology team at Chan Zuckerberg Initiative

More From Medium

More from CZI Technology

More from CZI Technology

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade