Photo by Matt Artz on Unsplash

Dynamic Certificate Management for Cloudflare Origins in Go

Today I integrated my origin border service with the Cloudflare Origin CA API to dynamically provision, rotate, and revoke service certificates. Now our border uses short-lived secrets without involving any human hands, or writing any certificates/keys to disk. Here is the project:

gotopple/cf-origin-cert

At Topple we use Cloudflare for secure DNS and TLS. Cloudflare terminates inbound TLS connections using certificates in the public trust. But it also uses TLS to secure connections to your backend services. You can configure connections from Cloudflare to your services to use mutually authenticated TLS. That lets your services authenticate inbound connections as originating from Cloudflare, and Cloudflare can authenticate your servers. Wonderful.

Today I automated provisioning, rotation, and revocation of certificates for our ingress services (called origins). Those certificates don’t need to be in the public trust chain, they only need to be trusted by Cloudflare. A solution like Let’s Encrypt uses a bunch of challenge methods to automate creating certificates in the public trust chain. But those methods can be a bit complicated (and they should be) compared to working with Cloudflare’s private Origin Certificate Authority. Creating and revoking origin certificates with Cloudflare’s API are simple authenticated API calls.

The API lets us automate everything and use short-lived certificates. I’m a fan of short-lived secrets.

We’re fans of configuration-as-code and keeping our systems simple. We shoot for small single-binary stacks when possible and avoid side-cars. We’re also averse to VC funded open source dependencies. That means when we can avoid a high-risk technical debt obligation we will. NGINX is just not something we need. Our borders use a popular Go OSS project for creating high-performance reverse proxies. Ironically, this ends up being much less work and results in a much simpler, fully instrumented service. But I’m getting distracted…

Manual origin certificate provisioning is a silly thing to do. Especially considering there is a perfectly good API (well not so perfect). So I created a little embeddable agent to do that work for us. That agent uses the API to provision a new certificate on startup, periodically rotates the certificate, revokes old certificates after successful rotations, and revokes the currently held certificates on shutdown.

A service that used this project to generate its own origin certificates on the fly would look like:

import (
    "github.com/gotopple/cf-origin-cert"
)
func SomeServiceFunction(...) {
    ...
    // setup the agent
    ca, err := agent.NewCertAgent(
        apiKey, 
        rotationFrequency, 
        certTTL,
        cacheDepth)
    // start the agent
    stopper := make(chan struct{})
    go ca.Run(myTLD, stopper)
    ...
    // at some point fetch the current cert material
    creds, err := ca.GetCertKeyPair(0)
    // increment to reach further back in the credential history
    oldCreds, err := ca.GetCertKeyPair(1)
    ...
    // do something foolish with the creds
    fmt.Println(creds.CertPEM)
    fmt.Println(creds.Key)
    ...
    // stop and cleanup
    close(stopper)
    // wait a few seconds for the cert revocation calls
    ...
}

Feel free to checkout the code. It is a solid proof-of-concept. At Topple we won’t use the standalone binary. Instead we’ll embed the agent in our service and coordinate a safe transition between rotated certificates. There is only one issue… the current API and official client are incompatible on the CreateOriginRequest call.

JSON Unmarshall error when using CreateOriginCertificate · Issue #190 · cloudflare/cloudflare-go

I opened a community topic here to discuss the service API issue and proposed fix:

Create Origin Certificate API call returns incorrect expires_on timestamp format

To work around the current API/Client contract discrepancy I initially made a garbage fork of the Go client library that treats the expires_on field as a string (I’m not using it anyway). When the bugs above are worked out I will just switch back to the official library. Actually, I might just implement the two calls I use with net/http. Who needs the extra library dependency?

Old-school certificate management is a human process nightmare. Thanks to Snowden and a huge anti-surveillance movement in 2012–2014 the world was blessed with new ways to reduce the operational pain of maintaining HTTPS everywhere.

Cloudflare is one of my favorite clouds because they go above and beyond to make operating secure services simple.

If you’re into confidentiality

Please consider supporting an organization like the Electronic Frontier Foundation, and their projects like HTTPS Everywhere.