Net::HTTP and x509 Client Certificate Chains and Monkey Patches! Oh My!

On the Infrastructure team here at Greenhouse, we’re in the midst of migrating most of our internal services onto Kubernetes. As part of that work, we’re adding Kubernetes support to our internal application platform, Dajoku. This will allow us to deploy all of our public facing applications onto Kubernetes as well. Part of that work involves getting Dajoku to talk to the Kubernetes API.

Dajoku is a Ruby application, so making this happen should have been as easy as adding kubeclient to our Gemfile and running bundle install . Unfortunately, due to a bug in how Ruby handles SSL certificates, getting this to work didn’t go as smoothly as we had hoped.

The Problem

The primary way of interacting with Kubernetes is through its command line client, kubectl. The config file for kubectl is just YAML, and it contains information about each Kubernetes cluster you’ve configured. Ours looks something like this:

Here we can see there are two clusters, dev which is our development cluster running on AWS and minikube which is a local minikube cluster for local development. One thing to note in the kubectl config is that we’re using x509 client certificates for authentication. This will be important later.

Luckily, kubeclient can read this config file, which makes our lives a little easier. Let’s try using kubeclient to connect to the dev cluster:

Well, that’s odd. Let’s make sure kubectl works:

Let’s try curl, with the client certificate:

Just to be sure, let’s try curl without the client certificate and confirm that 401 means we’re not authenticated:

Maybe there’s a problem with kubeclient or the way it’s reading our kubectl config? Let’s try something a little more low-level in Ruby. Here’s the same HTTP request that we did with curl but using just Net::HTTP:

OK, what is going on here? The same HTTP request succeeds with curl and fails with Net::HTTP? Could this be a bug in Ruby? That seems very unlikely. We’ve got another Kubernetes cluster, minikube, to test against. Let’s try that just to be sure. We need to add one more argument to curl here, to tell it to trust the Kubernetes API’s certificate authority:

That succeeds, how about with Net::HTTP:

And that succeeds.

Let’s summarize what we’ve found so far:

• We have two Kubernetes clusters, dev and minikube, both using x509 client certificates for authentication.
• We can use kubectl to connect to both clusters’ APIs.
• We can use curl to connect to both clusters’ APIs.
• We can use Ruby to connect to minikube, but not dev.

Maybe there’s something different about the client certificates, let’s take a look:

Now we’re getting somewhere! There are actually two certificates in the dev API’s client certificate file.

A Brief Digression into PKI

So why does the dev cluster’s client certificate file contain more than one certificate? This has to do with how we’ve designed our public key infrastructure (PKI). For internal services, we generate our own certificates. And for that purpose, we’ve created two certificate authorities (CAs), a root CA and an intermediate CA.

The root CA’s certificate is distributed to, and trusted by, all of our infrastructure. To ensure the security of the root CA, we only actually use it once, to issue the intermediate CA certificate. Then we stow the root CA’s private key on a USB stick in a safe deposit box somewhere.

The intermediate CA is then used to issue certificates including the client certificates that we use for authentication with Kubernetes.

Although this might seem unnecessarily complex, this improves the overall security of our PKI. If the intermediate CA were ever compromised we could revoke its certificate and issue a new one without having to redistribute the root CA’s certificate to all of our infrastructure.

And this is why the client certificate file for the dev cluster contains two certificates, the client certificate, and the intermediate CA certificate. Is Ruby not reading the whole cert file? Lets check in irb:

That is a problem. Ruby ignores the intermediate certificate and only sends the client certificate to the Kubernetes API and that’s why the certificate validation fails.

How Do We Fix This?

After much googling, I stumbled on this vaguely related StackOverflow answer with a reference to SSLExtraChainCert which seems like a good thing to search for. Searching the Ruby documentation turns up a similar attribute,OpenSSL::SSL::SSLContext#extra_chain_cert. Lets see if we can figure out how to use that.

How does SSLContext get used by Net::HTTP? Let’s go to the source code to find out:

In Net::HTTP#connect, we’re iterating over all the instance variable names in Net::HTTP::SSL_IVNAMES and building up a Hash of SSL parameters (ssl_parameters in the code above). But something is missing! extra_chain_cert isn’t in the array of instance variable names!

Some more searching finds this Ruby issue from 2015, which helpfully includes a patch. We can convert this to a monkey patch and test it out:

Now let’s try this again, this time we’ll split the client certificate data and pass the intermediate certificate in extra_chain_certs:

This looks promising. Let’s keep going!

Getting Closer to a Fix

We’ve got a fix for Net::HTTP, but remember, we’re using kubeclient to talk to Kubernetes, so we’re going to have to fix everything above Net::HTTP in the stack as well.

Kubeclient uses rest-client, so we’ll need to figure out how to pass the extra_chain_certoption down to Net::HTTP. rest-client maintains its own list of SSL options in a constant, RestClient::Request::SSLOptionList. We’ll need to add extra_chain_cert to that list and then use that option when RestClient::Request creates an instance of Net::HTTP.

First, let’s modify SSLOptionList :

Next, we can override RestClient::Request#net_http_object to set extra_chain_cert. For this, we’ll use Module#prepend to insert our method into the ancestors of RestClient::Request:

Now, we’ll need to monkey patch kubeclient as well. First, we need to patch Kubeclient::Config so it can parse the intermediate certificate from the client certificate file:

Then we’ll patch Kubeclient::Client to use the extra_chain_cert from theKubeclient::Config when it creates a RestClient::Resource for accessing the Kubernetes API.

Unfortunately, we’re going to have to duplicate nearly all of the code that creates the RestClient::Resource :

Finally, we can try this again:

Next Steps

The monkey patches above are, at best, a temporary solution to the problem. They’ll likely break in future versions of rest-client or kubeclient. I’ve reached out to the ruby-core list to see what needs to be done to get the Net::HTTP patch merged. Once that’s merged, we can open pull requests against rest-client and kubeclient.