Net::HTTP and x509 Client Certificate Chains and Monkey Patches! Oh My!
On the Infrastructure team here at Greenhouse, we’re in the midst of migrating most of our internal services onto Kubernetes. As part of that work, we’re adding Kubernetes support to our internal application platform, Dajoku. This will allow us to deploy all of our public facing applications onto Kubernetes as well. Part of that work involves getting Dajoku to talk to the Kubernetes API.
Dajoku is a Ruby application, so making this happen should have been as easy as adding kubeclient to our
Gemfile and running
bundle install . Unfortunately, due to a bug in how Ruby handles SSL certificates, getting this to work didn’t go as smoothly as we had hoped.
The primary way of interacting with Kubernetes is through its command line client,
kubectl. The config file for
kubectl is just YAML, and it contains information about each Kubernetes cluster you’ve configured. Ours looks something like this:
Here we can see there are two clusters,
dev which is our development cluster running on AWS and
minikube which is a local minikube cluster for local development. One thing to note in the
kubectl config is that we’re using x509 client certificates for authentication. This will be important later.
kubeclient can read this config file, which makes our lives a little easier. Let’s try using
kubeclient to connect to the
Well, that’s odd. Let’s make sure kubectl works:
Let’s try curl, with the client certificate:
Just to be sure, let’s try curl without the client certificate and confirm that 401 means we’re not authenticated:
Maybe there’s a problem with
kubeclient or the way it’s reading our
kubectl config? Let’s try something a little more low-level in Ruby. Here’s the same HTTP request that we did with curl but using just
OK, what is going on here? The same HTTP request succeeds with curl and fails with
Net::HTTP? Could this be a bug in Ruby? That seems very unlikely. We’ve got another Kubernetes cluster,
minikube, to test against. Let’s try that just to be sure. We need to add one more argument to curl here, to tell it to trust the Kubernetes API’s certificate authority:
That succeeds, how about with
And that succeeds.
Let’s summarize what we’ve found so far:
- We have two Kubernetes clusters,
minikube, both using x509 client certificates for authentication.
- We can use
kubectlto connect to both clusters’ APIs.
- We can use curl to connect to both clusters’ APIs.
- We can use Ruby to connect to
minikube, but not
Maybe there’s something different about the client certificates, let’s take a look:
Now we’re getting somewhere! There are actually two certificates in the
dev API’s client certificate file.
A Brief Digression into PKI
So why does the
dev cluster’s client certificate file contain more than one certificate? This has to do with how we’ve designed our public key infrastructure (PKI). For internal services, we generate our own certificates. And for that purpose, we’ve created two certificate authorities (CAs), a root CA and an intermediate CA.
The root CA’s certificate is distributed to, and trusted by, all of our infrastructure. To ensure the security of the root CA, we only actually use it once, to issue the intermediate CA certificate. Then we stow the root CA’s private key on a USB stick in a safe deposit box somewhere.
The intermediate CA is then used to issue certificates including the client certificates that we use for authentication with Kubernetes.
Although this might seem unnecessarily complex, this improves the overall security of our PKI. If the intermediate CA were ever compromised we could revoke its certificate and issue a new one without having to redistribute the root CA’s certificate to all of our infrastructure.
And this is why the client certificate file for the
dev cluster contains two certificates, the client certificate, and the intermediate CA certificate. Is Ruby not reading the whole cert file? Lets check in irb:
That is a problem. Ruby ignores the intermediate certificate and only sends the client certificate to the Kubernetes API and that’s why the certificate validation fails.
How Do We Fix This?
After much googling, I stumbled on this vaguely related StackOverflow answer with a reference to
SSLExtraChainCert which seems like a good thing to search for. Searching the Ruby documentation turns up a similar attribute,
OpenSSL::SSL::SSLContext#extra_chain_cert. Lets see if we can figure out how to use that.
SSLContext get used by
Net::HTTP? Let’s go to the source code to find out:
Net::HTTP#connect, we’re iterating over all the instance variable names in
Net::HTTP::SSL_IVNAMES and building up a
Hash of SSL parameters (
ssl_parameters in the code above). But something is missing!
extra_chain_cert isn’t in the array of instance variable names!
Some more searching finds this Ruby issue from 2015, which helpfully includes a patch. We can convert this to a monkey patch and test it out:
Now let’s try this again, this time we’ll split the client certificate data and pass the intermediate certificate in
This looks promising. Let’s keep going!
Getting Closer to a Fix
We’ve got a fix for
Net::HTTP, but remember, we’re using
kubeclient to talk to Kubernetes, so we’re going to have to fix everything above
Net::HTTP in the stack as well.
Kubeclient uses rest-client, so we’ll need to figure out how to pass the
extra_chain_certoption down to
rest-client maintains its own list of SSL options in a constant,
RestClient::Request::SSLOptionList. We’ll need to add
extra_chain_cert to that list and then use that option when
RestClient::Request creates an instance of
First, let’s modify
Next, we can override
RestClient::Request#net_http_object to set
extra_chain_cert. For this, we’ll use
Module#prepend to insert our method into the ancestors of
Now, we’ll need to monkey patch
kubeclient as well. First, we need to patch
Kubeclient::Config so it can parse the intermediate certificate from the client certificate file:
Then we’ll patch
Kubeclient::Client to use the
extra_chain_cert from the
Kubeclient::Config when it creates a
RestClient::Resource for accessing the Kubernetes API.
Unfortunately, we’re going to have to duplicate nearly all of the code that creates the
Finally, we can try this again:
The monkey patches above are, at best, a temporary solution to the problem. They’ll likely break in future versions of rest-client or kubeclient. I’ve reached out to the ruby-core list to see what needs to be done to get the
Net::HTTP patch merged. Once that’s merged, we can open pull requests against rest-client and kubeclient.