Kubernetes API Aggregation Setup — Nuts & Bolts
This article takes a stab at uncovering some of the details associated with the control flows that occur when one deploys an extension apiserver with the aggregation layer.
Concepts/Keywords:
- extension-apiserver: API servers written using the library kube-apiserver. Provides a way of defining custom APIs that are deeply integrated with core Kubernetes API machinery. Compared to CRDs, it offers more features and options for dealing with performance, policy and customization.
- kube-aggregator: It is the controller associated with apiregistration API and constitutes what we call the aggregation layer. It is responsible for discovering and registering extension apiservers and acting as a front-proxy to the individual extension apiservers.
- delegated authN/Z: The mechanism where-in API calls to the extension-apiserver are authenticated & authorized by the core kubernetes apiserver (kubernetes master)
For the purposes of demo, I will use the kubernetes sample-apiserver and the various deployment yamls it presents to illustrate the various control flow variables as well.
Lets say you have built the extension-apiserver sample-apiserver; it comes with the custom resources Flunder & Fischer under the wardle.k8s.io apigroup. You then deploy this alongside a kubernetes master, that has the aggregation layer configured, using the provided example artifacts (kubectl apply -f artifcats/example). And it all just works!
kubectl get wardle.k8s.io/flunder ; kubectl get wardle.k8s.io/fischer
But, what does having the “aggregation layer” configured mean? What do the “example” artifacts exactly set and enable? And how does the delegated authentication and authorization work?
Exactly what I will look to answer.
Primer on Authentication mechanisms
Before going further with the control flows associated with an extension-apiserver deployment, its essential to clear some concepts around the three primary authentication mechanisms that make up an extension-apiservers delegated authentication setup:
Client Certificate Authentication: In this mechanism, the client sends a certificate signed by a CA that the server trusts for validating the identity of the client.
The server request authentication decorator could read something like the below if it were written in Go:
func renewCert(w http.ResponseWriter, r *http.Request) {
if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 {
cn := strings.ToLower(r.TLS.PeerCertificates[0].Subject.CommonName)
fmt.Println("CN: %s", cn)
}
}Delegated Token Authentication: In this mechanism, the server extracts the Token present in Authorization: Bearer Token $TOKEN from the HTTP Request and forwards it for a review onto a different server.
In case of an extension-apiserver, the Token is sent for a TokenReview to the master Kubernetes API server.
RequestHeader Client Authentication: In this mechanism, there is a proxy in play, that is responsible for authenticating an API call before it makes to the apiserver.
From the apiserver’s point of view, the requests are coming in from the proxy, but it trusts the proxy to masquerade as any other user. This masquerading takes place via HTTP Header set by the proxy (e.g. X-Remote-User).
Now, lets look at the control flows and configurations associated with deploying an extension-apiserver with the kube-aggregator.
Registering an extension-API server
APIService resource under the group apiregistraion.k8s.io/v1beta1, serviced by the kube-aggregator controller, is the API used for registering extension apiserver. Following is the spec used for sample-apiserver:
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1alpha1.wardle.k8s.io
spec:
insecureSkipTLSVerify: true
group: wardle.k8s.io
groupPriorityMinimum: 1000
versionPriority: 15
service:
name: api
namespace: wardle
version: v1alpha1
When the above APIService resource is created, kube-aggregator, which lives inside of the kube-apiserver, as part of the registration and discovery process, initiates a TLS enabled HTTP2 connection with the sample-apiserver. This HTTP2 transport, going forward, is used for the purposes of proxying authenticated user requests onto the sample-apiserver. i.e. the kube-aggregator is setup to perform RequestHeader Client Authentication.
Configuring the Aggregator
Lets look at the flags that need to be setup for the kube-aggregator to perform RequestHeader Authentication. This is also what primarily constitutes the aggregation layer setup.
proxy-client-cert-file & proxy-client-key-file contain the cert/key pair used by the Aggregator to perform Client Certificate Authentication with an extension-apiserver.
But who/what signs the Client/Aggregator certificate? And how is the extension-apiserver configured to verify with this Client CA?
The Aggregator certificate is signed using the CA file specified in requestheader-client-ca-file .
requestheader-allowed-names carry a list of identities/names (CN used in the Client Certificate) that are allowed to act as a masquerading front-proxy.
requestheader-username-headers, requestheader-group-headers & requestheader-extraheaders-prefix carry a list of HTTP headers that will carry the remote user information and more.
The various requestheader-* variables are shared with the extension-apiserver using the extension-apiserver-authentication configMap. On startup, the extension-apiserver reads the said configMap to obtain the CA file to verify the identity of the Aggregator proxy and process the remote client identity variables.
Note: Since, Kubernetes by default is setup with RBAC, the sample-apiserver's ServiceAccount needs to be bound to the extension-apiserver-authentication-reader. Here's the yaml which does it.
What does the extension apiserver do with the remote client info obtained off the various requestheaders?
Once, the identity is obtained, the extension apiserver uses it perform a delegated authorization on the request.
Note: The following role binding is needed for the extension-apiserver to perform a authN/Z delegation request. And, that, pretty much sums up the Aggregation Layer.
Setting the extension-apiserver Cert
In the APIService object, insecureSkipTLSVerify is set to true. Meaning, the client (kube-aggregator) will skip verifying the identity of the extension-apiserver. It is recommended that we set this to false and the cabundle field in the spec is set to the base-64 encoded CA cert that was used to sign the extension-apiserver certificate. The kube-aggregator will then use the cabundle to verify the sample-apiserver certificate.
Note: Make sure to mount the signed cert/key onto the extension-apiserver Pod. The path used for reading the cert/key pair is as set in the argument --cert-dir to the apiserver which currently defaults to `/apiserver.local.config/certificates`.
One could also, instead, use the flags --tls-cert-file and --tls-private-key to set the cert/key for the apiserver.