Tailor Kubernetes to your needs with CRDs

A step-by-step tutorial how to extend Kubernetes using a Kubernetes Operator, Custom Resource Definitions and Initializer Configuration

Introduction

Kubernetes has built-in APIs that can drive the most common use-cases of deployment and operations of containers on production. But when you want to take the less common road, you need the necessary tools for the job. This is were Custom Resource Definitions com into place. They provide a way to define your own APIs schemas inside the Kubernetes REST API. And in order for those APIs to actually do something you can use Kubernetes Operator Pattern to watch and act on those shiny new APIs. Lastly you can leverage Dynamic Admission Controllers and more specifically Initializers to manage the lifecycle of those resources.

Photo Quino Al on Unsplash

What are we building?

In Kubernetes there is currently no way to grant two separate users the ability to create namespaces without granting them access to other user’s namespaces.

We’ll build a way to have multitenancy with dynamic namespaces.

This of course is for the purposes of the tutorial, since hard multitenancy in Kubernetes is not a solved issue yet.

How will it work?

We’ll build a Project API in Kubernetes. Users inside a Namespace (e.g. greenteam). Each Project created will trigger the creation of 3 Namespaces per Project prefixed by the Namespace where the Project is created (e.g. greenteam-g1-dev, greenteam-g1-pre, greenteam-g1-pro) and grant RoleBinding to specific User.

So Users that can manage Projects inside greenteam namespace can effectively create namespaces under the greenteam prefix and assign RoleBindings to those namespaces.

Project Initializer is responsible for creating Namespaces and granting RoleBindings

Getting Started

First of all let’s download the code for the tutorial:

➜ git clone https://github.com/harbur/kubernetes-project-initializer-tutorial.git
➜ cd kubernetes-project-initializer-tutorial

Create Kubernetes Cluster

We’ll need a Kubernetes Cluster to deploy to. The Cluster needs to support Initializers which is still an alpha feature.

• If you want to use a pre-existing cluster make sure you have Initializers activated:

➜ kubectl get initializerconfiguration
No resources found.

• If you want to use minikube to launch locally Kubernetes use the following command:

➜ minikube start --extra-config=apiserver.runtime-config=admissionregistration.k8s.io/v1alpha1

• If you want to launch a Kubernetes cluster in Google Kubernetes Engine make sure you have alpha features enabled:

➜ gcloud container clusters create k1 \
  --enable-kubernetes-alpha \
  --no-enable-autorepair \
  --no-enable-autoupgrade

Once the cluster is created in GKE, make sure your user has sufficient privileges to manage RBAC inside the Cluster with the following command:

➜ kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account)

Configure RBAC

A new ClusterRole project-admin is defined which has aggregated privileges of edit role and can also manage projects. Aggregated ClusterRoles are used to define the RBAC.

➜ kubectl create -f rbac/
clusterrole.rbac.authorization.k8s.io/aggregate:projects created
clusterrole.rbac.authorization.k8s.io/project-admin created

Once created you can see the ClusterRoles defined:

➜ kubectl get clusterrole -l app=project-initializer
NAME                 AGE
aggregate:projects   1m
project-admin        1m
➜ kubectl describe clusterrole project-admin
  ...
  Resources                Non-Resource URLs  Resource Names  Verbs
  ---------                -----------------  --------------  -----
  projects.tutorial.harbur.io []                 []              [*]
  ...

Create Projects Custom Resource

Let’s create the Project Custom Resource Definition.

Project contains spec.owner field that is used to grant access of the namespaces to that user.

➜ kubectl create -f crd/projects-crd.yaml
customresourcedefinition.apiextensions.k8s.io/projects.tutorial.harbur.io created

Once created you can see the created CRD:

➜ kubectl get crd
NAME                          CREATED AT
projects.tutorial.harbur.io   2018-09-28T10:11:05Z

Deploy The Project Initializer

The Project Initializer is a Kubernetes Initializer that automates the creation of the Project namespaces and grants privileges to the appropriate Users.

To build the Project Initializer controller:

./project-initializer/build-container

This will build and push the container image to Docker Hub.

The Image is already built and pushed so you can skip this step.

Deploy the project-initializer controller:

➜ kubectl apply -f project-initializer-k8s/
clusterrolebinding.rbac.authorization.k8s.io/add-on-cluster-admin created
serviceaccount/project-initializer created
deployment.apps/project-initializer created

A ServiceAccount project-initializer was created on kube-system namespace and was given cluster-admin privileges:

➜ kubectl get clusterrolebinding -l app=project-initializer
NAME                   AGE
add-on-cluster-admin   1m
➜ kubectl describe clusterrolebinding -l app=project-initializer
Name:         add-on-cluster-admin
Labels:       app=project-initializer
Annotations:  ...
Role:
  Kind:  ClusterRole
  Name:  cluster-admin
Subjects:
  Kind            Name                 Namespace
  ----            ----                 ---------
  ServiceAccount  project-initializer  kube-system

Also a deployment project-initializer was created on kube-system namespace:

➜ kubectl logs -n kube-system -l app=project-initializer
2018/09/28 14:23:09 using in-cluster configuration
2018/09/28 14:23:09 Starting the Kubernetes project initializer...
2018/09/28 14:23:09 Initializer name set to: project.initializer.kubernetes.io

Initializing Projects

In this section you will create an InitializerConfiguration that initializes Projects:

➜ kubectl create -f initializer-configurations/project.yaml
initializerconfiguration.admissionregistration.k8s.io/project created

At this point the Project Initializer is ready to initialize new Projects.

We created a project InitializerConfiguration:

➜ kubectl get initializerconfiguration
NAME      CREATED AT
project   2018-09-28T14:25:37Z

Demo

Now that we have the Project Initializer ready let’s see how it works.

Create Team Namespaces

We have two teams: blueteam and greenteam. For each team we'll use a separate namespaces to manage their projects:

➜ kubectl create ns blueteam
namespace/blueteam created
➜ kubectl create ns greenteam
namespace/greenteam created

• bob User is part of teamblueteam
• alice User is part of teamgreenteam

Grant project-admin RoleBindings

Let’s grant project-admin privileges to bob and alice on the namespace of their team.

➜ kubectl create rolebinding bob-project-admin \
  --clusterrole=project-admin \
  --user bob \
  -n blueteam
rolebinding.rbac.authorization.k8s.io/bob-project-admin created
➜ kubectl create rolebinding alice-project-admin \
  --clusterrole=project-admin \
  --user alice \
  -n greenteam
rolebinding.rbac.authorization.k8s.io/alice-project-admin created

Now bob User can manage Projects inside blueteam Namespace:

➜ kubectl get projects -n blueteam --as bob
No resources found.

While alice User cannot see Projects in the blueteam Namespace (alice belongs to greenteam)

➜ kubectl get projects -n blueteam --as alice
Error from server (Forbidden): projects.tutorial.harbur.io is forbidden: User "alice" cannot list projects.tutorial.harbur.io in the namespace "blueteam": Unknown user "alice"

Bob creates Project for Blueteam

Bob can now create a project inside blueteam

➜ kubectl create -f crd/blueteam-b1-project.yaml --as bob
project.tutorial.harbur.io/b1 created

If we check now the logs of the project-initializer operator we can see that it has created the Namespaces and the RoleBindings, and we can also check that Bob can operate inside the new Namespaces:

➜ kubectl logs -n kube-system -l app=project-initializer
2018/10/05 13:20:42 using in-cluster configuration
2018/10/05 13:20:42 Starting the Kubernetes project initializer...
2018/10/05 13:20:42 Initializer name set to: project.initializer.kubernetes.io
2018/10/05 13:31:18 Initializing Project b1 (blueteam)
2018/10/05 13:31:18 - Created Namespace blueteam-b1-dev
2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-dev)
2018/10/05 13:31:18 - Created Namespace blueteam-b1-pre
2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-pre)
2018/10/05 13:31:18 - Created Namespace blueteam-b1-pro
2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-pro)

➜ kubectl get namespaces
NAME              STATUS    AGE
blueteam          Active    13m
blueteam-b1-dev   Active    3m
blueteam-b1-pre   Active    3m
blueteam-b1-pro   Active    3m
...

➜ kubectl get pod -n blueteam-b1-dev --as blueteam
No resources found.

Cleaning Up

To cleanup the demo

kubectl delete ns blueteam-b1-{dev,pre,pro}
kubectl delete project b1 -n blueteam
kubectl delete ns blueteam
kubectl delete -f crd/projects-crd.yaml
kubectl delete -f project-initializer-k8s/
kubectl delete -f initializer-configurations/
kubectl delete -f rbac

If you created a cluster in GKE make sure to destroy it:

➜ gcloud container clusters delete k1

Wrap-up

We’ve used CRD in order to define a new API schema called Project in Kubernetes. We used Kubernetes Operator Pattern to deploy a controller inside Kubernetes that monitors those resources. And finally we used Initializers in order to force initialization of the projects to pass through the controller.

This is just the tip of the iceberg with the possibilities of extending Kubernetes with custom controllers and we didn’t even cover Custom Metrics.