Tailor Kubernetes to your needs with CRDs
A step-by-step tutorial how to extend Kubernetes using a Kubernetes Operator, Custom Resource Definitions and Initializer Configuration
Introduction
Kubernetes has built-in APIs that can drive the most common use-cases of deployment and operations of containers on production. But when you want to take the less common road, you need the necessary tools for the job. This is were Custom Resource Definitions com into place. They provide a way to define your own APIs schemas inside the Kubernetes REST API. And in order for those APIs to actually do something you can use Kubernetes Operator Pattern to watch and act on those shiny new APIs. Lastly you can leverage Dynamic Admission Controllers and more specifically Initializers to manage the lifecycle of those resources.
What are we building?
In Kubernetes there is currently no way to grant two separate users the ability to create namespaces without granting them access to other user’s namespaces.
We’ll build a way to have multitenancy with dynamic namespaces.
This of course is for the purposes of the tutorial, since hard multitenancy in Kubernetes is not a solved issue yet.
How will it work?
We’ll build a Project API in Kubernetes. Users inside a Namespace (e.g. greenteam
). Each Project created will trigger the creation of 3 Namespaces per Project prefixed by the Namespace where the Project is created (e.g. greenteam-g1-dev
, greenteam-g1-pre
, greenteam-g1-pro
) and grant RoleBinding to specific User.
So Users that can manage Projects inside greenteam
namespace can effectively create namespaces under the greenteam
prefix and assign RoleBindings to those namespaces.
Getting Started
First of all let’s download the code for the tutorial:
➜ git clone https://github.com/harbur/kubernetes-project-initializer-tutorial.git
➜ cd kubernetes-project-initializer-tutorial
Create Kubernetes Cluster
We’ll need a Kubernetes Cluster to deploy to. The Cluster needs to support Initializers which is still an alpha feature.
- If you want to use a pre-existing cluster make sure you have Initializers activated:
➜ kubectl get initializerconfiguration
No resources found.
- If you want to use minikube to launch locally Kubernetes use the following command:
➜ minikube start --extra-config=apiserver.runtime-config=admissionregistration.k8s.io/v1alpha1
- If you want to launch a Kubernetes cluster in Google Kubernetes Engine make sure you have alpha features enabled:
➜ gcloud container clusters create k1 \
--enable-kubernetes-alpha \
--no-enable-autorepair \
--no-enable-autoupgrade
Once the cluster is created in GKE, make sure your user has sufficient privileges to manage RBAC inside the Cluster with the following command:
➜ kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user $(gcloud config get-value account)
Configure RBAC
A new ClusterRole project-admin
is defined which has aggregated privileges of edit
role and can also manage projects
. Aggregated ClusterRoles are used to define the RBAC.
➜ kubectl create -f rbac/
clusterrole.rbac.authorization.k8s.io/aggregate:projects created
clusterrole.rbac.authorization.k8s.io/project-admin created
Once created you can see the ClusterRoles defined:
➜ kubectl get clusterrole -l app=project-initializer
NAME AGE
aggregate:projects 1m
project-admin 1m
➜ kubectl describe clusterrole project-admin
...
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
projects.tutorial.harbur.io [] [] [*]
...
Create Projects Custom Resource
Let’s create the Project Custom Resource Definition.
Project contains spec.owner
field that is used to grant access of the namespaces to that user.
➜ kubectl create -f crd/projects-crd.yaml
customresourcedefinition.apiextensions.k8s.io/projects.tutorial.harbur.io created
Once created you can see the created CRD:
➜ kubectl get crd
NAME CREATED AT
projects.tutorial.harbur.io 2018-09-28T10:11:05Z
Deploy The Project Initializer
The Project Initializer is a Kubernetes Initializer that automates the creation of the Project namespaces and grants privileges to the appropriate Users.
To build the Project Initializer controller:
./project-initializer/build-container
This will build and push the container image to Docker Hub.
The Image is already built and pushed so you can skip this step.
Deploy the project-initializer
controller:
➜ kubectl apply -f project-initializer-k8s/
clusterrolebinding.rbac.authorization.k8s.io/add-on-cluster-admin created
serviceaccount/project-initializer created
deployment.apps/project-initializer created
A ServiceAccount project-initializer
was created on kube-system
namespace and was given cluster-admin
privileges:
➜ kubectl get clusterrolebinding -l app=project-initializer
NAME AGE
add-on-cluster-admin 1m
➜ kubectl describe clusterrolebinding -l app=project-initializer
Name: add-on-cluster-admin
Labels: app=project-initializer
Annotations: ...
Role:
Kind: ClusterRole
Name: cluster-admin
Subjects:
Kind Name Namespace
---- ---- ---------
ServiceAccount project-initializer kube-system
Also a deployment project-initializer
was created on kube-system
namespace:
➜ kubectl logs -n kube-system -l app=project-initializer
2018/09/28 14:23:09 using in-cluster configuration
2018/09/28 14:23:09 Starting the Kubernetes project initializer...
2018/09/28 14:23:09 Initializer name set to: project.initializer.kubernetes.io
Initializing Projects
In this section you will create an InitializerConfiguration that initializes Projects:
➜ kubectl create -f initializer-configurations/project.yaml
initializerconfiguration.admissionregistration.k8s.io/project created
At this point the Project Initializer is ready to initialize new Projects.
We created a project
InitializerConfiguration:
➜ kubectl get initializerconfiguration
NAME CREATED AT
project 2018-09-28T14:25:37Z
Demo
Now that we have the Project Initializer ready let’s see how it works.
Create Team Namespaces
We have two teams: blueteam
and greenteam
. For each team we'll use a separate namespaces to manage their projects:
➜ kubectl create ns blueteam
namespace/blueteam created➜ kubectl create ns greenteam
namespace/greenteam created
bob
User is part of teamblueteam
alice
User is part of teamgreenteam
Grant project-admin RoleBindings
Let’s grant project-admin
privileges to bob
and alice
on the namespace of their team.
➜ kubectl create rolebinding bob-project-admin \
--clusterrole=project-admin \
--user bob \
-n blueteam
rolebinding.rbac.authorization.k8s.io/bob-project-admin created➜ kubectl create rolebinding alice-project-admin \
--clusterrole=project-admin \
--user alice \
-n greenteam
rolebinding.rbac.authorization.k8s.io/alice-project-admin created
Now bob
User can manage Projects inside blueteam
Namespace:
➜ kubectl get projects -n blueteam --as bob
No resources found.
While alice
User cannot see Projects in the blueteam
Namespace (alice
belongs to greenteam
)
➜ kubectl get projects -n blueteam --as alice
Error from server (Forbidden): projects.tutorial.harbur.io is forbidden: User "alice" cannot list projects.tutorial.harbur.io in the namespace "blueteam": Unknown user "alice"
Bob creates Project for Blueteam
Bob can now create a project inside blueteam
➜ kubectl create -f crd/blueteam-b1-project.yaml --as bob
project.tutorial.harbur.io/b1 created
If we check now the logs of the project-initializer operator we can see that it has created the Namespaces and the RoleBindings, and we can also check that Bob can operate inside the new Namespaces:
➜ kubectl logs -n kube-system -l app=project-initializer
2018/10/05 13:20:42 using in-cluster configuration
2018/10/05 13:20:42 Starting the Kubernetes project initializer...
2018/10/05 13:20:42 Initializer name set to: project.initializer.kubernetes.io
2018/10/05 13:31:18 Initializing Project b1 (blueteam)
2018/10/05 13:31:18 - Created Namespace blueteam-b1-dev
2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-dev)
2018/10/05 13:31:18 - Created Namespace blueteam-b1-pre
2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-pre)
2018/10/05 13:31:18 - Created Namespace blueteam-b1-pro
2018/10/05 13:31:18 - Created RoleBinding blueteam-admin (blueteam-b1-pro)
➜ kubectl get namespaces
NAME STATUS AGE
blueteam Active 13m
blueteam-b1-dev Active 3m
blueteam-b1-pre Active 3m
blueteam-b1-pro Active 3m
...
➜ kubectl get pod -n blueteam-b1-dev --as blueteam
No resources found.
Cleaning Up
To cleanup the demo
kubectl delete ns blueteam-b1-{dev,pre,pro}
kubectl delete project b1 -n blueteam
kubectl delete ns blueteam
kubectl delete -f crd/projects-crd.yaml
kubectl delete -f project-initializer-k8s/
kubectl delete -f initializer-configurations/
kubectl delete -f rbac
If you created a cluster in GKE make sure to destroy it:
➜ gcloud container clusters delete k1
Wrap-up
We’ve used CRD in order to define a new API schema called Project in Kubernetes. We used Kubernetes Operator Pattern to deploy a controller inside Kubernetes that monitors those resources. And finally we used Initializers in order to force initialization of the projects to pass through the controller.
This is just the tip of the iceberg with the possibilities of extending Kubernetes with custom controllers and we didn’t even cover Custom Metrics.