How to setup helm and tiller with RBAC and namespaces

How to setup tiller per namespace using RBAC on kubernetes.

Helm is installing tiller on the kube-system namespace by default.

If you need to segregate the permissions tiller has, you will need to create a tiller in every namespace and tie a specific service account to it via the command:

helm init --tiller-namespace <namespace> --tiller-service-account <service_account_name>

This will create tiller and run it as that service account (given that service account exists and has permission for that namespsace)

The scenario of which this setup might needed is a CI/CD or a specific user that needs to communicate with tiller using helm (which under the hood is using kubectl) and have a limited access to a namespace for example segregating staging deploy from production deploy.

My scenario is as such — separating staging and production tillers.

In my case I have a namespace per environment

• prod
• stg

In this post we will be creating Authentication (think of this like a passport — you’re allowed to travel) and Authorization (think of this like a visa in the passport allowing you to enter a country)

ofcourse you can create a namespace and service account from the command line but this post is showing you how to create a kubeconfig file which you can take to your CI/CD be it Jenkins / CircleCI /whetever.

Part 1 — Authentication:

I have made up a script that creates a service account and configure kube config file with the authentication and the default namespace the service account will be bound to. currently this script accepts 2 arguments
service_account_name and namespace.
this script must run as the cluster admin user.

note: the service account name only have to be unique per namespace, so it’s fine to have it on multiple namespaces with the same name

1. Download the script:

wget https://gist.github.com/innovia/fbba8259042f71db98ea8d4ad19bd708/raw/b3300b0bde484f4bcbfca6cc619d4cd702731e18/kubernetes_add_service_account_kubeconfig.sh
chmod a+x kubernetes_add_service_account_kubeconfig.sh

2. If you manage multiple clusters, make sure your context is set to the cluster you want to create the service account on

The script takes 2 arguments: service_account_name and namespace.

./kubernetes_add_service_account_kubeconfig.sh <service_account_name> <namespace>

It will create the service account and will extract authentication information from it to kubeconfig file which you can then use on your CI machine etc..

./kubernetes_add_service_account_kubeconfig.sh tiller stg                                                                 
Creating target directory to hold files in /tmp/kube...done
Creating a service account: tiller-stg
serviceaccount "tiller-stg" created
Getting secret of service account tiller-stg
Secret name: tiller-stg-token-hswq6
Extracting ca.crt from secret...done
Getting user token from secret...done
Setting current context to: some-cluster
Cluster name: some-cluster.domain.com
Endpoint: https://api.some-cluster.domain.com
Preparing k8s-tiller-stg-conf
Setting a cluster entry in kubeconfig...Cluster "some-cluster.domain.com" set.
Setting token credentials entry in kubeconfig...User "tiller-stg-some-cluster.domain.com" set.
Setting a context entry in kubeconfig...Context "tiller-stg-some-cluster.domain.com" modified.
Setting the current-context in the kubeconfig file...Switched to context "tiller-stg-some-cluster.domain.com".
All done! Test with:
KUBECONFIG=/tmp/kube/k8s-tiller-stg-conf kubectl get pods
you should not have any permissions by default - you have just created the authentication part
You will need to create RBAC permissions
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:tiller-stg" cannot list pods in the namespace "stg"

Here’s the script:

Part 2 — Authorization
So far we have created a service account with on a namespace and kubeconfig for authentication, it’s time to take care of the RBAC permissions.

What we want here is:

• A role (we’ll call this one tiller-manager) on a specific namespace
• A role binding between the service account we’ve created earlier to that role (tiller-manager).

Create the role for the tiller on the namespace (replace the namespace with the namespace you want)
note: you might need more / fewer rules for your tiller-manager role.

wget https://gist.github.com/innovia/d017080fcc0485157748e87ff82c8d17/raw/e228893c7f799973467705e29ab2ed8b98c92372/tiller_role_service_account_rbac.yaml
kubectl apply -f tiller_role_service_account_rbac.yaml

Bind the role to the service account

wget https://gist.github.com/innovia/b75e1cfa2837eb9b80a3485b45d328f2/raw/10a8d7168b2f2a6dbe074cc102d91630ef675e42/tiller_role_binding.yaml
kubectl apply -f tiller_role_binding.yaml

to test this service-account with the config run
note that the default namespace for the kube config is set to the namespace you gave so no need to explicitly use the -n <namespace>

$ KUBECONFIG=/tmp/kube/k8s-<service_account_name>-<namespace>-conf kubectl get pods
No resources found.
# Attempt to check permissions to noather namespace:
$ KUBECONFIG=/tmp/kube/k8s-<service_account_name>-<namespace>-conf kubectl get pods -n default
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:stg:tiller" cannot list pods in the namespace "default"

3.Install tiller using helm into the designated namespace

helm init --service-account tiller --tiller-namespace stg

4. Deploy to the namespace using the kubeconfig file

KUBECONFIG=k8s-tiller-stg-conf helm install --namespace stg stable/nginx --tiller-namespace stg