Deep Dive: Kubernetes Single Sign-On (SSO) with OpenID Connection via G Suite

Nowadays, Kubernetes is a de facto standard for the container orchestration and I’m using Kubernetes for 2+ years in production.

As a DevOps Engineer, we work closely with developers and use the same tools such as CI-CD, VCS, IaC, monitoring, logging, container orchestration, etc. Kubernetes is also one of these tools and we have to work together during the development, deployment, debugging and monitoring of the applications. As you know, this is a requirement of DevOps culture.

That’s why, we need to create usernames and passwords, tokens, configurations, etc or to share the existing information for accessing clusters while working with the team. Once you shared all of the necessary credentials with them, probably, you will lose control after a while and this would be a critical security problem even if the clusters are running in the private network or behind the VPN.

I will show you how to avoid this problem by using OpenID Connect (OIDC) authentication method and Kubernetes clusters together using this guide.

Let’s see the steps:

1. Setup KOPS cluster
2. Setup G Suite as an OIDC provider
3. Setup Kubernetes API server
4. Configure the cluster permissions
5. Setup the kubeconfig
6. Setup the kubelogin
7. Verify the cluster access
8. On-boarding and Off-boarding process

1. Setup KOPS Cluster

KOPS (Kubernetes Operations) helps you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes clusters from the command line.

Creating a KOPS Cluster

KOPS installation can be followed from here. I tried to explain below all of the steps basically.

1- Register a domain: If you already have a domain name, you can use it. Otherwise, register a domain.

2- Sign-in AWS account: If you already have an AWS account you can use it or you can create a free account.

3- Create an AWS IAM user: Create an IAM user called kops-admin following this guide then assign the AdministratorAccess permission using this guide.

4- Install and Configure aws-cli on your local machine following this guide.

5- Create a sub-DNS zone on Route53: We need a top-level domain or subdomain for creating the KOPS cluster. We will use a subdomain for now. To create a subdomain you can follow this guide on AWS

After creating the subdomain, specify the name server (NS) records from the created zone as NS records with the domain registrar.

6- To create an S3 bucket for KOPS state store, follow this guide

7- Install KOPS client and kubectl on your local machine following this guide.

8- Install KOPS Kubernetes cluster running the following command:

kops create cluster dev.automateops.co.uk \
  --node-count 1 \
  --zones us-west-2a \
  --node-size t2.medium \
  --master-size t2.medium \
  --master-zones us-west-2a,us-west-2b,us-west-2c \
  --networking calico \
  --topology private \
  --bastion="true" \
  --state=s3://dev.autotameops.co.uk-state-store \
  --kubernetes-version=1.14.6 --yes

2. Set up the OIDC provider

OpenID Connect is a simple identity layer on top of the OAuth 2.0 protocol. It allows Clients to verify the identity of the End-User based on the authentication performed by an Authorization Server, as well as to obtain basic profile information about the End-User in an interoperable and REST-like manner.

Google Identity Platform

We will use Google for the identity platform. Open Google Cloud Platform here and sign in. After that, you can select Create a new project option to reach the following screen:

After creating the project, click ENABLE APIS AND SERVICES for enabling Admin API

Search and select “Admin SDK” and then Enable it.

We have created a new project and enabled the API. Now, we need to setup GSuite as an OpenID Connector.

Before your users authenticate, the consent screen will allow them to choose whether they want to grant access to their private data, as well as give them a link to your terms of service and privacy policy.

Let’s configure OAuth consent screen by clicking APIs & Services > OAuth consent screen

If you want to authenticate your users from multiple domain names, you can choose Public application type. We will continue with Internal application type for now.

To protect your kubernetes cluster(s), you need to add your domain name(s) into Authorised domains. You can add multiple domain names here. We will add automateops.co.uk for now then click Save button.

You should see the following screen after click to the save button. Then to able to access APIs, we are going to select OAuth client ID from Create credentials list menu.

Choose Other application type through the following screen, give a name you want and then click Create button.

3. Setup the Kubernetes API server

Users can access the API using kubectl, client libraries, or by making REST requests. Both human users and Kubernetes service accounts can be authorized for API access. When a request reaches the API, it goes through several stages, illustrated in the following diagram:

Once TLS is established, the HTTP request moves to the Authentication step. This is shown as step 1 in the diagram.

If the request cannot be authenticated, it is rejected with HTTP status code 401. Otherwise, the user is authenticated as a specific username, and the user name is available to subsequent steps to use in their decisions. Some authenticators also provide the group memberships of the user, while other authenticators do not.

While Kubernetes uses usernames for access control decisions and in request logging, it does not have a user object nor does it store usernames or other information about users in its object store.

After a short Kubernetes API server brief, let’s configure the kube-apiserver.

We have already installed the Kubernetes cluster via KOPS but before kube-api configuration, we need to set OAuth credentials. You can find oidcIssuerURL information on Google Cloud Platform. As you can see the following screen, click Credentials from the left menu and select OIDC Client ID from the list which we have just created before.

Run kops edit cluster command and append the following settings.

spec:
  kubeAPIServer:
    oidcIssuerURL: https://accounts.google.com
    oidcClientID: YOUR_CLIENT_ID
    oidcUsernameClaim: email

Save and run the following commands to apply changes to the cluster.

export KOPS_STATE_STORE="s3://dev.automatesops.co.uk-state-store"

kops update cluster --yes
kops rolling-update cluster --yes

P.S: When rolling-update operation successfully completed, you should see oidc flags into the kube-apiserver pods.

kubectl describe pod -n kube-system kube-apiserver-<ip>.<region>.compute.internal
--oidc-client-id=YOUR_CLIENT_ID 
--oidc-issuer-url=https://accounts.google.com 
--oidc-username-claim=email

4. Configure cluster permissions

As we have mentioned before, in step 3, we need to define RBAC roles for authorization. For further details about RBAC permissions, you can jump here.

I’m using RBAC-Manager for managing permissions on Kubernetes cluster. RBAC-Manager installation is very straightforward:

kubectl apply -f https://raw.githubusercontent.com/FairwindsOps/rbac-manager/master/deploy/all.yaml

After RBAC-Manager installation, we will create 3 groups for users permissions:

• cluster-admins
• web-developers
• api-developers

apiVersion: rbacmanager.reactiveops.io/v1beta1
kind: RBACDefinition
metadata:
  name: rbac-manager-users
rbacBindings:
  - name: cluster-admins
    subjects:
      - kind: User
        name: halid@automateops.co.uk
      - kind: User
        name: manesh@automateops.co.uk
    clusterRoleBindings:
      - clusterRole: cluster-admin
  - name: web-developers
    subjects:
      - kind: User
        name: sarah@automateops.co.uk
      - kind: User
        name: oliver@automateops.co.uk
    roleBindings:
      - clusterRole: edit
        namespace: web
      - clusterRole: view
        namespace: api
  - name: api-developers
    subjects:
      - kind: User
        name: daniel@automateops.co.uk
      - kind: User
        name: sophia@automateops.co.uk
    roleBindings:
      - clusterRole: edit
        namespace: api
      - clusterRole: view
        namespace: web

kubectl apply -f permissions.yaml

5. Set up the kubeconfig

You already have a kubeconfig file under ~/.kube/config

Add the following information to the kubeconfig:

users:
- name: google
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://accounts.google.com
      - --oidc-client-id=YOUR_CLIENT_ID
      - --oidc-client-secret=YOUR_CLIENT_SECRET
      - --oidc-extra-scope=email
      - --oidc-extra-scope=profile
      command: kubectl

Your kubeconfig file should look like:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <your cluster certificate-authority-data>
    server: https://api.dev.automateops.co.uk
  name: dev.automateops.co.uk
contexts:
- context:
    cluster: dev.automateops.co.uk
    user: google
  name: dev.automateops.co.uk
current-context: dev.automateops.co.uk
kind: Config
preferences: {}
users:
- name: google
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://accounts.google.com
      - --oidc-client-id=YOUR_CLIENT_ID
      - --oidc-client-secret=YOUR_CLIENT_SECRET
      - --oidc-extra-scope=email
      - --oidc-extra-scope=profile
      command: kubectl

You can share the kubeconfig to your team members for on-boarding.

6. Installing kubelogin

Kubernetes has no web interface to trigger the authentication process. There is no browser or interface to collect credentials which is why you need to authenticate to your identity provider first.

That’s why we need to install a kubectl plugin. When you run kubectl, kubelogin opens the browser and you can log in to the provider. Then kubelogin gets a token from the provider and kubectl access Kubernetes APIs with the token.

You can install kubelogin as below. For more details click here.

# Homebrew
brew install int128/kubelogin/kubelogin

# Krew
kubectl krew install oidc-login

# GitHub Releases
curl -LO https://github.com/int128/kubelogin/releases/download/v1.15.0/kubelogin_linux_amd64.zip
unzip kubelogin_linux_amd64.zip
ln -s kubelogin kubectl-oidc_login

7. Verify cluster access

When you run kubectl command with any arguments Google sign-in page will be pop-up automatically. If you have permission on the cluster, which we defined in step 3, you will be accessed to the resources after the sign-in process as shown below.

kubectl get nodes
Open http://localhost:8000 for authentication
NAME                                    STATUS    ROLES     AGE       VERSION
ip-172-20-1-1.us-west-2.compute.internal   Ready     node      122m       v1.14.6
ip-172-20-1-2.us-west-2.compute.internal   Ready     node      123m      v1.14.6

8. On-boarding and Off-boarding process

Companies’ departments always have employee circulation and you need to re-configure the permissions of the clusters during the on-boarding and off-boarding process.

On-boarding: When a new employee joins the team such as the development team, you can share kubeconfig which we already prepared in step 5.

Off-boarding: When an employee leaves the company, you have to re-deploy RBAC configuration to the Kubernetes clusters which we already configured in step 4.

Conclusion

As a result, if you are looking for an SSO solution for Kubernetes, OpenID Connector will be the best alternative.

Stay safety!

Resources

kubernetes/kops

Controlling Access to the Kubernetes API

Authenticating

FairwindsOps/rbac-manager

int128/kubelogin