Uncloak the Key to Orchestration: Streamlining Keycloak in Kubernetes

Keycloak is an open-source identity and access management (IAM) solution that provides authentication, authorization, and user management features for modern applications and services.

Like other complex software applications, we have a significant number of configuration settings that need to be manually adjusted. This sometimes makes us hesitant to implement new tools in the system.
As a DevOps Engineer at Tikal, my approach to overcoming this is to first learn the manual operations and then automate them in a simple way.

In this blog post, I will describe how we achieved a fully automated way of managing Keycloak with its complex configuration using K8s’ native capabilities, Helm, and Python.
By automating Keycloak configuration, we can ensure consistency, reduce manual efforts, and enable scalable and repeatable deployments.

Uncloak the Key to Orchestration

So, after we know what is Keycloak and know it is a complex software that we want to deploy in our Kubernetes environment, but don’t want to spend our time setting its settings, we can deep dive into the details.

Our application(which uses Keycloak as its IAM) is developed on a Kubernetes cluster as a part of our microservices.
We are using Helm to deploy Keycloak and PostgreSQL in our cluster in a customizable way.

Prerequisites:

As prerequisites for this task, we used:

• Codecentric chart for Keycloak(v18.4.4)
• Bitnami chart for PostgreSQL(v13.2.19)

* We preferred to use a different PostgreSQL chart from the default one that comes with the Keycloak chart to have more control over our database configuration.

** To simplify things, I am presenting here a way to pass credentials via environment variables. There are other methods that are more secure for production environments.

Steps to achieve our target, before and after:

1. Instead of importing the Realm settings json file, save it on your container

Till now, when we wanted to deploy Keycloak in a new environment, we based on a json file that contains configurations related to all the environments, and loaded it via the Keycloak UI:

With the new approach, we are using Codecentric Keycloak chart capabilities to reference a Kubernetes Secret that contains the json file for our Realm configurations. We are mounting it directly to the container’s volume.
The values.yaml file should include that part for the Keycloak configurations:

extraVolumes: |
  - name: realm-secret
    secret:
      secretName: realm-secret

extraVolumeMounts: |
  - name: realm-secret
    mountPath: "/realm/"
    readOnly: true

extraEnv: |
  - name: KEYCLOAK_IMPORT
    value: /realm/realm.json

You can create this secret with the following command:

kubectl create secret generic realm-secret --from-file=<path-to-realm-file>

** Make sure you point to the realm json file path.

2. Instead of ClickOps on the UI, work with the API

As soon as we load the Realm settings, we can convert all manual operations (ClickOps) to work with the API provided by Keycloak to configure our environment behind the scenes.

My choice was to use the Python requests package, which provides a simple method for sending HTTP requests.

Our first step is to obtain the access token of our administrator user and send it as a header when making API calls to the Keycloak server.

def get_access_token(keycloak_url, admin_username, admin_password):
    try:
        token_url = f"{keycloak_url}/realms/master/protocol/openid-connect/token"
        token_payload = {
            "username": admin_username,
            "password": admin_password,
            "grant_type": "password",
            "client_id": "admin-cli"
        }
        token_response = requests.post(token_url, data=token_payload)
        token_response.raise_for_status()
        return token_response.json().get("access_token")
    except requests.exceptions.RequestException as e:
        print(f"Error occurred while obtaining access token: {e}")
        return None

Once we have it, we are able to perform any “Click” that we performed as administrators of the system previously, in several simple lines of code.

Examples of Programming Clicks:

Case 1 — Assign a User to Group

In my case, I needed to assign a user to a group, so I simply loaded the user’s ID and the group ID, and enforced their combination:

user_id = get_user_id(keycloak_url, realm, client_username, headers)
group_id = get_group_id(keycloak_url, realm, group_name, headers)
if not user_id:
  user_id = create_user(keycloak_url, realm, client_username, client_password, client_email, headers)

assign_user_to_group(keycloak_url, realm, user_id, group_id, headers)

The implementation is using PUT calls, for example:

def get_user_id(keycloak_url, realm, client_username, headers):
    try:
        user_url = f"{keycloak_url}/admin/realms/{realm}/users"
        response = requests.get(user_url, headers=headers)
        response.raise_for_status()
        users = response.json()
        for user in users:
            if user["username"] == client_username:
                return user["id"]
        return None
    except requests.exceptions.RequestException as e:
        print(f"Error occurred while retrieving user ID: {e}")
        return None

And so on for the other functions.

Case 2 — Password Generation

In another case, I required Keycloak to generate and pass a one-time secret to my application.

It was previously generated by clicking on the “generate” button, however, we can now generate it via the API, and it is then sent to my application via Kubernetes Secret, which is exported by one application and read by another.
For this purpose, I am using the kubernetes package in Python code.

def regenerate_client_secret(keycloak_url, realm, client_id, headers):
    try:
        client_secret_url = f"{keycloak_url}/admin/realms/{realm}/clients/{client_id}/client-secret"
        client_secret_response = requests.post(client_secret_url, headers=headers)
        client_secret_response.raise_for_status()
        return client_secret_response.json().get("value")
    except requests.exceptions.RequestException as e:
        print(f"Error occurred while regenerating client secret: {e}")
        return None

def create_kubernetes_secret(api_client, namespace, secret_data):
    try:
        encrypted_data = {key: base64.b64encode(value.encode()).decode() for key, value in secret_data.items()}
        secret = client.V1Secret(
            api_version="v1",
            kind="Secret",
            metadata=client.V1ObjectMeta(name="keycloak-data-secret"),
            type="Opaque",
            data=encrypted_data
        )
        api_client.create_namespaced_secret(namespace=namespace, body=secret)
    except client.rest.ApiException as e:
        print(f"Error occurred while creating Kubernetes secret: {e}")

# Regenerate the client secret
new_client_secret = regenerate_client_secret(keycloak_url, realm, client_id, headers)
if new_client_secret is None:
    raise Exception("Failed to regenerate client secret")
print(f"New client secret for {client_id}: {new_client_secret}")

# Create a K8s Secret to store the variables retrieved from Keycloak API
secret_data = {
    "CLIENT_SECRET": new_client_secret,
    "USER_ID": user_id,
    "GROUP_ID": subgroup_id,
    "CLIENT_ID": client_id
}
create_kubernetes_secret(api_client, current_namespace, secret_data)

We can get the data from the Secret as an environment variable in our application when the Secret is created:

  - name: CLIENT_SECRET
    valueFrom:
      secretKeyRef:
        name: keycloak-data-secret
        key: CLIENT_SECRET
        optional: true

To conclude, the main program looks as follows:

# Set the necessary variables
keycloak_url = os.environ.get("KEYCLOAK_URL")
realm = os.environ.get("REALM")
client_username = os.environ.get("CLIENT_USERNAME")
client_password = os.environ.get("CLIENT_PASSWORD")
client_email = os.environ.get("CLIENT_EMAIL")
admin_username = os.environ.get("ADMIN_USERNAME")
admin_password = os.environ.get("ADMIN_PASSWORD")
group_name = os.environ.get("GROUP_NAME")

# Connect to the K8s context
config.load_incluster_config()
# Create a K8s API client
api_client = client.CoreV1Api()
# Get the current namespace
with open("/var/run/secrets/kubernetes.io/serviceaccount/namespace", "r") as file:
    current_namespace = file.read().strip()

# Obtain an access token
access_token = get_access_token(keycloak_url, admin_username, admin_password)
if access_token is None:
    raise Exception("Failed to obtain access token")

headers = {
    "Authorization": f"Bearer {access_token}",
    "Content-Type": "application/json"
}

# Assign user to a group(create if that not exists)
user_id = get_user_id(keycloak_url, realm, client_username, headers)
group_id = get_group_id(keycloak_url, realm, group_name, headers)
if not user_id:
  user_id = create_user(keycloak_url, realm, client_username, client_password, client_email, headers)

assign_user_to_group(keycloak_url, realm, user_id, group_id, headers)

# Regenerate the client secret
new_client_secret = regenerate_client_secret(keycloak_url, realm, client_id, headers)
if new_client_secret is None:
    raise Exception("Failed to regenerate client secret")
print(f"New client secret for {client_id}: {new_client_secret}")

# Create a K8s Secret to store the variables retrieved from Keycloak API
secret_data = {
    "CLIENT_SECRET": new_client_secret,
    "USER_ID": user_id,
    "GROUP_ID": subgroup_id,
    "CLIENT_ID": client_id
}
create_kubernetes_secret(api_client, current_namespace, secret_data)

You can add or edit the operations as your use case.

3. Enable your code running in the cluster(RBAC)

Put your mind logic into your machines

However it seems a simple way to load your mind login into your Kubernetes cluster, I had some challenges related to permissions and access control.
To overcome these challenges, I created the necessary RBAC (Role-Based Access Control) objects, including a ServiceAccount, Role, and RoleBinding.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Values.global.serviceAccountName }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Values.global.roleName }}
rules:
  - verbs:
      - create
    apiGroups:
      - ''
    resources:
      - secrets
      - configmaps
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: {{ .Values.global.roleName }}-binding
subjects:
  - kind: ServiceAccount
    name: {{ .Values.global.serviceAccountName }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {{ .Values.global.roleName }}

A short description of that resources to understand what we are doing:

1. ServiceAccount — represents the identity under which the script runs in the cluster.
2. Role — defines the permissions granted to the ServiceAccount. In this case, the Role allows the creation of Secrets and ConfigMaps in the current namespace.
3. RoleBinding — associates the ServiceAccount with the Role, effectively granting the ServiceAccount the permissions defined in the Role.

By creating these RBAC resources, I ensured that the script running in the Kubernetes Job had the necessary permissions to create Secrets and ConfigMaps within the cluster, enabling it to store the retrieved Keycloak variables and configuration data.

4. Deploy the automation with Helm(Job + values)

After all that preparation, we are ready to start running our logic on the Kubernetes cluster. Our most straightforward method is to create a Kubernetes Job that executes our Python backend code and inserts the environment variables as configurable values.

To package the script and its dependencies into a container image, I created a Dockerfile with the following layers:
The Dockerfile is as follows:

FROM python:3.8-alpine

COPY requirements.txt .
RUN pip install -r requirements.txt

RUN adduser --disabled-password --gecos '' appuser


COPY --chown=appuser:appuser config_realm.py /app/
RUN chmod +x /app/config_realm.py


WORKDIR /app
USER appuser

CMD ["python", "/app/config_realm.py"]

By integrating this automation script into our deployment process and running it from a Kubernetes Job after Keycloak has been successfully deployed, I have improved our deployment process. Helm chart defines this job to be triggered when Keycloak is enabled to receive traffic (using an initContainer which listens to the Keycloak service and ensures it is running).

{{- if .Values.keycloak.enabled }}
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Values.global.jobName }}
spec:
  template:
    spec:
      serviceAccountName: {{ .Values.global.serviceAccountName }}
      initContainers:
      - name: test-connectivity
        image: curlimages/curl:7.73.0
        args:
        - /bin/sh
        - -c
        - >
            set -x;
            while [ $(curl -sw '%{http_code}' "http://<<keycloak_svc_name>>:80" -o /dev/null) -ne 200 ]; do # Replace with your Keycloak service name
            sleep 15;
            done
      containers:
      - name: realm-config
        image: <<image_tag_of_our_script>> # Replace with your Keycloak image tag that created from the previous Dockerfile
        env:
        - name: KEYCLOAK_URL
          value: {{ .Values.global.keycloak.url }}
        - name: REALM
          value: {{ .Values.global.keycloak.realmName }}
        - name: CLIENT_USERNAME
          value: {{ .Values.global.keycloak.clientUser }}
        - name: CLIENT_PASSWORD
          value: {{ .Values.global.keycloak.clientPassword }}
        - name: CLIENT_EMAIL
          value: {{ .Values.global.keycloak.clientEmail }}
        - name: ADMIN_USERNAME
          value: {{ .Values.global.keycloak.username }}
        - name: ADMIN_PASSWORD
          value: {{ .Values.global.keycloak.password }}
        resources:
          limits:
            memory: "128Mi"
            cpu: "100m"
          requests:
            memory: "128Mi"
            cpu: "100m"
      restartPolicy: Never
  backoffLimit: 4
  {{ end }}

* Be sure to change the placeholders: <<keycloak_svc_name>> <<image_tag_of_our_script>>` with the relevant values.

Here’s a simplified version of the Helm chart values.yaml file:

global:
  serviceAccountName: 'keycloak-job-sa'
  roleName: 'secret-configmap-creator'
  jobName: 'keycloak-realm-creator'
namespace: example
cluster: test
keycloak:
  enabled: true
  fullnameOverride: "keycloak"
  serviceAccount:
    create: false
    name: 'keycloak-job-sa'
  extraVolumes: |
    - name: keycloak-realm-json
      secret:
        secretName: keycloak-realm-json
  extraVolumeMounts: |
    - name: keycloak-realm-json
      mountPath: "/realm/"
      readOnly: true
  extraEnv: |
    - name: KEYCLOAK_IMPORT
      value: /realm/realm.json

Streamlining Keycloak in Kubernetes

Conclusion

Automating Keycloak configuration using Kubernetes’ native capabilities and Python has significantly enhanced our DevOps workflow, eliminating errors and ensuring consistent setups. This adaptable method can be extended to various scenarios, making it an invaluable DevOps tool.

Next time you face repetitive manual tasks, remember this approach. By embracing automation, you can transform time-consuming operations into efficient, reliable processes, boosting your productivity.