Announcing shell-operator to simplify creating of Kubernetes operators

We are happy to introduce our new Open Source solution which takes the development of Kubernetes operators to a whole new easy level. It gives you an appealing opportunity to turn your small Bash script into fully-fledged operator just in 15 minutes. Please welcome — shell-operator!

The purpose

The idea of shell-operator is quite simple. It subscribes to events from the Kubernetes’ objects and executes an external program once an event occurs, providing it with the information about the event:

Many small tasks started to emerge during our operation of Kubernetes clusters. In Flant, we really wanted to automate them in the right way, so we felt the need for a smart solution. Usually, you can solve all these small tasks with the basic bash scripts, although, as you know, the preferred way is to write operators in Golang. Obviously, the full-fledged development of the operator for every minor task would be inefficient.

An operator in 15 minutes

Let’s take an example of what can be automated in the Kubernetes cluster and how the shell-operator can help us. We will try to replicate the secret for accessing the Docker registry.

Pods that use private registry images should include in their manifest a link to the secret with data for accessing the registry. This secret must be created in each namespace before creating the pods. You can do it manually, however, if we will configure dynamic environments, we will get many namespaces for a single application. In the case of several applications (even two or three), the number of secrets becomes huge. One more thing about secrets: we would like to be able to change the registry access key occasionally. As a result the manual solution becomes completely inefficient — you have to automate the creation and updating of secrets.

Simple automation

Let’s write a shell script that is executed every N seconds and checks namespaces for the presence of secret. If the secret is not found, it is created. The advantage of this solution is that it looks like a shell script in cron — a classic and easy-to-understand approach. The downside is that during the interval between two launches of this script some new namespace might emerge, so for some time it would exist without the secret. This kind of situation will lead to errors in the process of starting the pods.

Automation with the shell-operator

To make our script behave correctly, the execution by classic cron should be replaced by the execution that happens when an event of adding a namespace occurs. In this case, you can create a secret before using it. Let’s see how to implement this functionality with a shell-operator.

Firstly, let’s analyze the script. In terms of a shell-operator, scripts are called “hooks.” Each hook when executed with the --config flag informs shell-operator about its bindings (i.e. what events it needs to be executed with). In our case we will use the onKubernetesEvent:

#!/bin/bash
if [[ $1 == "--config" ]] ; then
cat <<EOF
{
"onKubernetesEvent": [
{
"kind": "namespace",
"event": [ "add" ]
}
]
}
EOF
fi

Here we define that we are interested in the events of adding objects (add) of the namespace type.

Now we need to add code that will be executed when an event occurs:

#!/bin/bash
if [[ $1 == "--config" ]] ; then
# configuration
cat <<EOF
{
"onKubernetesEvent": [
{
"kind": "namespace",
"event": [ "add" ]
}
]
}
EOF
else
# response:
# find out what namespace has emerged
createdNamespace=$(jq -r '.[0].resourceName' $BINDING_CONTEXT_PATH)
# create the appropriate secret in it
kubectl create -n ${createdNamespace} -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
...
data:
...
EOF
fi

Awesome! Now we have a compact and pretty script. To make it actually work, let’s prepare an image and run it in the cluster.

Preparing our Docker image with a hook

You can easily note that we use kubectl and jq commands in our script. This means that the image should include the hook, shell-operator binary (it’ll watch the events and execute the hook), and these commands used by the hook (kubectl and jq). The ready-to-use image that already includes shell-operator, kubectl, and jq is available at hub.docker.com. Now it is time to add a hook using Dockerfile:

$ cat Dockerfile
FROM flant/shell-operator:v1.0.0-beta.1-alpine3.9
ADD namespace-hook.sh /hooks
$ docker build -t registry.example.com/my-operator:v1 .
$ docker push registry.example.com/my-operator:v1

Running in the cluster

Let’s look at the hook once again. This time we would note which actions and with what objects it performs in the cluster:

  1. It subscribes to namespace creation events;
  2. It creates a secret in namespaces different from the one where it is running.

It turns out that the pod in which our image will be run should have a permission for these actions. You can grant it with your own ServiceAccount. The permission should be made in a form of ClusterRole and ClusterRoleBinding since we are interested in the objects from the entire cluster.

The final description in YAML will be as follows:

---
apiVersion: v1
kind: ServiceAccount
metadata:
name: monitor-namespaces-acc
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: monitor-namespaces
rules:
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "watch", "list"]
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "create", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: monitor-namespaces
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: monitor-namespaces
subjects:
- kind: ServiceAccount
name: monitor-namespaces-acc
namespace: example-monitor-namespaces

You can deploy the created image as a simple Deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-operator
spec:
template:
spec:
containers:
- name: my-operator
image: registry.example.com/my-operator:v1
serviceAccountName: monitor-namespaces-acc

For convenience, we will create a separate namespace that will be used to run shell-operator and apply created manifests:

$ kubectl create ns example-monitor-namespaces
$ kubectl -n example-monitor-namespaces apply -f rbac.yaml
$ kubectl -n example-monitor-namespaces apply -f deployment.yaml

That’s all! The shell-operator starts up, subscribes to namespace creation events and executes the hook when needed.

This way a simple shell-script turns into a real operator for Kubernetes and becomes a part of the cluster. The advantage of this is that we avoid the complicated process of developing operators in Golang:

Filtering

Watching for objects is great, but it is often necessary to respond to changes in some of their properties, e.g. to increasing/decreasing number of replicas in Deployment or to any updates in object’s labels.

When an event occurs, the shell-operator receives the JSON manifest of the object. In this JSON you can select the properties that you want to monitor and to start the hook only when they change. The jqFilter field can help you with that: you should enter a jq expression that would be applied to JSON manifest.

For example, to respond to modifications in labels of the Deployment objects, you have to extract labels subsection from the metadata field. In this case you will get the following configuration:

cat <<EOF
{
"onKubernetesEvent": [
{
"kind": "deployment",
"event":["update"],
"jqFilter": ".metadata.labels"
}
]
}
EOF

This jqFilter expression converts a long JSON manifest of Deployment into a short JSON with labels:

The shell-operator will execute the hook only when changes in this short JSON occur. The changes in other properties will be ignored.

The hook’s execution context

The hook’s configuration allows you to specify several varieties of events. For example, you can define two events from Kubernetes and two schedules:

{
"onKubernetesEvent": [
{
"name": "OnCreatePod",
"kind": "pod",
"event": [
"add"
]
},
{
"name": "OnModifiedNamespace",
"kind": "namespace",
"event": [
"update"
],
"jqFilter": ".metadata.labels"
}
],
"schedule": [
{
"name": "every 10 min",
"crontab": "0 */10 * * * *"
},
{
"name": "on Mondays at 12:10",
"crontab": "0 10 12 * * 1"
}
]
}

N.B.: Yes, the shell-operator supports running scripts in crontab style! You can find additional information in the documentation.

To distinguish causes of the hook execution, the shell-operator creates a temporary file and saves its path into BINDING_CONTEXT_TYPE variable. This file contains a JSON description of the reason for executing the hook. For example, every 10 minutes the hook will be started with the following contents:

[{ "binding": "every 10 min" }]

… and on Monday it will start with this:

[{ "binding": "every 10 min" }, { "binding": "on Mondays at 12:10" }]

There will be more detailed JSON for onKubernetesEvent calls since it contains a description of the object:

[
{
"binding": "onCreatePod",
"resourceEvent": "add",
"resourceKind": "pod",
"resourceName": "foo",
"resourceNamespace": "bar"
}
]

You can get an overall idea of the fields’ contents from their names (more details can be found in the documentation). An example of getting a resource name from resourceName using jq has already been shown in a hook that replicates secrets:

jq -r '.[0].resourceName' $BINDING_CONTEXT_PATH

In a similar way you can get the other fields as well.

What’s next?

In the project repository, in the /examples directory there are sample hooks that are ready to use in a cluster. You can use them as a basis for developing your own hooks.

Shell-operator also supports collecting metrics with Prometheus. The available metrics are described in the METRICS section.

As you can easily guess, shell-operator is written in Go and distributed under the terms of the Open Source license (Apache 2.0). We would be really grateful for any help with the development of the project on GitHub. Please, support us with stars, issues and pull requests!