Kubernetes Initializers Deep Dive and Tutorial

Guang Ya Liu
Oct 18, 2017 · 10 min read

Before you read this article, you might want to review the How Kubernetes Initializers work blog post. This article builds on the concepts presented in that post.

Pod Presets and initializers

The PodPreset supports injecting data into Kubernetes resource files without the need of writing custom initializers. While the PodPreset may be sufficient for most use cases, the initializers enables more customized logic for this data injection.

When choosing to use Pod Preset resources or Initializers, consider these best practices:

1) Initializers are just a special case of generic admission webhooks. They make it easy for you to use the existing Kubernetes controller client code to handle common actions. The ability to easily and flexibly extend Kubernetes is important.

2) Cluster admins need to be careful about which initializers they install. A buggy initializer can block everyone’s ability to create things in the cluster, and initializers are powerful, so you need to be able to trust their authors.

3) PodPreset is a an initializer that someone in Kubernetes community wrote for you in the source code of Kubernetes, which your cluster administrator can trust.

4) A Pod Preset might be valid for the majority of initializer use cases. Not everyone should be required to write an initializer.

How initializers work

An initializer needs three objects to function:

The initializer itself

The initializer controller observes an uninitialized workload as it is created. First, t finds the name of its configured initializer first in the list of pending initializers.

Then the initializer checks to see if it was responsible for initializing the namespace of the workload. The initializer will ignore the workload if the initializer is not assumed to initialize the workload, otherwise, it will try to the initialization work for the workload.

After the initialization work finishes, the initializer removes itself from the list of pending initializers for the workloads.

initializerConfigMap

The configMap usually includes some data for initializer, such as the injection data or some other configuration data of the initializer.

initializerConfig

The initializerConfig resource is used to configure what initializers are enabled and what resources are subject to the initializers.

You should first deploy the initializer and make sure that it is working properly before creating the initializerConfiguration (We usually have the initializer as Kubernetes deployment, and you can just check if the pod of the initializer is running). Otherwise, any newly created resources that configured in initializerConfig will be stuck in an uninitialized state and then timeout. Check here for how to workaround such issues.

You can have multiple initializers in your cluster and each initializer can focus on different tasks in different namespaces.

The following is a diagram which describes the relationship of the above three objects.

Image for post
Image for post

Tutorial for initializers

My tutorial was based on https://github.com/kelseyhightower/kubernetes-initializer-tutorial , I updated it a bit by making it more simple and committed all of the code here https://github.com/gyliu513/jay-work/tree/master/k8s/example/kube-initializer-tutorial

Now we can go through the tutorial step by step and also try to see how to troubleshoot if you encounter issues.

Configuration

The `initializers` is an alpha feature for 1.7, so you need to enable it first.

Enable `admissionregistration.k8s.io/v1alpha1` for `runtime-config` and `Initializers` for `admission-control` in apiserver.

Deploy the sidecar initializer configMap

The configMap in the above only includes container info that will be injected to the workloads. You can add more data here, such as volumes, sidecar initializer configuration etc.

Deploy the sidecar initializer

You may see that the `initializers` for above deployment is `[]` as follows:

The reason that we define it as above is because initializers should explicitly set the list of pending initializers to exclude itself, or to an empty array, to avoid getting stuck waiting to initialize.

Set the list of pending initializers to exclude itself:

Set the pending initializers to an empty array:

Then deploy the sidecar deployment, using deployment can facilitate upgrades and auto restarts.

When an object is POSTed, it is checked against all existing initializerConfiguration objects (explained below). For all that it matches, all `spec.initializers[].names` are appended to the new object’s `metadata.initializers.pending` field.

An initializer controller should list and watch for uninitialized objects, by using the query parameter `?includeUninitialized=true`. If using client-go, just set `listOptions.includeUninitialized` to true.

Some pseudo code as following to describe the logic, you can get more detail from https://github.com/gyliu513/jay-work/blob/master/k8s/example/kube-initializer-tutorial/sidecar-initializer/main.go#L58-L129

Create the initializeConfig

The above means that the sidecar initializer will only impact the deployment resources.

Create an nginx application

Now let us create an nginx application to see if the sidecar can be injected to the nginx application.

From above, we can see that sidecar from the sidecar initializer configMap has been injected to the nginx application.

But there is a problem for this, with the above configurations, all of the deployments will be injected with a sidecar, this may be not expected for some cases.

For such case, we can update the sidecar initializer to filter applications with specified annotations, using annotations to enable opting in or out of initialization.

Now let us delete the sidecar initializer, sidecar initializerConfig and nginx, then re-create the sidecar initializer with logic to filter out some deployments.

Re-deploy the sidecar initializer with annotation filter

You may see that we have added a new parameter named as `-annotation=initializer.kubernetes.io/sidecar` and this parameter will help check if the deployment does not include the required annotation, sidecar initializer will not inject data to the deployment.

Some pseudo code as following for how to use `annotation` to filter out some applications, you can get more detail from https://github.com/gyliu513/jay-work/blob/master/k8s/example/kube-initializer-tutorial/sidecar-initializer/main.go#L131-L192

Re-deploy the initializer config

Deploy the nginx without annotation

Due to above nginx do not include the annotation, so after the deployment was created, it will not have inject data.

From above output, we can see that the nginx include only one container, and no data was injected.

Deploy an nginx with annotation

Now, let us deploy an nginx with annotation as following:

From above, we can see that the deployment including an annotation as follows:

This means that sidecar initializer will inject data to this deployment.

You can see the new created pod of nginx `nginx-with-annotation-5cf4d7fcdb-f7b9l` has two containers which means that the injection has taken effect.

Troubleshooting

Sometimes, you may found that you cannot deploy applications and all applications will be pending for about 30s and then timeout. For such kind of issue, it is usually caused by the initializerConfig when the sidecar initializer not working well, it maybe caused by some bugs in initializer which caused the initializer stopped to work.

Let us simulate the error case step by step.

Delete the sidecar initializer

Till now, we have the initializer working well, now let’s delete the sidecar initializer and both of the nginx, this will left a initializerConfig without initializer.

No pods left.

The initializerConfig still exist.

Create an nginx

From above, we can see that the nginx create failed because of timeout.

Then I try to get the deployment, no deployment.

But when I create the nginx deployment again, it failed!!

Get the uninitialized deploy

For such kind of issues, it was usually caused by the uninitialized objects, we can get all of the uninitialized objects by adding a flag named as ` — include-uninitialized` to `kubectl`, such as `kubectl get deploy — include-uninitialized`.

You can see that above output includes the following:

As I do not have such as initializer, the deployment keeps pending.

Check out more detail at here https://github.com/kubernetes/kubernetes/issues/51883 .

Resolve this issue

We need to first delete the initializeConfig.

As the nginx deployment already include the `pending initializers`, so we need edit the deployment by removing the `pending initializers` and delete the deployment.

Then re-create the nginx deployment, it will be succeed.

Summary /Take Aways

With Kubernetes Initializers, we can customize the logic of how to inject data to Kubernetes resources more flexible.

The initializers can help a lot for isito service mesh. With istio 0.1.X, if you want to deploy an application to be managed by istio, you need to call the command of `istioctl kube-inject` first to inject the sidecar container and volumes into the deployment before you call `kubectl apply` to deploy the application. This is not convenient for the end users, as deploying one application for istio would request two API calls.

In istio 0.2.X, we are introducing an istio sidecar initializer which can help inject data dynamically to Kubernetes resources, this makes end user can deploy applications to be managed by istio more easily as this will not need to call `istioctl kube-inject` to inject data manually but all of the data will be injected automatically by istio sidecar initializer.

There will be another article to deep dive for how does istio sidecar initializer works in a follow up article.

Reference

- https://github.com/kubernetes/kubernetes/pull/50497

- https://ahmet.im/blog/initializers/

- https://kubernetes.io/docs/admin/extensible-admission-controllers/

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise…

Guang Ya Liu

Written by

STSM@IBM, Member - IBM Academy of Technology, Kubernetes Member, Istio Maintainer, Apache Mesos Committer & PMC Member.

IBM Cloud

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise with effective security and data protection at scale.

Guang Ya Liu

Written by

STSM@IBM, Member - IBM Academy of Technology, Kubernetes Member, Istio Maintainer, Apache Mesos Committer & PMC Member.

IBM Cloud

IBM Cloud

Understand how to bring elastic runtimes to the Enterprise with effective security and data protection at scale.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store