Use start up scripts to tailor-make worker nodes that match your workloads

Oracle Container Engine for Kubernetes (OKE) Supports Custom Worker Node Start Up Scripts

Introduction

Oracle Container Engine for Kubernetes (OKE) now supports customizing Kubernetes worker nodes using start up scripts. This functionality provides users greater control over configuring the worker node hosts that run Kubernetes workloads.

Background

Cloud-init is the industry standard method for initialization and configuring cloud instances. It’s supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare metal installations. For more information about cloud-init, see the cloud-init documentation.

OKE installs a default cloud-init start-up script on each Oracle Cloud Infrastructure (OCI) Compute instance which is used to configure the instance to be a Kubernetes worker node. When an instance boots up for the first time, the cloud-init script runs. The default start-up script contains the following logic:

#!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/
opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-
init.sh
bash /var/run/oke-init.sh

The default host configurations set by OKE don’t necessary match every user’s use case, which may lead to a need for users to make their own customizations to worker nodes. Previously, users could customize worker nodes either one at a time or at scale by connecting to worker nodes via SSH, making changes, and then saving those changes as a custom image to be used as the template for future worker nodes. However, this approach had a high operational cost: it required users to create new custom images whenever a new image was released. In contrast, supporting custom cloud-init enables users to write a customization script once and have it applied to all future worker nodes created in the node pool.

Use Cases

Users can customize the default start-up script by adding their own logic to the script, either before or after the OKE logic runs. This provides users the flexibility to perform certain actions that need to be run prior to the running of the OKE script. For example, this could be used to configure an internal yum repository or setup a corporate proxy.

Customizing the default start-up script enables a large number of use cases. Here are a few:

• Use kubelet-extra-args to configure kubelet taints to control which nodes pods are scheduled onto.
• Configure an SELinux policy on all worker node hosts for security and compliance purposes.
• Unassign an instance’s ephemeral public IP on start-up, and reassign the instance a reserved public IP instead.
• Configure a corporate proxy.
• Configure custom yum proxies.
• Install mandated anti-virus software, and other security tools

Users can customize the default start-up script when creating a new cluster, creating new node pools, and modifying existing node pools. This can be done in the OKE console and using the CLI, SDK, API, and other surfaces. The customized start-up script runs when an instance hosting a worker node is booted up. After customizing the default start-up script, it is a good idea to run the Node Doctor script to confirm that worker nodes on newly started instances are working as expected. Refer to Node Troubleshooting Using the Node Doctor Script for more information.

Using a Custom Cloud-init Script to Set kubelet-extra-args

You can use a custom cloud-init script to configure a number of extra options on the kubelet (the primary node agent) on worker nodes. These extra options are sometimes referred to as kubelet-extra-args. One such kubelet- extra-args option is the option to configure debug level log verbosity.

To configure debug level log verbosity, use the following cloud-init script:

#!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/
opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-
init.sh
bash /var/run/oke-init.sh --kubelet-extra-args "--v=4"

To confirm the setting of debug level log verbosity, connect to a worker node and use the sudo systemctl status -l kubelet command. When the above cloud-init script has been run on worker nodes, the sudo systemctl status -l kubelet command returns the verbosity level as 4. The kubelet logs also contain more detail.

Using a Custom Cloud-init Script to Configure SELinux (Security-Enhanced Linux)

SELinux is a security enhancement to Linux that enables administrators to constrain which users and applications can access which resources based on rules in a policy. SELinux also adds finer granularity to access controls. Users can configure SELinux on worker nodes using a custom cloud-init script.

SELinux can be in one of two states, either enabled or disabled. When it is enabled, SELinux can run in one of two modes, either enforcing or permissive. By default, SELinux is enabled on OKE worker nodes and set to run in permissive mode. When running in permissive mode, SELinux does not enforce access rules and only performs logging.

To enforce access rules, set SELinux to run in enforcing mode. When running in enforcing mode, SELinux blocks actions that are contrary to the policy and logs a corresponding event in the audit log.

To set SELinux to run in enforcing mode, use the following cloud-init script:

#!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/
opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-
init.sh
bash /var/run/oke-init.sh
setenforce 1
sed -i 's/^SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config

To confirm the status and mode of SELinux that is running on a worker node, connect to the worker node and use the getenforce command. When the above cloud-init script has been run on worker nodes, the getenforce command returns Enforcing.

Conclusion

Customizing worker nodes using startup scripts enables advanced users to tailor-make worker nodes that match their use case. Users are no longer stuck deciding between whether to use the default node configuration or accept the additional management overhead of manually managing custom images. Now users can simply add a custom script before or after the OKE script runs in order to modify their worker nodes as they see fit.

Want to know more?

To learn more or get hands-on, use the following resources:

• Access OKE resource center for product details and testimonials.
• Learn more about OKE from our documentation.
• Try out using a custom worker node startup script yourself.
• Get started with Oracle Cloud Infrastructure today with our Oracle Cloud Free Tier.