CentOS 7 with CRI-o on EKS
A hands-on guide to configure CentOS 7 as a worker node to EKS using CRI-o runtime.
Updated for EKS 1.11.5 kubernetes version
There’s a small bonus at the end that builds on top of CRI-o. *wink* *wink*
Foreword
Some steps have been inspired by the EKS custom AMI guide.
Creating an EKS cluster is a topic outside the scope of this article.
You will need a running EKS cluster before you begin performing steps outlined below.
The setup
Instantiate a CentOS 7 AWS marketplace AMI.
- Place it in the existing EKS worker node AZ of your choice.
- Add to it any IAM roles associated with the existing EKS worker nodes. It would be extremely useful if these IAM roles, associated with the EKS worker nodes would allow worker nodes to list clusters (many of the following steps leverage this); i.e.
eks:ListClusters
- Add to it any security groups associated with the existing EKS worker nodes.
- Add a TAG to it, in the format (replace $MY_CLUSTER_NAME with the actual name of the EKS cluster)
kubernetes.io/cluster/$MY_CLUSTER_NAME: owned
Enable ip_forwarding
echo 'net.ipv4.ip_forward=1' >> /etc/sysctl.conf && sysctl -p
Set iptables default policy for the FORWARD chain to ACCEPT
iptables -P FORWARD ACCEPT
Update the OS
yum update -y
Install pre-requisite packages
ruby + nokogiri will be needed for the POD calc script later on
yum install -y epel-release
yum install -y conntrack curl nfs-utils ntp socat unzip wget bsdtar ruby rubygem-nokogiri
Enable ntpd
systemctl enable ntpd
Install awscli
wget -qO- https://bootstrap.pypa.io/get-pip.py | python
pip install --upgrade awscli
If everything’s ok, you should see that the awscli has been installed successfully.
Export some variables
Grab local IPv4 address
LOCAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)
Set kube-dns CLUSTER-IP
DNS_CLUSTER_IP=10.100.0.10
[[ $LOCAL_IP == 10.* ]] && DNS_CLUSTER_IP=172.20.0.10
Kubernetes and aws-iam-authenticator versions
K8S_VER=v1.11.5
HEPTIO_VER=1.11.5
Grab EKS region
MY_REGION=$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone | sed 's/\(.\)$//')
Grab EKS cluster name
EKS_CLUSTER_NAME=$(aws eks list-clusters --region=$MY_REGION --output text --query 'clusters')
Grab EKS control plane API endpoint
EKS_API_ENDPOINT=$(aws eks describe-cluster --region=$MY_REGION --name=$EKS_CLUSTER_NAME --output=text --query 'cluster.{endpoint: endpoint}')
Set CNI and CNI plugin versions
CNI_VERSION=${CNI_VERSION:-"v0.6.0"}
CNI_PLUGIN_VERSION=${CNI_PLUGIN_VERSION:-"v0.7.1"}
CRI-o setup
Create a repo file
cat << EOF > /etc/yum.repos.d/crio.repo
[cri-o]
name=CRI-O Packages for EL 7 — $basearch
baseurl=https://cbs.centos.org/repos/paas7-crio-311-candidate/x86_64/os
enabled=1
gpgcheck=0
EOF
Install CRI-o runtime and tools
yum -y install cri-o cri-tools
Configure CRI-o
cat << EOF > /etc/crio/crio.conf
[crio]
storage_driver = "overlay2"[crio.api]
listen = "/var/run/crio/crio.sock"
stream_address = ""
stream_port = "10010"
stream_enable_tls = false
stream_tls_cert = ""
stream_tls_key = ""
stream_tls_ca = ""
file_locking = false[crio.runtime]
runtime = "/usr/bin/runc"
runtime_untrusted_workload = ""
default_workload_trust = "trusted"
no_pivot = false
conmon = "/usr/libexec/crio/conmon"
conmon_env = [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
]
selinux = true
seccomp_profile = "/etc/crio/seccomp.json"
apparmor_profile = "crio-default"
cgroup_manager = "cgroupfs"
default_capabilities = [
"CHOWN",
"DAC_OVERRIDE",
"FSETID",
"FOWNER",
"NET_RAW",
"SETGID",
"SETUID",
"SETPCAP",
"NET_BIND_SERVICE",
"SYS_CHROOT",
"KILL",
]
hooks_dir_path = "/usr/share/containers/oci/hooks.d"
default_mounts = [
"/usr/share/rhel/secrets:/run/secrets",
]
pids_limit = 1024
log_size_max = -1
read_only = false
log_level = "error"
uid_mappings = ""
gid_mappings = ""[crio.image]
default_transport = "docker://"
pause_image = "k8s.gcr.io/pause-amd64:3.1"
pause_command = "/pause"
signature_policy = ""
image_volumes = "mkdir"
registries = [
"602401143452.dkr.ecr.$MY_REGION.amazonaws.com",
"docker.io",
"registry.hub.docker.com",
"k8s.gcr.io",
][crio.network]
network_dir = "/etc/cni/net.d/"
plugin_dir = "/opt/cni/bin"
EOF
Remove CRI-o default CNI configuration
rm -rf /etc/cni/net.d/*
Enable CRI-o daemon
systemctl daemon-reload
systemctl enable crio
Kubernetes setup
Create directories
mkdir -p /etc/kubernetes/{manifests,pki,kubelet}
mkdir -p /opt/cni/bin
Grab CNI
wget -qO- https://github.com/containernetworking/cni/releases/download/${CNI_VERSION}/cni-amd64-${CNI_VERSION}.tgz | bsdtar -xvf - -C /opt/cni/bin
Grab CNI plugins
wget -qO- https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGIN_VERSION}/cni-plugins-amd64-${CNI_PLUGIN_VERSION}.tgz | bsdtar -xvf - -C /opt/cni/bin
Grab kubelet and kubectl binaries
BIN_BASE_URL="https://storage.googleapis.com/kubernetes-release/release/$K8S_VER/bin/linux/amd64"BINARIES=(
kubelet
kubectl
)
for binary in ${BINARIES[*]} ; do
wget $BIN_BASE_URL/$binary -O /usr/bin/$binary
chmod +x /usr/bin/$binary
done
Grab heptio IAM authenticator (aws-iam-authenticator)
wget https://amazon-eks.s3-us-west-2.amazonaws.com/$HEPTIO_VER/2018-12-06/bin/linux/amd64/aws-iam-authenticator -O /usr/bin/aws-iam-authenticator
chmod +x /usr/bin/aws-iam-authenticator
Fetch EKS CA
cat << EOF > /etc/kubernetes/pki/ca.crt
$(aws eks describe-cluster \
--region=$MY_REGION \
--name=$EKS_CLUSTER_NAME \
--output=text --query 'cluster.{certificateAuthorityData: certificateAuthority.data}' | base64 -d)
EOF
Create kubeconfig
cat << EOF > /etc/kubernetes/kubelet/kubeconfig
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: $EKS_API_ENDPOINT
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubelet
name: kubelet
current-context: kubelet
users:
- name: kubelet
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
command: /usr/bin/aws-iam-authenticator
args:
- "token"
- "-i"
- "$EKS_CLUSTER_NAME"
EOF
Generate POD per ENI file
With EKS you can only run a certain number of PODs per ENI on your worker nodes.
This script fetches the table contents available via the available URL and generates a file with values that the kubelet can consume in order to set --max-pods
argument properly
wget -qO- https://raw.githubusercontent.com/errnothxbye/centos-eks/master/extra/pods_per_eni.rb | ruby
If everything’s ok, you should have a file /etc/kubernetes/misc/eni-max-pods.txt generated with proper values.
Create kubelet config file (v1.11.5)
cat << EOF > /etc/kubernetes/kubelet/config
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: "/etc/kubernetes/pki/ca.crt"
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: cgroupfs
clusterDomain: cluster.local
maxPods: $(grep $(curl -s http://169.254.169.254/latest/meta-data/instance-type) /etc/kubernetes/misc/eni-max-pods.txt | awk '{print $2}')
runtimeRequestTimeout: "10m"
kubeletCgroups: /system.slice
clusterDNS:
- $DNS_CLUSTER_IP
featureGates:
RotateKubeletServerCertificate: true
serverTLSBootstrap: true
EOF
Create kubelet service
cat << EOF > /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=crio.service
Requires=crio.service
[Service]
ExecStart=/usr/bin/kubelet \\
--node-ip=$LOCAL_IP \\
--config=/etc/kubernetes/kubelet/config \\
--allow-privileged=true \\
--cloud-provider=aws \\
--container-runtime=remote \\
--container-runtime-endpoint=unix:///var/run/crio/crio.sock \\
--image-service-endpoint=unix:///var/run/crio/crio.sock \\
--network-plugin=cni \\
--root-dir=/etc/kubernetes/kubelet \\
--cert-dir=/etc/kubernetes/kubelet/pki \\
--register-node=true \\
--kubeconfig=/etc/kubernetes/kubelet/kubeconfig \\
--node-labels=runtime=crio
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
Enable and start kubelet
systemctl daemon-reload
systemctl enable kubelet
systemctl start kubelet
If everything's ok, in a couple of seconds you should see your CentOS 7 AMI with CRI-o latched onto EKS cluster.
kubectl get node --selector=runtime=crio
NAME STATUS ROLES AGE VERSION
xxx.xxx.xxx.xxx Ready <none> 1m v1.11.5
List containers on the node
crictl ps
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
fa25443c53ca6 602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni@sha256:1fafb3a9a12feef7105986c0ed5e3a6b327f2b5a5356e64f026509614dab921d 23 hours ago Running aws-node 0
2e808c0d754ac c6fc6eef666a5ec079aa5249aacfa2443c96e06aa2b5d18e351d6cfff6c5f00c 23 hours ago Running kube-proxy
BONUS!!
Enable ClearContainers runtime (cc-runtime
)as an untrusted workload runtime with CRI-o.
Foreword
ClearContainers are extremely lightweight virtual machines that synergize well with the container ecosystem.
They run their own lightweight OS and dedicated kernel, providing isolation of network, I/O, memory and can also utilize hardware-enforced isolation with virtualization VT extensions.
Prerequisites
Since they are extremely lightweight machines, you will need
- An instance type that has VT extensions; i.e.
i3.metal
- All the previous steps for configuring a node with CRI-o runtime as EKS worker node apply.
The setup
Instantiate an i3.metal
machine with CentOS 7 as OS.
Follow the steps above to configure this instance as EKS worker node but stop before the CRI-o setup section/steps.
Install ClearLinux containers 3.0
Grab cc repo
wget http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/CentOS_7/home:clearcontainers:clear-containers-3.repo -O /etc/yum.repos.d/cc.repo
Install cc-runtime and relevant packages
yum -y install cc-* qemu-cc*
Configure clear-containers
sed -i 's/qemu-system-x86_64/qemu-cc-system-x86_64/g' /usr/share/defaults/clear-containers/configuration.tomlsed -i 's/macvlan/bridged/g' /usr/share/defaults/clear-containers/configuration.toml
Configure CRI-o
The only relevant change to the CRI-o config is adding /usr/bin/cc-runtime as a value to theruntime_untrusted_workload
configuration option.
All other steps outlined in the article above, are the same.
For visibility and clarity, the complete here-doc is shown below (incl. runtime_untrusted_workload
)
cat << EOF > /etc/crio/crio.conf
[crio]
storage_driver = "overlay2"[crio.api]
listen = "/var/run/crio/crio.sock"
stream_address = ""
stream_port = "10010"
stream_enable_tls = false
stream_tls_cert = ""
stream_tls_key = ""
stream_tls_ca = ""
file_locking = false[crio.runtime]
runtime = "/usr/bin/runc"
runtime_untrusted_workload = "/usr/bin/cc-runtime"
default_workload_trust = "trusted"
no_pivot = false
conmon = "/usr/libexec/crio/conmon"
conmon_env = [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
]
selinux = true
seccomp_profile = "/etc/crio/seccomp.json"
apparmor_profile = "crio-default"
cgroup_manager = "cgroupfs"
default_capabilities = [
"CHOWN",
"DAC_OVERRIDE",
"FSETID",
"FOWNER",
"NET_RAW",
"SETGID",
"SETUID",
"SETPCAP",
"NET_BIND_SERVICE",
"SYS_CHROOT",
"KILL",
]
hooks_dir_path = "/usr/share/containers/oci/hooks.d"
default_mounts = [
"/usr/share/rhel/secrets:/run/secrets",
]
pids_limit = 1024
log_size_max = -1
read_only = false
log_level = "error"
uid_mappings = ""
gid_mappings = ""[crio.image]
default_transport = "docker://"
pause_image = "k8s.gcr.io/pause-amd64:3.1"
pause_command = "/pause"
signature_policy = ""
image_volumes = "mkdir"
registries = [
"602401143452.dkr.ecr.$MY_REGION.amazonaws.com",
"docker.io",
"registry.hub.docker.com",
"k8s.gcr.io",
][crio.network]
network_dir = "/etc/cni/net.d/"
plugin_dir = "/opt/cni/bin"
EOF
Configure kubelet
For better granulation, add more labels to the i3.metal
instance’s kubelet arguments.
--node-labels=runtime=crio,hw=metal,workload=untrusted
Verify workloads
Trusted
Create a POD manifest and add a relevant annotation to have it ran by a “trusted” runc
runtime
apiVersion: v1
kind: Pod
metadata:
name: nginx-trusted
annotations:
io.kubernetes.cri-o.TrustedSandbox: "true"
spec:
containers:
- name: nginx
image: nginx:1.15.5
ports:
- containerPort: 80
nodeSelector:
hw: metal
Apply above manifest using kubectl command.
Untrusted
Create a POD manifest and add a relevant annotation to have it ran by a “untrusted” cc-runtime
runtime
apiVersion: v1
kind: Pod
metadata:
name: nginx-untrusted
annotations:
io.kubernetes.cri-o.TrustedSandbox: "false"
spec:
containers:
- name: nginx
image: nginx:1.15.5
ports:
- containerPort: 80
nodeSelector:
hw: metal
Apply above manifest using kubectl command.
Verify your workloads (on the worker node)
Run cc-runtime list
command to get a list of untrusted container workloads running on the node
ID PID STATUS BUNDLE CREATED OWNER
1f1acc3a0595d49c0c93fb89a785c66b80fbe4c94331f25f20af7b6b8c589dcd 12292 running /run/containers/storage/overlay-containers/1f1acc3a0595d49c0c93fb89a785c66b80fbe4c94331f25f20af7b6b8c589dcd/userdata 2018–11–20T09:25:28.937418117Z #0
77730a648caafb727c2d6a2b52b8c28e9aef18e40e3b9c64f0524bef9f1a5fcf 12483 running /run/containers/storage/overlay-containers/77730a648caafb727c2d6a2b52b8c28e9aef18e40e3b9c64f0524bef9f1a5fcf/userdata 2018–11–20T09:25:34.671725933Z #0
Run crictl ps
command to get a list of all container workloads running on the node
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
77730a648caaf docker.io/library/nginx@sha256:d98b66402922eccdbee49ef093edb2d2c5001637bd291ae0a8cd21bb4c36bebe 3 hours ago Running nginx 0
c47e18087a3b7 docker.io/library/nginx@sha256:d98b66402922eccdbee49ef093edb2d2c5001637bd291ae0a8cd21bb4c36bebe 3 hours ago Running nginx 0
fc5d666325e18 602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni@sha256:1fafb3a9a12feef7105986c0ed5e3a6b327f2b5a5356e64f026509614dab921d 3 hours ago Running aws-node 0
0185c5f50f320 602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy@sha256:76927fb03bd6b37be4330c356e95bcac16ee6961a12da7b7e6ffa50db376438c 3 hours ago Running kube-proxy
You should see two nginx containers running on the node, verify their properties (replace $NGINX_CONTAINER_ID with the actual container ID running on your node)
# Trusted container sharing host's kernel
crictl exec -it $NGINX_CONTAINER_ID1 uname -r
3.10.0–862.14.4.el7.x86_64# Untrusted container running its own kernel
crictl exec -it $NGINX_CONTAINER_ID2 uname -r
4.14.22-86.1.container
Conclusion
Not even the sky is the limit.
It’s possible to run various combinations of distributions and runtimes as worker nodes to the EKS.
Big thanks to Amazon for respecting the malleable properties of Linux and services based on Linux.
My little tutti-frutti of EKS worker nodes and runtimes.
OS-IMAGE CONTAINER-RUNTIME
Ubuntu 18.04.1 LTS containerd://1.2.0-rc.0
CentOS Linux 7 (Core) cri-o://1.11.10
Amazon Linux 2 docker://17.6.2
CentOS Linux 7 (Core) docker://18.6.1