Building a hybrid x86–64 and ARM Kubernetes Cluster

Carlos Eduardo
Jan 29 · 10 min read

Overview

Hardware

Pine64 RockPro64
I might build a case to assemble the SBCs properly soon

Software

$ docker container run — rm mplatform/mquery gcr.io/google-containers/kube-apiserver:v1.13.1Image: gcr.io/google-containers/kube-apiserver:v1.13.1
* Manifest List: Yes
* Supported platforms:
- linux/amd64
- linux/arm
- linux/arm64
- linux/ppc64le
- linux/s390x

Preparing the nodes

[all:vars]
ansible_connection=ssh
ansible_ssh_user=root
ansible_ssh_pass=rootpassword
[nodes]
192.168.1.10
192.168.1.11
192.168.1.12

Step-by-step manual setup

sudo apt-get update# Install SBC utility packagessudo apt install -y nfs-common less vim ack git build-essential iptables ipset pciutils lshw file iperf3 net-tools lsb-release# Fix ping permission
sudo chmod +s /bin/ping*
# Install Docker pre-requisitessudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg2 \
software-properties-common
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -echo “deb [arch=arm64] https://download.docker.com/linux/debian \
$(lsb_release -cs) stable” | \
sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install docker-ce=18.06.1~ce~3–0~debian
# Check if the interface MAC addresses does not conflict between boards
ip link
# Check if the board UUID and Machine-Id does not conflict between boards
sudo cat /sys/class/dmi/id/product_uuid
sudo cat /etc/machine-id
apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl

Creating the Cluster

kubeadm join --token secret.verysecrettokk8s 192.168.1.50:6443 --discovery-token-ca-cert-hash sha256:a57508843e3a356303d87972288571064cbf215f4fba5cb502f9d8370ef5c354
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f “https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d ‘\n’)&env.NO_MASQ_LOCAL=1”# Download the Weave Net tool to troubleshoot in case needed
sudo curl -L git.io/weave -o /usr/local/bin/weave
sudo chmod a+x /usr/local/bin/weave
kubectl taint nodes --all node-role.kubernetes.io/master-
...
spec:
containers:
- command:
- kube-apiserver
- --authorization-mode=Node,RBAC
...
- --feature-gates=TTLAfterFinished=true
...
NAME STATUS ROLES AGE VERSION LABELSk8s-armnode1 Ready <none> 4m18s v1.13.1 beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=k8s-armnode1k8s-armnode2 Ready <none> 4m8s v1.13.1 beta.kubernetes.io/arch=arm64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=k8s-armnode2k8s-master Ready master 10m v1.13.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=k8s-master,node-role.kubernetes.io/master=

Additional Cluster Tools

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yamlkubectl apply -f ./1-MetalLB/metallb-conf.yaml
cd 2-Traefik/
./deploy
# Run all on the Master Nodesudo apt-get install nfs-kernel-server nfs-common
sudo systemctl enable nfs-kernel-server
sudo systemctl start nfs-kernel-server
sudo cat >> /etc/exports <<EOF
/data/kubernetes-storage/ 192.168.1.*(rw,sync,no_subtree_check,no_root_squash)
EOF
sudo exportfs -a
cd 3-NFS_Storage
kubectl apply -f *
cd 4-Dashboard
kubectl apply -f dashboard-admin-account.yaml
kubectl apply -f dashboard.yaml
kubectl apply -f dashboard-ingress.yaml
kubectl patch deployment kubernetes-dashboard -n kube-system — patch ‘{“spec”: {“template”: {“spec”: {“nodeSelector”: {“beta.kubernetes.io/arch”: “arm64”}}}}}’
cd 5-Metrics-server
kubectl apply -f *
$ kubectl top nodesNAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
k8s-master 1152m 19% 1311Mi 35%
k8s-node1 292m 4% 689Mi 18%
$ kubectl top podsNAME CPU(cores) MEMORY(bytes)
coredns-86c58d9df4-tk7g9 14m 11Mi
coredns-86c58d9df4-vp942 16m 11Mi
etcd-k8s-master 130m 132Mi
kube-apiserver-k8s-master 207m 423Mi
kube-controller-manager-k8s-master 170m 61Mi
kube-proxy-6xbsp 23m 20Mi
kube-proxy-7zxth 27m 15Mi
cd 6-Helm
./deploy
cd 7-WeaveScope
kubectl apply -f .

Conclusion

Carlos Eduardo

Written by

Writing everything cloud and all the tech behind it. If you like my projects and would like to support me, check my Patreon on https://www.patreon.com/carlosedp

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade