OK, this is indeed my first article :)
What I would like to talk is how to build a bare-metal Serverless cloud by using OpenFaaS + Kubernetes + runV on ARM64 Packet.net machines.
This should be simple and nothing special, but after I complete the work, I found it’s actually very interesting, I will explain this later.
Now let’s go to the tech part first!
Step 0: Glossary
- Kubernetes: everyone knows this :)
- OpenFaaS: an awesome Serverless framework.
- runV: a hypervisor based OCI container runtime [1].
- hyperd: a API daemon for runV.
- kubernetes/frakti: Kubernetes CRI runtime shim for runV.
Step 1: Prepare ARM64 bare-metal servers on Packet.net
The machine size does not matter, just make sure it has access to external Internet. I will leave rest of work to you :)
This is information of my machines (provided by Qualcomm):
$ uname -m
aarch64$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"$ uname -r
4.10.0-19-generic
NOTE: Step 2~5 should be done on every node you want to install Kubernetes.
Step 2: Install Docker engine
We are using mixed runtimes mode in Kubernetes here. All user workloads created by OpenFaaS will be run inside runV with better isolation, while Docker is used as auxiliary runtime to run system containers like kube-apiserver since they require host network.
The easiest way should be: curl -ssL get.docker.com | bash
. But you can choose your favourite way.
Step 3: Install runV runtime
- We will install
hyperd
(runV with grpc API daemon) from debs.
wget https://s3-us-west-1.amazonaws.com/hypercontainer-build/arm64/debian/hypercontainer_1.0.0-1_arm64.debwget https://s3-us-west-1.amazonaws.com/hypercontainer-build/arm64/debian/hyperstart_1.0.0-1_arm64.debdpkg -i hyperstart_1.0.0-1_arm64.deb
dpkg -i hypercontainer_1.0.0-1_arm64.deb
Please do apt-get install -f
if any dependency issue occurs.
2. Start hyperd
service, then the runV runtime should be ready to use:
systemctl enable hyperd && systemctl start hyperd
systemctl status hyperdhyperctl list
POD ID POD Name VM name Status
So far so good!
Step 4: Install kubeadm tool for Kubernetes
Not surprisingly, we will use kubeadm to deploy Kubernetes.
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
This will install kubeadm
and other tool binaries for you.
Step 5: Install runtime shim for Kubernetes
The key of Kubernetes’ ability to support multiple runtimes (Docker + runV in our case) is: it replies on CRI to communicate with underlying runtimes.
The runtime shim (CRI implementation) is exactly the intermedia layer between kubelet and runtimes. In our case, kubernetes/frakti
shim is what we need.
Install runtime shim is very easy:
curl -sSL https://github.com/kubernetes/frakti/releases/download/v1.1.1/frakti.arm64 -o /usr/bin/frakti
chmod +x /usr/bin/frakticat <<EOF > /lib/systemd/system/frakti.service
[Unit]
Description=Hypervisor-based container runtime for Kubernetes
Documentation=https://github.com/kubernetes/frakti
After=network.target
[Service]
ExecStart=/usr/bin/frakti --v=3 \
--log-dir=/var/log/frakti \
--logtostderr=false \
--listen=/var/run/frakti.sock \
--hyper-endpoint=127.0.0.1:22318
MountFlags=shared
TasksMax=8192
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
EOF
Then do:
systemctl enable frakti
systemctl start frakti
Finally, configure kubelet
to claim you want to use kubernetes/frakti
as runtime shim.
cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=/var/run/frakti.sock --feature-gates=AllAlpha=true"
EOFsystemctl daemon-reload
(Optional) Configure the node with a simplest bridge CNI plugin.
You don’t need to this step if you want to use third-party plugins like Flannel, Weave, Calico etc.
mkdir -p /etc/cni/net.d
cat >/etc/cni/net.d/10-mynet.conf <<-EOF
{
"cniVersion": "0.3.0",
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.1.0/24",
"routes": [
{ "dst": "0.0.0.0/0" }
]
}
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<-EOF
{
"cniVersion": "0.3.0",
"type": "loopback"
}
EOF
Please use 10.244.x.0/24
for different node.
Step 6: Install Kubernetes cluster
Guess what? This is the easiest part of this guide!
1. Init master node:
kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version stablemkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
(Optional) You can also un-taint master node so you can also deploy pod to it:
kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule-
2. Install network plugin to existing cluster:
This step is not needed if you’ve configured bridge CNI network on every node.
Otherwise, you should install you own network plugin like this:
export ARCH=arm64curl -sSL "https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml?raw=true" | sed "s/amd64/${ARCH}/g" | kubectl create -f -
For any other plugins, please follow: https://github.com/kubernetes/frakti/blob/master/docs/networking.md, but please make sure you are using ARM64 version, I did not test them.
3. Patch kube-dns to make sure it has enough resource limit:
kubectl -n kube-system patch deployment kube-dns -p '{"spec":{"template":{"spec":{"containers":[{"name":"kubedns","resources":{"limits":{"memory":"256Mi"}}},{"name":"dnsmasq","resources":{"limits":{"memory":"128Mi"}}},{"name":"sidecar","resources":{"limits":{"memory":"64Mi"}}}]}}}}'
4. (Optional) Join more nodes to this cluster:
kubeadm join --token <token> <master-ip>:<master-port>
Step 7: Install OpenFaaS
This is quite simple, only require one line command on master node:
mkdir -p $GOPATH/src/github.com/openfaas
cd $GOPATH/src/github.com/openfaasgit clone https://github.com/openfaas/faas-netes
cd faas-neteskubectl apply -f faas.arm64.yml
kubectl apply -f rbac.yml
The
monitoring
components seems not ready for AMR64faas-netes
as there’s nomonitoring.arm64.yml
present. I will update the guide as soon as it’s ready.
Then check if OpenFaaS
pods are running:
kubectl get podsNAME READY STATUS RESTARTS AGEfaas-netesd-6cc689c4cf-5ppr8 1/1 Running 0 8mgateway-9b97bb58-pk4wx 1/1 Running 0 8m
Done!
Please note the gateway
is running behind a NodePort
Service defined in the cluster:
kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEfaas-netesd NodePort 10.100.134.153 <none> 8080:31111/TCP 8mgateway NodePort 10.104.88.179 <none> 8080:31112/TCP 8mkubernetes ClusterIP 10.96.0.1 <none> 443/TCP 9d
So it should be able to reached by <NodeIP>:31112
from outside world. In my case the the public IP of node which gateway-9b97bb58-pk4wx
is running on is 147.xx.xx.xx
. I will use it to play.
Let’s play with OpenFaaS!
The UI/UX of OpenFaaS is awesome!
Alex has written a super cool guide to explore OpenFaaS: https://blog.alexellis.io/quickstart-openfaas-cli/, so I will skip functions creation part in this blog and re-use existing functions.
As long as you’ve created your function Docker image and pushed it to Docker Hub, then you can deploy it to Kubernetes with faas-cli.
In our demo, I will re-use a simple alpine
Docker image as the function and claim to use its md5sum
as my function process by — fprocess
. So when I pass any parameter when invoking this Serverless function, it should return a corresponding md5sum
result for me.
Let’s try:
$ faas-cli deploy --image resouer/alpine:latest-arm64 --name checksum --fprocess md5sum --gateway http://147.xx.xx.xx:31112Server returned unexpected status code 500 deployments.extensions "checksum" not foundDeployed.URL: http://147.xx.xx.xx:31112/function/checksum202 Accepted$ echo "hello" | faas-cli invoke checksum --gateway http://147.xx.xx.xx:31112b1946ac92492d2347c6235b4d2611184 -
Work as charming!
Demo
You can find a screenshot of main steps of this demo here: https://asciinema.org/a/4CITJ9JCySnfoleRhwZ6yPm9p
What we learned
Well, here is the interesting part, let me explain.
If you look at this work from up-to-down, the cluster is actually composed of :
- OpenFaaS as API layer
- Kubernetes as orchestration and scheduling
- runV as container runtime
- Bare-metal Packet.net servers as infrastructure
See? There’s no either VM or IaaS: it’s “real” serverless.
Figure 1.1: User Pods are directly running on bare-metal servers, created by OpenFaaS, controlled by Kubernetes, that’s all.
This cluster is ready (almost) to serve public Internet since runV uses hardware virtualization to isolate user Pods.
What I found interesting is: OpenFaaS can act as an perfect API layer to hide Kubernetes from end users. The API of later, which is super powerful, but has a steep learning curve for developers. Comparing to that, I feel more conformable when reading Alex’s quickstart-openfaas-cli/.
The future?
This begin to remind me of the shape of public cloud in future. Still PaaS and SaaS layers on top of traditional IaaS cloud? Together with complex network and storage stack? Billing by machine size or Kubernetes node numbers?
Or, a unified and clean serverless API for users to consume? Use Kubernetes to package a group of bare-metal servers? Billing by seconds? And user workloads are running inside hypervisor based container runtime, connected by CNI network, do not need extra VM wrapper, and totally transparent to users?
The later, at least in many use cases I can think of, will attract me more.
What’s next?
In the subsequent articles, I would like to explain more details of this solution. And includes some performance data about runV runtime (do you know that a hypervisor pod starts within 376ms in the demo?).
And I will also update the doc as soon as monitoring.yaml for ARM64 are ready
- [1] Expect name change of this project in near future.