Kubernetes + OpenFaaS + runV + ARM64 Packet.net

OK, this is indeed my first article :)

What I would like to talk is how to build a bare-metal Serverless cloud by using OpenFaaS + Kubernetes + runV on ARM64 Packet.net machines.

This should be simple and nothing special, but after I complete the work, I found it’s actually very interesting, I will explain this later.

Now let’s go to the tech part first!

Step 0: Glossary

• Kubernetes: everyone knows this :)
• OpenFaaS: an awesome Serverless framework.
• runV: a hypervisor based OCI container runtime [1].
• hyperd: a API daemon for runV.
• kubernetes/frakti: Kubernetes CRI runtime shim for runV.

Step 1: Prepare ARM64 bare-metal servers on Packet.net

The machine size does not matter, just make sure it has access to external Internet. I will leave rest of work to you :)

This is information of my machines (provided by Qualcomm):

$  uname -m
aarch64
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"
$ uname -r
4.10.0-19-generic

NOTE: Step 2~5 should be done on every node you want to install Kubernetes.

Step 2: Install Docker engine

We are using mixed runtimes mode in Kubernetes here. All user workloads created by OpenFaaS will be run inside runV with better isolation, while Docker is used as auxiliary runtime to run system containers like kube-apiserver since they require host network.

The easiest way should be: curl -ssL get.docker.com | bash . But you can choose your favourite way.

Step 3: Install runV runtime

1. We will install hyperd (runV with grpc API daemon) from debs.

wget https://s3-us-west-1.amazonaws.com/hypercontainer-build/arm64/debian/hypercontainer_1.0.0-1_arm64.deb
wget https://s3-us-west-1.amazonaws.com/hypercontainer-build/arm64/debian/hyperstart_1.0.0-1_arm64.deb
dpkg -i hyperstart_1.0.0-1_arm64.deb
dpkg -i hypercontainer_1.0.0-1_arm64.deb

Please do apt-get install -f if any dependency issue occurs.

2. Start hyperd service, then the runV runtime should be ready to use:

systemctl enable hyperd && systemctl start hyperd
systemctl status hyperd
hyperctl list
POD ID               POD Name            VM name         Status

So far so good!

Step 4: Install kubeadm tool for Kubernetes

Not surprisingly, we will use kubeadm to deploy Kubernetes.

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl

This will install kubeadm and other tool binaries for you.

Step 5: Install runtime shim for Kubernetes

The key of Kubernetes’ ability to support multiple runtimes (Docker + runV in our case) is: it replies on CRI to communicate with underlying runtimes.

The runtime shim (CRI implementation) is exactly the intermedia layer between kubelet and runtimes. In our case, kubernetes/frakti shim is what we need.

Install runtime shim is very easy:

curl -sSL https://github.com/kubernetes/frakti/releases/download/v1.1.1/frakti.arm64 -o /usr/bin/frakti
chmod +x /usr/bin/frakti
cat <<EOF > /lib/systemd/system/frakti.service
[Unit]
Description=Hypervisor-based container runtime for Kubernetes
Documentation=https://github.com/kubernetes/frakti
After=network.target

[Service]
ExecStart=/usr/bin/frakti --v=3 \
          --log-dir=/var/log/frakti \
          --logtostderr=false \
          --listen=/var/run/frakti.sock \
          --hyper-endpoint=127.0.0.1:22318
MountFlags=shared
TasksMax=8192
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0
Restart=on-abnormal

[Install]
WantedBy=multi-user.target
EOF

Then do:

systemctl enable frakti
systemctl start frakti

Finally, configure kubelet to claim you want to use kubernetes/frakti as runtime shim.

cat > /etc/systemd/system/kubelet.service.d/20-cri.conf <<EOF
[Service]
Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --container-runtime-endpoint=/var/run/frakti.sock --feature-gates=AllAlpha=true"
EOF
systemctl daemon-reload

(Optional) Configure the node with a simplest bridge CNI plugin.

You don’t need to this step if you want to use third-party plugins like Flannel, Weave, Calico etc.

mkdir -p /etc/cni/net.d
cat >/etc/cni/net.d/10-mynet.conf <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
    "type": "bridge",
    "bridge": "cni0",
    "isGateway": true,
    "ipMasq": true,
    "ipam": {
        "type": "host-local",
        "subnet": "10.244.1.0/24",
        "routes": [
            { "dst": "0.0.0.0/0"  }
        ]
    }
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<-EOF
{
    "cniVersion": "0.3.0",
    "type": "loopback"
}
EOF

Please use 10.244.x.0/24 for different node.

Step 6: Install Kubernetes cluster

Guess what? This is the easiest part of this guide!

1. Init master node:

kubeadm init --pod-network-cidr 10.244.0.0/16 --kubernetes-version stable
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

(Optional) You can also un-taint master node so you can also deploy pod to it:

kubectl taint nodes --all node-role.kubernetes.io/master:NoSchedule-

2. Install network plugin to existing cluster:

This step is not needed if you’ve configured bridge CNI network on every node.

Otherwise, you should install you own network plugin like this:

export ARCH=arm64
curl -sSL "https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml?raw=true" | sed "s/amd64/${ARCH}/g" | kubectl create -f -

For any other plugins, please follow: https://github.com/kubernetes/frakti/blob/master/docs/networking.md, but please make sure you are using ARM64 version, I did not test them.

3. Patch kube-dns to make sure it has enough resource limit:

kubectl -n kube-system patch deployment kube-dns -p '{"spec":{"template":{"spec":{"containers":[{"name":"kubedns","resources":{"limits":{"memory":"256Mi"}}},{"name":"dnsmasq","resources":{"limits":{"memory":"128Mi"}}},{"name":"sidecar","resources":{"limits":{"memory":"64Mi"}}}]}}}}'

4. (Optional) Join more nodes to this cluster:

kubeadm join --token <token> <master-ip>:<master-port>

Step 7: Install OpenFaaS

This is quite simple, only require one line command on master node:

mkdir -p $GOPATH/src/github.com/openfaas
cd $GOPATH/src/github.com/openfaas
git clone https://github.com/openfaas/faas-netes
cd faas-netes
kubectl apply -f faas.arm64.yml
kubectl apply -f rbac.yml

The monitoring components seems not ready for AMR64 faas-netes as there’s no monitoring.arm64.yml present. I will update the guide as soon as it’s ready.

Then check if OpenFaaS pods are running:

kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
faas-netesd-6cc689c4cf-5ppr8   1/1       Running   0          8m
gateway-9b97bb58-pk4wx         1/1       Running   0          8m

Done!

Please note the gateway is running behind a NodePort Service defined in the cluster:

kubectl get svc
NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
faas-netesd   NodePort    10.100.134.153   <none>        8080:31111/TCP   8m
gateway       NodePort    10.104.88.179    <none>        8080:31112/TCP   8m
kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP          9d

So it should be able to reached by <NodeIP>:31112 from outside world. In my case the the public IP of node which gateway-9b97bb58-pk4wx is running on is 147.xx.xx.xx . I will use it to play.

Let’s play with OpenFaaS!

The UI/UX of OpenFaaS is awesome!

Alex has written a super cool guide to explore OpenFaaS: https://blog.alexellis.io/quickstart-openfaas-cli/, so I will skip functions creation part in this blog and re-use existing functions.

As long as you’ve created your function Docker image and pushed it to Docker Hub, then you can deploy it to Kubernetes with faas-cli.

In our demo, I will re-use a simple alpine Docker image as the function and claim to use its md5sum as my function process by — fprocess. So when I pass any parameter when invoking this Serverless function, it should return a corresponding md5sum result for me.

Let’s try:

$ faas-cli  deploy --image resouer/alpine:latest-arm64 --name checksum --fprocess md5sum --gateway http://147.xx.xx.xx:31112
Server returned unexpected status code 500 deployments.extensions "checksum" not found
Deployed.
URL: http://147.xx.xx.xx:31112/function/checksum
202 Accepted
$ echo "hello" | faas-cli invoke checksum  --gateway http://147.xx.xx.xx:31112
b1946ac92492d2347c6235b4d2611184  -

Work as charming!

Demo

You can find a screenshot of main steps of this demo here: https://asciinema.org/a/4CITJ9JCySnfoleRhwZ6yPm9p

What we learned

Well, here is the interesting part, let me explain.

If you look at this work from up-to-down, the cluster is actually composed of :

• OpenFaaS as API layer
• Kubernetes as orchestration and scheduling
• runV as container runtime
• Bare-metal Packet.net servers as infrastructure

Figure 1.1 Kubernetes + OpenFaaS + runV

See? There’s no either VM or IaaS: it’s “real” serverless.

Figure 1.1: User Pods are directly running on bare-metal servers, created by OpenFaaS, controlled by Kubernetes, that’s all.

This cluster is ready (almost) to serve public Internet since runV uses hardware virtualization to isolate user Pods.

What I found interesting is: OpenFaaS can act as an perfect API layer to hide Kubernetes from end users. The API of later, which is super powerful, but has a steep learning curve for developers. Comparing to that, I feel more conformable when reading Alex’s quickstart-openfaas-cli/.

The future?

This begin to remind me of the shape of public cloud in future. Still PaaS and SaaS layers on top of traditional IaaS cloud? Together with complex network and storage stack? Billing by machine size or Kubernetes node numbers?

Or, a unified and clean serverless API for users to consume? Use Kubernetes to package a group of bare-metal servers? Billing by seconds? And user workloads are running inside hypervisor based container runtime, connected by CNI network, do not need extra VM wrapper, and totally transparent to users?

The later, at least in many use cases I can think of, will attract me more.

What’s next?

In the subsequent articles, I would like to explain more details of this solution. And includes some performance data about runV runtime (do you know that a hypervisor pod starts within 376ms in the demo?).

And I will also update the doc as soon as monitoring.yaml for ARM64 are ready

• [1] Expect name change of this project in near future.