Deploying OpenWhisk on Kubernetes

I have been running an experiment from past couple of days to deploy OpenWhisk on Kubernetes to create an easy and quick playground. I am newbie 👶 to Kubernetes and I guess that’s the reason I took one and half day to have a successful deployment otherwise GitHub repo incubator-openwhisk-deploy-kube has very clear instructions on how to deploy OpenWhisk on Kubernetes.

Here is the list of instructions I followed on my Mac, but before you start experimenting, please note that:

OpenWhisk deployment only and only works with Kubernetes 1.8.0 or 1.9.0 and minikube 0.25.2. I experimented using Kubernetes 1.9.0 with single node cluster.

1. Install asdf package manager and edit ~/.bash_profile or equivalent:

$ brew install asdf
# add the following statement in ~/.bash_profile
[ -s "/usr/local/opt/asdf/asdf.sh" ] && . /usr/local/opt/asdf/asdf.sh
$ source ~/.bash_profile

2. Add minikube and kubectl plugins:

$ asdf plugin-add kubectl
$ asdf plugin-add minikube

3. Install minikube and kubectl using asdf :

$ asdf install kubectl 1.9.0
$ asdf global kubectl 1.9.0
$ asdf install minikube 0.25.2
$ asdf global minikube 0.25.2

4. Create the minikube VM:

$ minikube start --cpus 2 --memory 4096 --kubernetes-version=v1.9.0 --extra-config=apiserver.Authorization.Mode=RBAC

5. Setup Docker network in promiscuous mode:

$ minikube ssh -- sudo ip link set docker0 promisc on

So far, I have kubectl and minikube:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2018-01-26T19:04:38Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}
$ minikube version
minikube version: v0.25.2

6. Create namespace OpenWhisk and label Kubernetes worker nodes:

$ kubectl create namespace openwhisk
$ kubectl label nodes --all openwhisk-role=invoker

7. Clone GitHub repo and define the type of ingress in yaml file:

$ git clone https://github.com/apache/incubator-openwhisk-deploy-kube.git
$ cd incubator-openwhisk-deploy-kube/helm
$ vim mycluster.yaml
whisk:
    ingress:
        type: NodePort
            api_host_name: 192.168.99.100
            api_host_port: 31001
nginx:
    httpsNodePort: 31001

8. Deploy with Helm:

$ brew install kubernetes-helm
$ helm init # init Helm Tiller
$ kubectl get pods -n kube-system # verify that tiller-deploy is in the running state
$ kubectl create clusterrolebinding tiller-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
$ helm install . --namespace=openwhisk -f mycluster.yaml

9. Verify that all pods are in running state:

$ kubectl get pods -n openwhisk
NAME                         READY     STATUS    RESTARTS   AGE
apigateway-7dbf56f8b-ppqjf   1/1       Running   0          55m
controller-0                 1/1       Running   4          55m
couchdb-674b4dbd8d-q22lc     1/1       Running   0          55m
invoker-cl6js                1/1       Running   0          55m
kafka-0                      1/1       Running   0          55m
nginx-6d66cccd6-qxgns        1/1       Running   0          55m
redis-7d4dbd56c7-qnzbq       1/1       Running   0          55m
zookeeper-66774495d6-m8phc   1/1       Running   0          55m

Hooray! 💃 We have a working instance of OpenWhisk on Kubernetes cluster. We can start playing with our OpenWhisk instance using wsk CLI or wskdeploy, after configuring host and auth with:

$ wsk property set --apihost 192.168.99.100:31001
$ wsk property set --auth `cat ../kubernetes/cluster-setup/auth.guest`
$ wsk -i list
Entities in namespace: default
packages
actions
triggers
rules

Troubleshooting:

I ended up spending more than a day troubleshooting and debugging OpenWhisk deployment on Kubernetes due to following issues:

Issue 1: getsockopt: connection refused

After carefully following through the instructions above, while verifying that OpenWhisk, I was bummed to see getsockopt error.

$ wsk -i list
error: Unable to obtain the list of entities for namespace 'default': Get http://192.168.99.100:31001/api/v1/namespaces/_/actions?limit=0&skip=0: dial tcp 192.168.99.100:31001: getsockopt: connection refused

After hitting this error, I checked the state of pods under openwhisk namespace with kubectl get pods -n openwhisk and found one faulty with CreateContainerConfigError.

Issue 2: CreateContainerConfigError — Ngnix Pod

Some versions of Kubernetes does not work for OpenWhisk deployment due to bug with volume mount subpaths (see issue #61076). Ngnix configuration has configMap volume in ngnix.yaml. The fix is to pick a version of Kubernetes which doesn’t have this bug, like 1.9.0. Now, my concern was I had made sure to install 1.9.0 version of Kubernetes than why am I running into this kind of issue. I tried to verify and found that my server was running 1.9.4 😏

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"", Minor:"", GitVersion:"v1.9.4", GitCommit:"bee2d1505c4fe820744d26d41ecd3fdd4a3d6546", GitTreeState:"clean", BuildDate:"2018-03-21T21:48:36Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

The reason was I started minikube without any parameters and by default it picked up 1.9.4 😱

minikube start

v/s

minikube start --cpus 2 --memory 4096 --kubernetes-version=v1.9.0 --extra-config=apiserver.Authorization.Mode=RBAC

Issue 3: Invoker still initializing — Init:1/2

Invoker might take long time to pull all possible runtimes specified in runtimes.json. If it takes too long and if you would like to have OpenWhisk running quickly, you can pull minimal runtimes by adding runtimes-minimal-travis.json to mycluster.yaml:

whisk:
  runtimes: "runtimes-minimal-travis.json"

Please feel free to give it a try and let me know if you have any questions or run into any issues. Good Luck 👍