End-to-end Kubeflow installation on your personal computer with Kind

Need to try Kubeflow on your computer ? There are few solutions to do that : MiniKF, MiniKube, Kind, etc. In this article, I detail the installation process to make Kubeflow works locally with Kind.

Kind is a tool for running Kubernetes locally on your machine. It was released in order to easily test and debug kubernetes clusters on your machine.

The official documentation to install Kind can be found here; the official github can be found here.

FYI, everything done below was done with a Linux Ubuntu 18.04.

KinD installation

In order to install Kind, we first need to install Go (an open source programming language) and docker. Kubectl, a command line tool for controlling a K8s (Kubernetes) cluster, should be installed too.

Install go

Go to the official DL page of go and download the binary release that matches your configuration. I chose the binary release 1.15 for Linux.

Extract the downloaded file to /usr/local :

tar -C /usr/local/ -xzf go1.15.linux-amd64.tar.gz

Add go binaries to your path. Open /etc/profile and add the following line at the end of the file :

export PATH=$PATH:/usr/local/go/bin

Restart your computer and you should be OK with Go installation.

Install Docker

In order to run Kind, you will also need to install Docker. Go to the official DL page and select your environnement. I chose Ubuntu and was redirect to that page.

First, remove your old Docker installation :

sudo apt-get remove docker docker-engine docker.io containerd runc

Then, install the new one :

sudo apt-get update

sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common

Add docker GPD key :

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Set up the stable repository (case of x86_64) :

sudo add-apt-repository “deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable”

Install docker with the updated package index :

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

At that step, I had trouble with the deamon.json file. If you have this error too :

unable to configure the Docker daemon with file /etc/docker/daemon.json: EOF

Then, don’t panic and edit file /etc/docker/daemon.json by adding ‘{}’.

You are now able to test your installation using :

sudo docker run hello-world

Now, you should be OK with the Docker installation.

Common issue : Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/json: dial unix /var/run/docker.sock: connect: permission denied

You can do :

sudo chmod 666 /var/run/docker.sock

Install Kubectl

Kubectl is your favorite tool to manage your kubernetes cluster. To install kubectl, follow official instruction here. Here are my installation steps (yours can differ).

Download kubectl :

curl -LO “https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"

Make it executable :

chmod +x ./kubectl

Move it with other executables :

sudo mv ./kubectl /usr/local/bin/kubectl

And you should be good with kubectl too !

Install Kind

Installing Kind with official command (GO111MODULE=”on” go get sigs.k8s.io/kind@v0.8.1 && kind create cluster) did not work for.

So I created a directory named kind-source and clone the official repo in it with (you’ll need to have git installed) :

git clone https://github.com/kubernetes-sigs/kind.git

Build from sources with :

cd kind

make build

Then, add kind to your path, open /etc/profile file and add the following line at the end :

export PATH=${PATH}:$(go env GOPATH)/bin

You sould be OK with the Kind installation.

Kubeflow installation

After installing Kind, you are now ready to install Kubeflow.

First, create a kubernetes cluster (v1.14.1 as recommended by Kubeflow developpers) with kind :

kind create cluster — image kindest/node:v1.14.1 — name kind-1.14.1

Download kfctl v1.1.0 here.

Move the downloaded file where you want Kubeflow to be installed. I created ~/Kubeflow-with-kind dir. Then move the .tar.gz to it :

mv kfctl_v1.1.0–0-g9a3621e_linux.tar.gz ~/Kubeflow-with-Kind

Tar it and add kfctl to your path :

tar -xvf kfctl_v1.1.0–0-g9a3621e_linux.tar.gz

export PATH=$PATH:/home/your-user/Kubeflow-with-Kind

You should be OK with Kubeflow installation.

Deploy Kubeflow

At this step, everything is installed; we are now able to dpeloy Kubeflow to our Kind cluster.

Create a kf directory where you want and go in it :

mkdir -p kf-dir

cd kf-dir

Deploy Kubeflow using the official manifest :

kfctl apply -V -f https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_k8s_istio.v1.0.2.yaml

The deployment can be stuck. I let it loop and it solves itself with time…

In order to access the Kubeflow UI, you will need to do it through the Istio gateway (istio-ingressgateway). Playing with kind, there is no loadbalancer available, so you need to use nodePort or port forwarding to access to the KF dashboard.

Example using port forwarding to access to KF pipeline:

kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80

Or you can access to the KF dashboard by using (does not work for me, currently trying to find a solution):

kubectl port-forward -n istio-system svc/istio-ingressgateway 8080:80

Now if you go to http://localhost:8080/, surprise !

Conclusion

Installing Kubeflow is not very difficult if you are a data engineer. But it can be tricky for data scientists…
Hope this article will save few hours to data scientists.

Comment if you have any question or correction.