Deploying Rook-Ceph on Google Kubernetes Engine(GKE)

Ashish Sharma
Jul 10 · 3 min read

Rook has gotten really popular by becoming a part of Cloud Native foundation and giving a solution for persistent storage on Kubernetes. Rook orchestrates different storage solution, but we would be setting up the ceph solution that has reached its first release.

I will be taking you through the requirements needed to setup rook on GKE, and setting them up.

Step 1. Start a Kubernetes Cluster with the ubuntu image

The Ceph requires RBD module which is not present in GKE’s default Container-Optimised OS(cos) as mentioned in the Issues 2448 & 2456. Thus, we cannot use the GKE’s default configuration. We will need to override GKE to use Ubuntu images for the node pool. Boot up the cluster with ubuntu image.

Step 2.Clone git repo of Rook and add FlexVolume environment variable

Download the rook release. Change the RELEASE variable value to download the release of your choice.

RELEASE="1.0.3"; wget https://github.com/rook/rook/archive/v${RELEASE}.zip && unzip v${RELEASE}.zip && rm v${RELEASE}.zip && cd rook-${RELEASE}cd cluster/examples/kubernetes/ceph

Rook uses FlexVolume to integrate with Kubernetes for performing storage operations. In GKE, the default Flexvolume plugin directory (the directory where FlexVolume drivers are installed) is read-only. So, the kubelet needs to be told to use a different FlexVolume plugin directory that is accessible and has read/write (rw) permission. This could be done by adding an environment variable in operator.yaml

Configure the Kubelet to use “/home/kubernetes/flexvolume” directory, by adding it to operator.yaml file as an environment variable

env:
[...]
- name: FLEXVOLUME_DIR_PATH
value: "/home/kubernetes/flexvolume"

Step 3. Deploy Rook operator

Next, deploy the Rook system components, which includes the Rook agent running on each node in your cluster as well as Rook’s operator pod.

kubectl create -f common.yaml
kubectl create -f operator.yaml
# verify the rook-ceph-operator, rook-ceph-agent, and rook-discover # pods are in the `Running` state before proceeding
kubectl -n rook-ceph get pod

Step 4. Deploy Rook cluster

Now that the Rook’s operator, agent, and discover pods are running, we can create the Rook Ceph cluster.

Save the cluster spec as cluster-test.yaml :

apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
# For the latest ceph images, see
# https://hub.docker.com/r/ceph/ceph/tags
image: ceph/ceph:v14.2.1-20190430
dataDirHostPath: /var/lib/rook
mon:
count: 3
dashboard:
enabled: true
storage:
useAllNodes: true
useAllDevices: false
# Important: Directories should only be used in pre-production
# environments
directories:
- path: /var/lib/rook

Create the cluster:-

kubectl create -f cluster-test.yaml

Step 5. Deploy StorageClass

Now, that the Rook-Ceph cluster is running we would be deploying the storage class so that we can request the persistent volume from ceph.

Save it as storage-class.yaml

apiVersion: ceph.rook.io/v1beta1
kind: Pool
metadata:
name: replicapool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: ceph.rook.io/block
parameters:
pool: replicapool
clusterNamespace: rook-ceph
fstype: xfs

Deploy it, kubectl apply -f storage-class.yaml

Step 6. Test PVC

Now test the Storage Class by requesting a Persistent Volume Claim.

Save it as pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc-rook-ceph-test
spec:
storageClassName: rook-ceph-block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi

Apply it, kubectl apply -f pvc.yaml, watch the PVC is bound.

Conclusion

From this Rook is configured on GKE you can test your chart at this deployment of Rook. But if you want a more detailed setup, follow the rook example: https://github.com/ceph/ceph-container/tree/master/examples/kubernetes/gke

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade