Cross-region file replication with Longhorn in Oracle Cloud and Kubernetes

Deploying a stateless application in multiple Kubernetes clusters on OCI is relatively straightforward. Create your desired number of clusters, use a global load balancer like OCI Traffic Steering, add health checks et voila!

What if your application is stateful and has persistence needs? How to ensure your data is persisted in another region? In this article, we will explore using Longhorn to replicate and persist your data in multiple OCI regions.

Introduction to Longhorn

Longhorn is a distributed block storage system for Kubernetes and is (at the time of writing) an incubating CNCF project. You can read more about Longhorn here but what attracted my attention was the ability to “create cross-cluster disaster recovery volumes so that data from a primary Kubernetes cluster can be quickly recovered from backup in a second Kubernetes cluster”.

This is the use case we will explore.

OKE Setup

First, create two OKE clusters. You can create them using the console or with terraform using the terraform-oci-oke module. My clusters are created in Sydney and Melbourne. You can choose your regions as appropriate.

I created the clusters using the Terraform module and enabled the operator in Sydney. As I’m working with multiple clusters, I’ll install kubectx and kubens to simplify my kubectl commands. If you are using this approach, it’s a good to set them up so you can switch clusters conveniently without constantly changing kubeconfigs or context. Run these on the operator:

oci ce cluster create-kubeconfig --cluster-id <clusterid> \
--file $HOME/.kube/config --region ap-melbourne-1 --token-version 2.0.0 --kube-endpoint PUBLIC_ENDPOINT
alias ktx='kubectx'
alias kns='kubens'
# use the contexts from your config file
ktx sydney=context-cenpnxriqhq
ktx melbourne=context-cydn4hk3qgq

To find the value of the context, you can check them in the $HOME/.kube/config.

Now to change a cluster, you can simply do the following:

[opc@lh-operator ~]$ ktx sydney
✔ Switched to context "sydney".

Longhorn pre-requisites

Let us now verify that our clusters can deploy Longhorn by running the Longhorn environment check script:

curl -sSfL https://raw.githubusercontent.com/longhorn/longhorn/v1.2.4/scripts/environment_check.sh -o environment_check.sh
for ctx in sydney melbourne; do
  ktx $ctx
  bash environment_check.sh
done
✔ Switched to context "sydney".
daemonset.apps/longhorn-environment-check created
waiting for pods to become ready (0/0)
waiting for pods to become ready (0/3)
waiting for pods to become ready (1/3)
all pods ready (3/3)
MountPropagation is enabled!
cleaning up...
daemonset.apps "longhorn-environment-check" deleted
clean up complete
✔ Switched to context "melbourne".
daemonset.apps/longhorn-environment-check created
waiting for pods to become ready (0/0)
waiting for pods to become ready (0/3)
all pods ready (3/3)
MountPropagation is enabled!
cleaning up...
daemonset.apps "longhorn-environment-check" deleted
clean up complete

We are now ready to deploy Longhorn in our clusters.

Deploy Longhorn

Let’s add the helm repo and update:

helm repo add longhorn https://charts.longhorn.io
helm repo update

And install Longhorn in both regions:

for ctx in sydney melbourne; do
  ktx $ctx
  helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace
done

Change the default namespace for both clusters:

for ctx in sydney melbourne; do
  ktx $ctx
  kns longhorn-system
done

Verify the pods are running in both clusters:

for ctx in sydney melbourne; do
  k get pods
done

You should see all the pods in Running state.

Prepare backup using OCI Object Storage

Create a bucket in the primary region (note that I’m equating a region with a cluster). We’ll imaginatively call this longhorn.

At the moment, Longhorn supports only S3 and NFSv4 for backup. Since OCI Object Storage already provides an S3 compatibility layer, we’ll use that.

First, create a Customer Secret Key. Ensure you note the secret key as it won’t be displayed again.

We also need to know the tenancy’s object storage namespace. Navigate to your tenancy’s page and look for this value.

In both clusters, create the same secret:

for ctx in sydney melbourne; do
  ktx $ctx
  kubectl create secret generic ocisecret \
    --from-literal=AWS_ACCESS_KEY_ID=123456ab7890123c45c \
    --from-literal=AWS_SECRET_ACCESS_KEY=1abcDEF3fgdg9d798dfgdsgd9gHHHD= \
 --from-literal=AWS_ENDPOINTS=<object-storage-namepace>.compat.objectstorage.<region>.oraclecloud.com \
 --from-literal=VIRTUAL_HOSTED_STYLE=dHJ1ZQ== \
    -n longhorn-system
done

Create the above secret, replacing the values for the access key, the secret access key as well as the object-storage-namespace and region. The region value should be for the primary cluster’s region.

Configure a backup target for Longhorn

Let’s access the Longhorn UI in both regions to configure a backup target by using port-forwarding:

ktx sydney
kubectl port-forward longhorn-ui-84f97cf569-z9t85 8000:8000

You should now be able to access the Longhorn UI in your browser at http://localhost:8000/:

Longhorn UI

Navigate to Setting > General and look for Backup Target. Enter the backup target and the name of the secret you created earlier:

and click Save. In the backup target above, longhorn is bucket name and “ap-sydney-1” is your OCI region identifier.

Repeat the above in the secondary cluster.

Let’s now test if the backup works. Create a Pod that uses Longhorn volumes by running this command:

kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v1.2.4/examples/pod_with_pvc.yaml

In the Longhorn UI, you should now be able to see a volume:

Select and create a backup:

Navigate in the OCI console and you should now see a backupstore subdirectory:

Now, access the Longhorn UI in the secondary cluster and navigate to backup page:

You should now be able to select the backup and create a Disaster Recovery Volume.

On the volume page, you can now see the DR volume:

DR volume

Activate it.

Activating the DR Volume

Once the DR Volume is activated, you can create a PV/PVC with it:

Data replication test scenario

We’ll now test data replication between the 2 clusters. The test is roughly based on this demo.

We’ll deploy a WordPress in the primary region, create some content and then switch to the secondary region. Below is the architecture we’ll create:

Let’s add the bitnami helm charts first:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Deploy WordPress

Let’s generate the helm chart yaml file:

helm show values bitnami/wordpress > wordpress.yaml

We can now edit it:

# parameters  #values  
wordpressUsername user
wordpressPassword "Secre7Pa55w0rd"
persistence.storageClass "longhorn"
mariadb.enabled true
mariadb.auth.rootPassword: "aVerySecureR007"
mariadb.auth.password: "bnwp123!@#"
mariadb.primary.persistence.storageClass: "longhorn"

Then, install WordPress

helm install wordpress bitnami/wordpress -f wordpress.yaml --namespace wordpress --create-namespace

Verify that 2 PVCs of storageclass longhorn have been created. One will be for WordPress itself and another will be for the database:

$ kubectl get pvc
NAME      STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-mysql-0   Bound    pvc-b2ffe5d4-728f-42d3-8e06-a4caffac260c   8Gi        RWO            longhorn       21m
wordpress      Bound    pvc-2e3d017a-7663-4ed9-88ea-86dc5e6d8e08   10Gi       RWO            longhorn       69s

Extract the load balancer ip:

export SERVICE_IP=$(kubectl get svc wordpress --output jsonpath=’{.status.loadBalancer.ingress[0].ip}’)
echo wordpress_url=http://$SERVICE_IP/

Use the WordPress URL to access WordPress. Login with wordpressUsername, wordpressPasswordand create a new post. Below is the post I created. It includes some text and the Longhorn image I uploaded.

Create Longhorn backups

Use the Longhorn UI to create a snapshot, backups for both the database and WordPress volumes.

On the backup page, you’ll now see two more backups:

Create DR volumes

Let’s now pretend we want to do some maintenance in Sydney and we want to switch traffic to the cluster in Melbourne. Remember to change the context to Melbourne and create a “wordpress” namespace:

ktx melbourne
kubectl create ns wordpress

First, switch to the Longhorn UI in Melbourne. Next, create a DR volume for both the database and WordPress. You should now be able to see the 2 additional DR volumes you created in the volume page:

Now, let’s activate the DR volumes by selecting and activating them from the volume menu:

Finally, we create a PVC for each.

Deploy WordPress in secondary region

We are now ready to deploy WordPress in Melbourne. Copy the wordpress.yaml to wordpress-dr.yaml. You can leave everything the same except for:

persistence.existingClaim: "wordpress"

This will ensure that when you deploy WordPress in the secondary region (aka Melbourne in this case), the helm chart will reuse the PVC created from the DR volume instead of creating a new one:

helm install wordpress bitnami/wordpress -f wordpress-dr.yaml

Repeat the steps above to retrieve the secondary region’s Load Balancer IP address. Login into WordPress and you should be seeing the post you created in Sydney:

Finally, if you click on the post you created, you should be able to see both the text and the image:

Conclusion

In this post, I wanted to show how to use Longhorn with OCI Object Storage, deploy it on OKE and mimic the process of performing a DR switch of a stateful application to an OKE cluster in another OCI region. In a future post, we’ll explore using Longhorn a bit more.

I hope you find this post useful.

Want to talk about it? Join us on our public Slack channel!