Using GlusterFS on Oracle Cloud Infrastructure
This tutorial describes the deployment steps of a high availability GlusterFS Storage environment on Oracle Bare Metal Instance using a Distributed Glusterfs Volume.
There are different GlusterFS types of volumes you can use as explained in the GlusterFS public documentation and it’s important that you understand them first before proceeding with the initial configuration. This tutorial demonstrates how to use a Distributed GlusterFS Volume on two Bare Metal Instances being accessed by one CentOS 7 Client.
As for the configuration option on Oracle Cloud Infrastructure (OCI), you might either keep all GlusterFS resources in a single availability domain (AD) or use all ADs as shown below. Depending on the network traffic utilization you should consider having all GlusterFS resources in a single AD or use a distributed option across all availability domains for fault tolerance. Make sure you add a Gluster Replicated Volume option in your setup which is recommended to avoid data loss and for production environments. In addition to using a replicated GlusterFS Volume for fault tolerance (Distributed and Replicated Volume) you should enable GlusterFS Trash Translator and Snapshots to assist you with file recovery if needed.
Here is an example of a GlusterFS architecture that can be used on OCI
What is GlusterFS?
Gluster is a distributed scale out filesystem that allows rapid provisioning of additional storage based on your storage consumption needs. It incorporates automatic failover as a primary feature. All of this is accomplished without a centralized metadata server.
First of all, this tutorial is based on GlusterFS Storage Cluster on CentOS 7 article. Secondly, for this setup you need to create at least three Oracle Bare Metal instances; two instances will be used to hold the Distributed GlusterFS Volume and the third instance as a GlusterFS Client with a local mount point to GlusterFS volume. As an optional step for this tutorial, you can create two 100GB block volumes and attach them to each GlusterFS server (two GlusterFS Servers with two 100GB block volumes) that will be used later to create a single 200GB GlusterFS volume as shown in the above picture.
GlusterFS Server Installation
Choose a package source: either the CentOS Storage SIG or Gluster.org. This article describes how to use CentOS Storage SIG Packages
$ sudo yum search centos-release-gluster
$ sudo yum install centos-release-gluster310.noarch
Install GlusterFS Server and Samba packages on both storage cluster nodes.
$ sudo yum install glusterfs-server samba –y
Skip these optional steps if your environment is already have prepared XFS bricks (partitions). For this article, you will use /dev/sdb device which is the 100GB attached block volume you have created before as described above. Change the device name if necessary based on your current configuration. Now, create brick1 Logical Volumes for XFS bricks on both cluster nodes as shown below
$ sudo pvcreate /dev/sdb
$ sudo vgcreate vg_gluster /dev/sdb
$ sudo lvcreate -L 100000M -n brick1 vg_gluster
Setup XFS file systems:
$ sudo mkfs.xfs /dev/vg_gluster/brick1
Create a mount point and mount the XFS brick:
$ sudo mkdir -p /bricks/brick1
$ sudo mount /dev/vg_gluster/brick1 /bricks/brick1
Extend the /etc/fstab inserting the following line
/dev/vg_gluster/brick1 /bricks/brick1 xfs defaults,_netdev,nofail 0 0
Important Note: as described in the Oracle Bare Metal Public documentation about when connecting with iSCSI on linux instances block volumes, it’s important to include the “_netdev” and “nofail” options on every non-root block volume in /etc/fstab or the instance may fail to launch as the OS tries to mount the volume(s) before the iSCSI initiator has started.
For example, a single volume in /etc/fstab might look like this:
/dev/sdb /data1 ext4 defaults,noatime,_netdev,nofail 0 2
If you reboot without these options in fstab, the instance will fail to start after the next reboot. Instances in this state are not recoverable.
Trusted Pool (Storage Cluster)
Enable and start glusterfsd.service on both nodes:
$ sudo systemctl enable glusterd.service
$ sudo systemctl start glusterd.service
Ports TCP:24007–24008 are required for communication between GlusterFS nodes and each brick requires another TCP port starting at 24009.
Enable required ports on the firewall:
You also need to open ports 24007–24008 on the BMCS Dashboard. You need to edit the VCN Security List and either open all ports for the Bare Metal Internal Network (NOT PUBLIC NETWORK) as shown below for
IP Protocol: All Protocols
Allows: all traffic for all ports
or only opening the required 24007–24008 ports for the internal network. Here is an example:
IP Protocol: TCP
Source Port Range: All
Destination Port Range: 24007–24008
Allows: TCP traffic for ports: 24007–24008
Now, use gluster command to connect the second GlusterFS node and create a Trusted Pool (Storage Cluster).
$ sudo gluster peer probe gfsserver1
peer probe: success
Verify cluster peer:
$ sudo gluster peer status
Number of Peers: 1
State: Peer in Cluster (Connected)
High Availability GlusterFS Volumes
GlusterFS Volume works with Gluster File System which is a logical collection of XFS bricks. The following table shows dependencies between Volume types and sizes:
NOTE: for production environments it is recommended using Distributed GlusterFS volume along with Replicated option for security reasons.
Now, open the required port on the firewall. Remember, each brick in the GlusterFS Volume requires a TCP port starting at 24009:
$ sudo firewall-cmd --zone=public --add-port=24009/tcp -\-permanent
$ sudo firewall-cmd --reload
NOTE: the same port needs to be opened on BMCS Dashboard as well in case you haven’t opened all ports for internal network.
Use the /bricks/brick1 XFS partition on both nodes to create a highly available Replicated Volume. First create a sub-directory in /bricks/brick1 mount point. It will be necessary for GlusterFS.
$ sudo mkdir /bricks/brick1/brick
Create a Distributed GlusterFS Volume running these commands on the first node only (gfsserver)
$ sudo gluster volume create glustervol1 transport tcp gfsserver:/bricks/brick1/brick gfsserver1:/bricks/brick1/brick
$ sudo gluster volume start glustervol1
Verify the GlusterFS volume
$ sudo gluster volume info all
Volume Name: glustervol1
Volume ID: c660712f-29ea-4288–96b6–2c0a0c85a82a
Snapshot Count: 0
Number of Bricks: 1
GlusterFS volumes can be accessed using GlusterFS Native Client (CentOS / RedHat / OracleLinux 6.5 or later), NFS v3 (other Linux clients), or CIFS (Windows clients).
Open the Firewall for Glusterfs/NFS/CIFS Clients
NOTE: as already mentioned above the same ports need to be opened on BMCS Dashboard.
Access from another CentOS 7 machine via GlusterFS Native Client
All required packages are available by default in the CentOS 7 Base repository.
Install GlusterFS Client packages:
$ sudo yum install glusterfs glusterfs-fuse attr -y
Mount GlusterFS Volumes on the client:
$ sudo mkdir /glusterfs
$ sudo mount -t glusterfs gfsserver:/glustervol1 /glusterfs
$ sudo mount |grep -i glusterfs
gfsserver:/glustervol1 on /glusterfs type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
$ df -kh |grep glusterfs
gfsserver:/glustervol1 196G 65M 98G 1% /glusterfs
to make it permanent across restarts you need to add the following line in /etc/fstab on all GlusterFS Linux clients
gfsserver:/glustervol1 /glusterfs glusterfs defaults,_netdev,nofail 0 0
Check the GlusterFS public documentation for more details about:
- Troubleshooting steps
- How to to setup Windows/Linux machines access via CIFS to GlusterFS Volumes
Originally published at community.oracle.com on June 28, 2017.