Using GlusterFS on Oracle Cloud Infrastructure

This tutorial describes the deployment steps of a high availability GlusterFS Storage environment on Oracle Bare Metal Instance using a Distributed Glusterfs Volume.

There are different GlusterFS types of volumes you can use as explained in the GlusterFS public documentation and it’s important that you understand them first before proceeding with the initial configuration. This tutorial demonstrates how to use a Distributed GlusterFS Volume on two Bare Metal Instances being accessed by one CentOS 7 Client.

As for the configuration option on Oracle Cloud Infrastructure (OCI), you might either keep all GlusterFS resources in a single availability domain (AD) or use all ADs as shown below. Depending on the network traffic utilization you should consider having all GlusterFS resources in a single AD or use a distributed option across all availability domains for fault tolerance. Make sure you add a Gluster Replicated Volume option in your setup which is recommended to avoid data loss and for production environments. In addition to using a replicated GlusterFS Volume for fault tolerance (Distributed and Replicated Volume) you should enable GlusterFS Trash Translator and Snapshots to assist you with file recovery if needed.

Here is an example of a GlusterFS architecture that can be used on OCI

What is GlusterFS?

Gluster is a distributed scale out filesystem that allows rapid provisioning of additional storage based on your storage consumption needs. It incorporates automatic failover as a primary feature. All of this is accomplished without a centralized metadata server.

Getting Started

First of all, this tutorial is based on GlusterFS Storage Cluster on CentOS 7 article. Secondly, for this setup you need to create at least three Oracle Bare Metal instances; two instances will be used to hold the Distributed GlusterFS Volume and the third instance as a GlusterFS Client with a local mount point to GlusterFS volume. As an optional step for this tutorial, you can create two 100GB block volumes and attach them to each GlusterFS server (two GlusterFS Servers with two 100GB block volumes) that will be used later to create a single 200GB GlusterFS volume as shown in the above picture.


GlusterFS Server Installation

Choose a package source: either the CentOS Storage SIG or This article describes how to use CentOS Storage SIG Packages

$ sudo yum search centos-release-gluster

$ sudo yum install centos-release-gluster310.noarch

Install GlusterFS Server and Samba packages on both storage cluster nodes.

$ sudo yum install glusterfs-server samba –y

XFS Bricks

Skip these optional steps if your environment is already have prepared XFS bricks (partitions). For this article, you will use /dev/sdb device which is the 100GB attached block volume you have created before as described above. Change the device name if necessary based on your current configuration. Now, create brick1 Logical Volumes for XFS bricks on both cluster nodes as shown below

$ sudo pvcreate /dev/sdb

$ sudo vgcreate vg_gluster /dev/sdb

$ sudo lvcreate -L 100000M -n brick1 vg_gluster

Setup XFS file systems:

$ sudo mkfs.xfs /dev/vg_gluster/brick1

Create a mount point and mount the XFS brick:

$ sudo mkdir -p /bricks/brick1

$ sudo mount /dev/vg_gluster/brick1 /bricks/brick1

Extend the /etc/fstab inserting the following line

/dev/vg_gluster/brick1 /bricks/brick1 xfs defaults,_netdev,nofail 0 0

Important Note: as described in the Oracle Bare Metal Public documentation about when connecting with iSCSI on linux instances block volumes, it’s important to include the “_netdev” and “nofail” options on every non-root block volume in /etc/fstab or the instance may fail to launch as the OS tries to mount the volume(s) before the iSCSI initiator has started.

For example, a single volume in /etc/fstab might look like this:

/dev/sdb /data1 ext4 defaults,noatime,_netdev,nofail 0 2

If you reboot without these options in fstab, the instance will fail to start after the next reboot. Instances in this state are not recoverable.

Trusted Pool (Storage Cluster)

Enable and start glusterfsd.service on both nodes:

$ sudo systemctl enable glusterd.service

$ sudo systemctl start glusterd.service

Ports TCP:24007–24008 are required for communication between GlusterFS nodes and each brick requires another TCP port starting at 24009.

Enable required ports on the firewall:

You also need to open ports 24007–24008 on the BMCS Dashboard. You need to edit the VCN Security List and either open all ports for the Bare Metal Internal Network (NOT PUBLIC NETWORK) as shown below for



IP Protocol: All Protocols

Allows: all traffic for all ports

or only opening the required 24007–24008 ports for the internal network. Here is an example:


IP Protocol: TCP

Source Port Range: All

Destination Port Range: 24007–24008

Allows: TCP traffic for ports: 24007–24008

Now, use gluster command to connect the second GlusterFS node and create a Trusted Pool (Storage Cluster).

$ sudo gluster peer probe gfsserver1

peer probe: success

Verify cluster peer:

$ sudo gluster peer status

Number of Peers: 1

Hostname: gfsserver1

Uuid: 955326ef-fb6f-4d51–8c2a-9166b7d3b5f8

State: Peer in Cluster (Connected)

High Availability GlusterFS Volumes

GlusterFS Volume works with Gluster File System which is a logical collection of XFS bricks. The following table shows dependencies between Volume types and sizes:

NOTE: for production environments it is recommended using Distributed GlusterFS volume along with Replicated option for security reasons.

Now, open the required port on the firewall. Remember, each brick in the GlusterFS Volume requires a TCP port starting at 24009:

$ sudo firewall-cmd --zone=public --add-port=24009/tcp -\-permanent

$ sudo firewall-cmd --reload

NOTE: the same port needs to be opened on BMCS Dashboard as well in case you haven’t opened all ports for internal network.

Use the /bricks/brick1 XFS partition on both nodes to create a highly available Replicated Volume. First create a sub-directory in /bricks/brick1 mount point. It will be necessary for GlusterFS.

$ sudo mkdir /bricks/brick1/brick

Create a Distributed GlusterFS Volume running these commands on the first node only (gfsserver)

$ sudo gluster volume create glustervol1 transport tcp gfsserver:/bricks/brick1/brick gfsserver1:/bricks/brick1/brick

$ sudo gluster volume start glustervol1

Verify the GlusterFS volume

$ sudo gluster volume info all

Volume Name: glustervol1

Type: Distribute

Volume ID: c660712f-29ea-4288–96b6–2c0a0c85a82a

Status: Started

Snapshot Count: 0

Number of Bricks: 1

Transport-type: tcp


Brick1: gfsserver1:/bricks/brick1/brick

Options Reconfigured:

transport.address-family: inet

nfs.disable: on

GlusterFS Clients

GlusterFS volumes can be accessed using GlusterFS Native Client (CentOS / RedHat / OracleLinux 6.5 or later), NFS v3 (other Linux clients), or CIFS (Windows clients).

Open the Firewall for Glusterfs/NFS/CIFS Clients

NOTE: as already mentioned above the same ports need to be opened on BMCS Dashboard.

Access from another CentOS 7 machine via GlusterFS Native Client

All required packages are available by default in the CentOS 7 Base repository.

Install GlusterFS Client packages:

$ sudo yum install glusterfs glusterfs-fuse attr -y

Mount GlusterFS Volumes on the client:

$ sudo mkdir /glusterfs

$ sudo mount -t glusterfs gfsserver:/glustervol1 /glusterfs

$ sudo mount |grep -i glusterfs

gfsserver:/glustervol1 on /glusterfs type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

$ df -kh |grep glusterfs

gfsserver:/glustervol1 196G 65M 98G 1% /glusterfs

to make it permanent across restarts you need to add the following line in /etc/fstab on all GlusterFS Linux clients

gfsserver:/glustervol1 /glusterfs glusterfs defaults,_netdev,nofail 0 0

Check the GlusterFS public documentation for more details about:

  • Troubleshooting steps
  • How to to setup Windows/Linux machines access via CIFS to GlusterFS Volumes

Originally published at on June 28, 2017.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.