Anyone Having Trouble Setting Up Gluster Geo Replication?

Photo by James Rush

You only need it to work once, or whatever you say to yourself when your on your millionth try. I searched the web relentlessly looking to solve what started out a small hiccup of speedy long distance file replication and turned into a long distance exhaustive task of setting up what was meant to be a little geo-replication with Gluster.

I looked at every post, tutorial, error and bug report. I was on the cusp of being convinced it was a iceberg sized bug and I didn’t exactly fancy all my hours spent becoming the Titanic. So I dove in and thousands of slightly different sequences of commands later. I finally reached a working set up. So, to anyone having trouble setting up Gluster Geo Replication on Ubuntu!?!? Put on “Trouble” by Iggy and “I Knew You Were Trouble” by Taylor and I’ll get you through.

Let’s start with the versions. This set up is on Ubuntu 16.04, running GlusterFS-Server 3.11.3, Rsync 3.1.1. This is on a fresh install of Ubuntu 16.04 any problem encountered with the below guide are likely from other versions or applications installed.

IF YOU ARE NOT SURE, EVERYTHING BELOW IN CAPITALS IS AN 100% INSTRUCTION FOR YOU AND NOT A COMMAND FOR THE SERVER.

SERVER-ONE WILL BE OUR MASTER

FOR THE BELOW, SERVER-ONE will have hostname gluster0 and SERVER-TWO will have hostname gluster1. A quick addition to the /etc/hosts file on each server will add these hostnames. Edit the hosts file to add the below.

SERVER-ONE-IP gluster0
SERVER-TWO-IP gluster1

THE GLUSTER INSTALLATION — RUN ON BOTH SERVERS

sudo apt-get install python-software-properties
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:gluster/glusterfs-3.11
sudo apt-get update
sudo apt-cache madison glusterfs-server
CHECK THAT glusterfs-3.11 IS LISTED IN OUTPUT OF ABOVE
IF (THERE IS A VERSION HIGHER THAN 3.11) {
 sudo apt-get install glusterfs-server=glusterfs-3.11
} ELSE {
 sudo apt-get install glusterfs-server
}

WHEN THE IP MET THE SSH KEY

Lets set up the SSH Keys run these on the respective servers
ON SERVER-ONE ssh-keygen -R SERVER-ONE-IP
ON SERVER-TWO ssh-keygen -R SERVER-TWO-IP
You’ll be asked some questions:
1. STORE IN: /root/.ssh/id_rsa
2. PASSPHRASE: Leave blank and hit enter and again on confirm passphrase.
Now, ago to /root/.ssh on both servers. Open both sets of id_rsa.pub and authorized_keys. Copy the contents of id_rsa.pub and place it in the other servers authorized_keys file and also in the same authorized_keys. So each server has its own SSH key and the one for the other server plus any pre-existing keys.
Next, run these commands on the opposite server in the commands.
1. ssh root@SERVER-ONE-IP
2. ssh gluster0
3. rsync -avP /root/.ssh/ gluster0:/root/.ssh/
AND again on the other server: 
1. ssh root@SERVER-TWO-IP
2. ssh gluster1
3. rsync -avP /root/.ssh/ gluster1:/root/.ssh/
Now we will add the servers to make a cluster
RUN ON SERVER ONE sudo gluster peer probe gluster0
RUN ON SERVER TWO sudo gluster peer probe gluster1
RUN ON BOTH sudo gluster peer status
If the above command, sudo gluster peer status, reads connected you can continue. Otherwise review the commands above and try again.
RUN ON MASTER gluster system:: execute gsec_create
ON MASTER IN FTP BROWSER (I LIKE FILEZILLA) NAVAGATE TO /var/lib/glusterd/geo-replication 
MOVE FILE common_secret.pem.pub TO YOUR LOCAL MACHINE 
THEN PLACE THAT FILE IN /var/lib/glusterd/geo-replication ON YOUR OTHER SERVER
RUN sudo mkdir /var/storage ON BOTH SERVERS

MOVING SWIFTLY ALONG

LETS CREATE OUR NEW VOLUMES
ON SERVER-ONE RUN
sudo gluster volume create NAME-FOR-VOL-1 gluster0:/var/storage/vol-1 force
ON SERVER-TWO
sudo gluster volume create NAME-FOR-VOL-2 gluster1:/var/storage/vol-2 force
ON BOTH SERVERS RUN gluster volume info CHECK THAT BOTH VOLUMES ARE LISTED ON BOTH SERVERS BEFORE CONTINUING
NOW CREATE THE GEO-REP ON SERVER-ONE RUN gluster volume geo-replication NAME_FOR_VOL_1 gluster1::NAME_FOR_VOL_2 create
ON BOTH SERVERS NAVAGATE TO /var/lib/glusterd/geo-replication AND GO TO THE NEW DIRECTORY NAMED “NAME-FOR-VOL-1_gluster1_NAME-FOR-VOL-2”
ON BOTH SERVERS OPEN gsyncd.conf AND MAKE THE BELOW CHANGES — EACH ADD FOLLOWING A REMOVE IS REPLACE THE REMOVED LINE.

### EDIT BEGIN
REMOVE /nonexistent/gsyncd
ADD /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd
REMOVE ssh_command = ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem 
ADD ssh_command = ssh
REMOVE ssh_command_tar = ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/tar_ssh.pem
ADD ssh_command_tar = ssh

ALSO IF NOT PRESENT ADD access_mount = true UNDER [peers gluster0 gluster1]

### EDIT END

THE FUN PART

ON BOTH SERVERS systemctl enable glusterd 
ON MASTER gluster volume geo-replication NAME_FOR_VOL_1 gluster1:: NAME_FOR_VOL_2 config access_mount true
ON MASTER gluster volume geo-replication NAME_FOR_VOL_1 gluster1:: NAME_FOR_VOL_2 start
WAIT A FEW SECONDS
ON MASTER gluster volume geo-replication NAME_FOR_VOL_1 gluster1:: NAME_FOR_VOL_2 status
IF (CRAWL STATUS IS DIFFERENT) {
 WAIT AND RUN THE STATUS COMMAND AGAIN UNTIL IT DOES
} ELSE IF (STATUS DOES NOT READ Active) {
 TRY AND FIND THE ERROR AND/OR RESTART THIS GUIDE
} ELSE IF (IF STATUS READS Active AND CRAWL STATUS READS Changelog Crawl) {
 CONTINUE
}

THE MOUNTING OF THE SERVERS

ON ANOTHER SERVER OR YOUR CURRENT ONE (BELOW NAMED MOUNT_SERVER_1) MOUNT THE MASTER GLUSTER VOLUME AND ON THE OTHER INTENDED SERVER (BELOW NAMED MOUNT_SERVER_2) MOUNT THE OTEHR GLUSTER VOLUME LIKE SO.

ON MOUNT_SERVER_1 RUN COMMAND sudo mount -t glusterfs gluster0:NAME_FOR_VOL_1 PATH-WHERE-TO-MOUNT

ON MOUNT_SERVER_2 RUN COMMAND sudo mount -t glusterfs gluster1:NAME_FOR_VOL_2 PATH-WHERE-TO-MOUNT

WRAPPING UP

Any files won’t replicate on both volumes unless transferred or created on the MASTER volume and any directly replicating at its location and only from the directory where it is mounted on the MOUNT-SERVERS.
Files will only replicate one way with geo-replication from the MASTER to the other sever/s and only from a server where the volume is mounted — directly adding files to the directory on either Gluster server won’t replicate those files. The reason for geo-replication is to provide identical files from clusters or server separated by long distances, this is as normal Gluster replication isn’t meant for long distances and becomes slow, geo-replication solves this slow read problem. To write from the other servers mount NAME_FOR_VOL_1 in a different directory on your second mount server and write to that directory.

 I hope all this helps you with your Glustering.

THATS ALL FOLKS

Like what you read? Give James Rush a round of applause.

From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.