Eventing Feature in Glusterfs

schaffung
schaffung
Sep 4 · 6 min read

This post is about one of the features in glusterfs known as events. As the name suggests this is related to whatever event takes place in glusterfs, that might be something like a volume creation, volume deletion, geo-rep events, peer events, so on and so forth.

The question is how is it captured? Well that is what this post is about…


How it works

Now, if you are someone who has explored or is keen on looking under the hood then you’d have noticed something called as gf_event in the glusterfs code. ( If you haven’t you can explore glusterfs in github at github/glusterfs or if you are keen to jump on and start using this feature, just go to the next section)

Image for post
Image for post
Event Flow

Now, one can consider an event as some significant change happening in glusterfs, for example consider the volume creation.

Now, when a user triggers the volume creation through the cli, the request is pushed to the glusterd and the required operations for a volume creation are done and then once it has been successfully completed, before the cli displays the success message, gf_event function is invoked with parameter being passed being the volume information which was just created.

Now, gf_event will internally invoke the _gf_event function which creates the UDP socket and pushes the message.

At the other end, we have a service called glustereventsd which has a UDP server listening for any incoming packets. Once it receives the packet, it is parsed and then finally pushed to all the web-hooks which have been registered to listen to events.

I guess we can jump into the commands now…let’s get our hands dirty with the events!


Using Events

Before moving ahead with the steps, let me clear what environment I’m using. I’m using Fedora 32 VM and the Glusterfs is of version 9dev (the master branch ).

Lets begin with the web-hook first, i.e. nothing but our simple API server.

Now I’m using a simple flask application here. One can either use this or even use the flask application present in glusterfs at events/tools/eventsdash.py.

This simple Flask program has two routes, both serving POST method.

  1. A route for “/”.
  2. A route for “/listen”.

The port being 9000 and just to show the output, I’ve turned on the debug mode.

As the flask application is ready, let us move on to the eventing part in glusterfs, which has to be turned on.

We need to start the glustereventsd service, so..

[root@server1 ~]# systemctl start glustereventsd
[root@server1 ~]# systemctl enable glustereventsd
Synchronizing state of glustereventsd.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable glustereventsd
[root@server1 ~]#

Once the event is started, we need to start the flask application and register it as a webhook.

[root@server1 home]# python3 flask_client.py 
* Serving Flask app "flask_client" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
* Running on http://0.0.0.0:9000/ (Press CTRL+C to quit)
* Restarting with stat
* Debugger is active!
* Debugger PIN: 353-160-127

The next step being, registering the webhook. For this, we’d use the command, gluster-eventsapi. Checking the status,

[root@server1 ~]# gluster-eventsapi status
Webhooks: None
+-----------+-------------+-----------------------+
| NODE | NODE STATUS | GLUSTEREVENTSD STATUS |
+-----------+-------------+-----------------------+
| localhost | UP | OK |
+-----------+-------------+-----------------------+
[root@server1 ~]#

Before registering, let’s check if we are able to communicate with the flask application,

[root@server1 ~]# gluster-eventsapi webhook-test http://0.0.0.0:9000/
+-----------+-------------+----------------+
| NODE | NODE STATUS | WEBHOOK STATUS |
+-----------+-------------+----------------+
| localhost | UP | OK |
+-----------+-------------+----------------+
[root@server1 ~]#

So, the node as well as the webhook is up. On checking the flask output,

<Request 'http://0.0.0.0:9000/' [POST]>
127.0.0.1 - - [04/Sep/2020 09:19:02] "POST / HTTP/1.1" 200 -

the next step is to register the webhook,

[root@server1 ~]# gluster-eventsapi webhook-add http://0.0.0.0:9000/listen
+-----------+-------------+-------------+
| NODE | NODE STATUS | SYNC STATUS |
+-----------+-------------+-------------+
| localhost | UP | OK |
+-----------+-------------+-------------+
[root@server1 ~]# gluster-eventsapi status
Webhooks:
http://0.0.0.0:9000/listen
+-----------+-------------+-----------------------+
| NODE | NODE STATUS | GLUSTEREVENTSD STATUS |
+-----------+-------------+-----------------------+
| localhost | UP | OK |
+-----------+-------------+-----------------------+
[root@server1 ~]#

As we can see, the webhook is registered and events in glusterfs would be then communicated to the webhook.

Let’s create a gluster volume. We can then observe the event being communicated to the webhook. On creating a volume,

[root@server1 /]# gluster vol create vol1 replica 3 server1:/data/brick/br1 server1:/data/brick/br2 server1:/data/brick/br3 force
volume create: vol1: success: please start the volume to access data
[root@server1 /]#

We get the below response in the webhook,

{'event': 'VOLUME_CREATE', 'message': {'bricks': ' server1:/data/brick/br1 server1:/data/brick/br2 server1:/data/brick/br3', 'name': 'vol1'}, 'nodeid': 'c61dc1be-14b1-45b1-8a5a-9f401f6dcb22', 'ts': 1599193580}
127.0.0.1 - - [04/Sep/2020 09:56:20] "POST /listen HTTP/1.1" 200 -

On deleting the volume,

[root@server1 /]# gluster vol delete vol1
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: vol1: success
[root@server1 /]#

we see the response,

{'event': 'VOLUME_DELETE', 'message': {'name': 'vol1'}, 'nodeid': 'c61dc1be-14b1-45b1-8a5a-9f401f6dcb22', 'ts': 1599193693}
127.0.0.1 - - [04/Sep/2020 09:58:13] "POST /listen HTTP/1.1" 200 -

Let’s look a little deeper into the contents of the response JSON,

{
'event' : 'VOLUME_DELETE',
'message' : {'name' : 'vol1'},
'nodeid' : 'c61dc1be-14b1-45b1-8a5a-9f401f6dcb22',
'ts' : 1599193693
}

Hence, the events response has four components inside it, the event, the message corresponding to the event, the node id and the time stamp. Using these information, one can actually create a webhook which can then go on to do some log analysis or event predictive analysis on the health of the glusterfs cluster in use.

Apart from these commands, to see the configuration of the eventing system, i.e. the Logging mode as well as the UDP port being used to push the messages to the Glustereventsd, one can use the config-get command,

[root@server1 glusterfs]# gluster-eventsapi config-get
+--------------------+-------+
| NAME | VALUE |
+--------------------+-------+
| log-level | INFO |
| port | 24009 |
| disable-events-log | False |
+--------------------+-------+
[root@server1 glusterfs]#

To set the configuration, the command is config-set. For example, let’s set the log-level to DEBUG,

[root@server1 glusterfs]# gluster-eventsapi config-set log-level DEBUG
+-----------+-------------+-------------+
| NODE | NODE STATUS | SYNC STATUS |
+-----------+-------------+-------------+
| localhost | UP | OK |
+-----------+-------------+-------------+
[root@server1 glusterfs]# gluster-eventsapi config-get
+--------------------+-------+
| NAME | VALUE |
+--------------------+-------+
| log-level | DEBUG |
| port | 24009 |
| disable-events-log | False |
+--------------------+-------+
[root@server1 glusterfs]#

Also, suppose, one wants to remove the webhook in use, they can use the command,

[root@server1 glusterfs]# gluster-eventsapi webhook-del http://localhost:9000/listen
+-----------+-------------+-------------+
| NODE | NODE STATUS | SYNC STATUS |
+-----------+-------------+-------------+
| localhost | UP | OK |
+-----------+-------------+-------------+
[root@server1 glusterfs]# gluster-eventsapi status
Webhooks: None
+-----------+-------------+-----------------------+
| NODE | NODE STATUS | GLUSTEREVENTSD STATUS |
+-----------+-------------+-----------------------+
| localhost | UP | OK |
+-----------+-------------+-----------------------+
[root@server1 glusterfs]#

For getting more information, one can use th ehelp command to see what all subcommands does gluster-eventsapi support.

Coming to the logging part, one can find the logging being done in /var/log/glusterfs/events.log.


This was a basic introduction as to what Events in Glusterfs mean and how to use it. It is an interesting feature if one considers the scenario, wherein the logs can be used for further analysis and prediction.

A food for thought…

Dev Genius

Coding, Tutorials, News, UX, UI and much more related to development

Sign up for Best Stories

By Dev Genius

The best stories sent monthly to your email. Take a look

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.

Check your inbox
Medium sent you an email at to complete your subscription.

schaffung

Written by

schaffung

Recursive…

Dev Genius

Coding, Tutorials, News, UX, UI and much more related to development

schaffung

Written by

schaffung

Recursive…

Dev Genius

Coding, Tutorials, News, UX, UI and much more related to development