GlusterFS quota: fix accounting

Quota Xattrs:

The values used for calculating are a part of the xattr of the directory.

To get the xattr the following command can be used:

getfattr -d -m. -e hex <path on brick>

EG:

trusted.glusterfs.quota.15bd14e3–9abe-49fc-91c4–63492cd12dc9.contri.1=0x0000000000a000000000000000000001000000000000002c

trusted.glusterfs.quota.dirty=0x3000

trusted.glusterfs.quota.size.1=0x0000000000a000000000000000000001000000000000002c

The size xattr of the directory in every brick is used aggregated.

Basically, to fix it we need to find which directory has a mismatch and on that directory, we need to set the dirty flag (the xattr mentioned above) and then do a lookup on that directory.

Note: the above xattrs are for directories. For files with regard to quota, only contri will be available. Size is got using a stat on the file and the dirty is usually set for directories.

Ways to find the mismatch in directories and fix are:

First approach:

This is the older one still being used. It’s a lot of work. This has to be done on every brick.

1) Use the following command and get the output file (log_gluster_xattr)

# find <brick path> | xargs getfattr -d -m. -e hex > log_gluster_xattr

2) Get this output file for that particular brick and send it to this script file in the extras/quota/xattr_analysis.py ( available in the source code).

Usage:

# python ./xattr_analysis.py log_gluster_xattr

This will show the directories with contri or size mismatch for that particular brick.

NOTE: we use the hurry filesize package. so we might have to install it.

pip install hurry.filesize

3) Repeat the same process for the other bricks.

4) Once you get the list of all the directories with the size or contri mismatch.

Set the dirty flag on the directories with a mismatch on the respective bricks.

To set dirty:

# setfattr -n trusted.glusterfs.quota.dirty -v 0x3100 <path to the directory>

5) After setting dirty on all these directories in the brick, do a lookup on the topmost directory from the client.

To do a lookup:

# du <path to directory on the mount point>

Second approach:

As the above approach is a lot of steps, we came up with this method. It has lesser steps.

1) run this script in extras/quota/quota_fsck.py with the following command on each brick

# python quota_fsck.py $brickspath > $tmpfilenameforbrickx

The above tmpfilenameforbrickx file has the directories that have a mismatch.

Repeating it on every brick will give all the directories where dirty have to be set and then lookup has to be performed (steps 4 and 5 of the xattr_analysis approach).

If the above command is run with the option — fix-issues then it will make changes to the backend directly and fix the xattrs.

Eg:

 # quota_fsck.py --fix-issues brick_path

If you want to do it from a certain subdir then you can use the command with the option

--sub-dir

Eg:

 # quota_fsck.py — sub-dir <sub directory> brick_path

After performing all this, we have to wait for a while. The update takes a while to be complete. This method needs more testing to be done. But we need not be worried about using the fix xattr option, as it sets the quota xattrs alone. which can be fix if anything goes wrong with a simple quota disable and enable.

Note: though a quota disable and enable will fix accounting issues, it is a resource-consuming process. It does a crawl of the whole file system. This crawl will take a lot of time and CPU based on the size of the filesystem and other such factors. That's the reason behind going for such methods.

Obsolete approach:

This involves using both the mount and the server. It collects the directory layout from Glusterfs mount on the client-side and then du done from the backend filesystem on the bricks. The following script works from the mount point. So on the client run:

extras/quota/log_accounting.sh <mountpoint> <volname>

This will give us a tar file that will help us identify the layout of the filesystem and mismatch with the files being available or not between the bricks and the client. Unless there aren’t any such operations we won't need this. Then on all the bricks run(same as the xattr_analysis method):

# find /brick/path | xargs getfattr -d -m. -e hex > /tmp/log_gluster_xattr

The log_gluster_xattr is then used by the xattr_analysis.py to find the mismatch. The file can be checked for mismatch by issuing the command:

# python xattr_analysis log_gluster_xattr > /tmp/result

On checking the /tmp/result file we will see the last line showing the directories with mismatch

(‘14G’, ‘20G’, ‘# file: bricks/gfs/admin’)
(‘215M’, ‘429M’, ‘# file: bricks/gfs/admin/scripts’)
(‘30K’, ‘60K’, ‘# file: bricks/gfs/admin/IBshrink’)

We can manually check the values by grepping for that directory in the /tmp/results. It should be having a mismatch in at least one of the bricks. An example of the mismatch is provided below:

=====# file: bricks/gfs/admin=======
        …
        trusted.glusterfs.dht=0x000000010000000000000000ffffffff
        trusted.glusterfs.quota.380f969f-4c7b-4625-a178-97f3b7fa1d7d.contri.1=0x000000000d75b20000000000000002f40000000000000019
        trusted.glusterfs.quota.dirty=0x3000
        trusted.glusterfs.quota.size.1=0x000000000d75b20000000000000002f40000000000000019
       {'contri_size': '215M', 'contri_file_count': 756, 'file_count': 756, 'dirty': False, 'contri_dir_count': 25, 'dir_count': 25, 'size': '215M'}

We can see that the size and contri don't match. This confirms that these are the ones that contributed to the mismatch in the quota list.

With the files having a mismatch, we set dirty flag to it and then do a lookup from the mount, this is similar to the xattr_analysis approach.

These are the ways we fix accounting issues in Glusterfs quota.