Sunday, April 10, 2011

Gluster 3.0.x to 3.1.x migration

Migrating from GlusterFS 3.0.x to 3.1.x is explained in http://bit.ly/ibgF6K, however this migration process leaves room for errors. More precaution is necessary while migrating between these major versions.

I have listed a few steps which have to be followed during the 3.0.x to 3.1.x migration. And the error one might encounter due to faulty migration and the steps to overcome them.

One of the new things that came in 3.1 is a concept called gfid, gfid is a extended attribute that gets set to every file and directory on a GlusterFS file system. So, essentially after migrating to 3.1 every file that is accessed from the mount point hence after is assigned this new extended attribute.

As a first step after migrating from 3.0.x to 3.1.x is to mount the cluster with a `single' client and run stat on the mount point recursively for e.g `ls -lR' >/dev/null. Double check if other clients are accessing the cluster, and shut them down.

After upgrade, if more than one client accesses the cluster, there is a possibility that directories on the backends might end up with different gfids, I have illustrated this in the below example. In such cases directory or file removal fails and you might see some unexpected behaviors. Below is a error due to gfid mismatch...

root@odin:/mnt/distribute1# rm -rf glusterfs-3.*
rm: cannot remove `glusterfs-3.0.5/extras/volgen': Directory not empty


The fix for this is to recognize such directories and remove the extended attribute trusted.gfid on the backend and run stat from the mount point. Make sure no other clients are accessing these directories at the same time.

A illustration of how it looks:

root@odin:/mnt/distribute1# rm -rf glusterfs-3.2.0qa8/
rm: cannot remove `glusterfs-3.2.0qa8/: Directory not empty
root@odin:/mnt/distribute1#

Examining the backend I see libglusterfs is a directory within glusterfs-3.2.0qa8/ examining further you see...

root@odin:/media# find /media/ -type d -name 'libglusterfs' | \
xargs -d'\n' getfattr -d -m trusted.gfid -e hex

getfattr: Removing leading '/' from absolute path names
# file: media/5/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0x9c3986db772d413a97ba79549b57370f

# file: media/4/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0x8ae60902d0894c7ea52ad1061ee1e158

# file: media/1/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0x8ae60902d0894c7ea52ad1061ee1e158

# file: media/3/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0x8ae60902d0894c7ea52ad1061ee1e158

# file: media/2/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0x9c3986db772d413a97ba79549b57370f


Notice that the gfids are not same on all the backends, which is a requirement.

Solution:

On the backend remove the extended attribute trusted.gfid for the problem directory:

root@odin:/media# find /media/ -type d -name 'libglusterfs' | \
xargs -d'\n' setfattr -x trusted.gfid
root@odin:/media# find /media/ -type d -name 'libglusterfs' | \
xargs -d'\n' getfattr -d -m trusted.gfid -e hex
root@odin:/media#

No attributes all right. Now run a stat on the mount point to fix the gfid.

root@odin:/mnt/distribute1# stat glusterfs-3.2.0qa8/libglusterfs

File: `glusterfs-3.2.0qa8/libglusterfs'
Size: 20480 Blocks: 80 IO Block: 131072 directory
Device: 16h/22d Inode: 39460 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2011-04-11 17:25:52.000000000 +0530
Modify: 2011-04-11 16:50:33.000000000 +0530
Change: 2011-04-11 17:27:00.000000000 +0530
root@odin:/mnt/distribute1#

On the backend the directory should now have same gfid for the directory on all the nodes.

root@odin:/media# find /media/ -type d -name 'libglusterfs' | \

xargs -d'\n' getfattr -d -m trusted.gfid -e hex
getfattr: Removing leading '/' from absolute path names
# file: media/5/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0xcfeeacae15b54738b8fc6d60bd1ff05c

# file: media/4/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0xcfeeacae15b54738b8fc6d60bd1ff05c

# file: media/1/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0xcfeeacae15b54738b8fc6d60bd1ff05c

# file: media/3/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0xcfeeacae15b54738b8fc6d60bd1ff05c

# file: media/2/glusterfs-3.2.0qa8/libglusterfs
trusted.gfid=0xcfeeacae15b54738b8fc6d60bd1ff05c

root@odin:/mnt/distribute1# rm -rf glusterfs-3.2.0qa8/
root@odin:/mnt/distribute1#

The the directory can be removed from the mount point provided we have fixed the layouts of the directory and its sub-directories.

This problem was first spotted on one of our customers and thanks to Avati for extensive debugging and figuring out the root cause and the solution.