[Linux-cluster] GFS/GFS2 problems with iozone

[Bob Peterson]

----- "Michael O'Sullivan" <michael.osullivan at auckland.ac.nz> wrote:
| Hi everyone,
| 
| I am having some problems testing a GFS system using iozone. I am 
| running CentOS 2.6.18-128.1.6.el5 and have a two node cluster with a
| GFS 
| installed on a shared iSCSI target. The GFS sits on top of a 1.79TB 
| clustered logical volume and can be mounted successfully on both
| cluster 
| nodes.
| 
| When using iozone to test performance everything goes smoothly until I
| 
| get to a file size of 2GB and a record length of 2048. Then iozone
| exits 
| with the error
| 
| Error fwriting block 250, fd= 7
| 
| and (as far as I can tell) the GFS becomes corrupted
| 
| fatal: invalid metadata block
| bh = 12912396 (magic)
| function = gfs_get_meta_buffer
| file =
| /builddir/build/BUILD/gfs-kmod-0.1.31/_kmod_build_/src/gfs/dio.c, 
| line = 1225
| 
| Can anyone shed some light on what is happening?
| 
| Kind regards, Mike O'S

Hi Mike,

Are you running iozone on a single node or both simultaneously?
If it's running on two nodes, please make sure that both nodes have
the iSCSI target mounted with lock_dlm protocol (not lock_nolock).
Also, we need to make sure that they're not trying to use the same
files in the file system because I think iozone is not cluster-aware.
But even so, the file system should not be corrupted unless one of
the nodes is using lock_nolock protocol, or if other boxes are
using the iSCSI target without the knowledge of GFS.

We regularly run iozone here, in single-node performance trials, and
we have never seen this kind of problem.

Also, you didn't specify what version of the kmod-gfs package you have
installed.  I've fixed at least one bug that might account for it,
depending on what version of kmod-gfs you're running.

I'm not aware of any other problems in the GFS kernel code that can
account for this kind of corruption, except for possibly this one:

https://bugzilla.redhat.com/show_bug.cgi?id=491369

(A gfs bug that really goes well beyond the nfs usage described in the bug).
You can find the patch in the attachments, although I won't guarantee
it'll solve your problem.  There's a slight chance though.
My apologies if you don't have permission to see the bug; that sometimes
happens and it's out of my control.  I can, however, post the patch
if needed.

If iozone is being run on a single node, this might be a new bug.  If you can
still recreate the problem with that patch in place, or if you don't want
to try the patch for some reason, perhaps you should open up a bugzilla
record and we'll investigate the problem.  If we can reproduce it, we'll
figure it out and fix it.

Regards,

Bob Peterson
Red Hat GFS