[Linux-cluster] mod_log_sql killing GFS mount?

[isplist at logicore.net]

Anyone?

All of the nodes are updated to the same software versions, why would this 
occurring now?


On Sun, 25 Nov 2007 16:45:43 -0600, isplist at logicore.net wrote:
> Some new information about this.
> 
> From the console;
> 
> Nov 25 16:43:11 compdev kernel: GFS: fsid=vgcomp:web.3: Found quota changes
> for 0 IDs
> Nov 25 16:43:11 compdev kernel: GFS: fsid=vgcomp:web.3: Done
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3: fatal: filesystem
> consistency error
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3:   RG = 31104599
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3:   function =
> gfs_setbit
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3:   file =
> /home/xos/gen/updates-2007-10/xlrpm922/rpm/BUILD/gfs-kernel-2.6.9-
> 72/up/src/gf
> s/bits.c, line = 71
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3:   time = 1196030598
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3: about to withdraw
> from
> the cluster
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3: waiting for
> outstanding I/O
> Nov 25 16:43:18 compdev kernel: GFS: fsid=vgcomp:web.3: telling LM to
> withdraw
> Nov 25 16:43:20 compdev kernel: lock_dlm: withdraw abandoned memory
> Nov 25 16:43:20 compdev kernel: GFS: fsid=vgcomp:web.3: withdrawn
> 
> 
>> From the console;
>> 
>> Nov 25 14:27:11 compdev kernel: GFS: Trying to join cluster "lock_dlm",
>> "vgcomp:web"
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: Joined cluster.
>> Now
>> mounting FS...
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: jid=3: Trying to
>> acquire journal lock...
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: jid=3: Looking at
>> journal...
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: jid=3: Done
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: Scanning for log
>> elements...
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: Found 1 unlinked
>> inodes
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: Found quota
>> changes
>> for 0 IDs
>> Nov 25 14:27:14 compdev kernel: GFS: fsid=vgcomp:web.3: Done
>> 
>> Within seconds of mounting the /Vol_web storage, I get an I/O error;
>> 
>> # df
>> Filesystem           1K-blocks      Used Available Use% Mounted on
>> /dev/hda1             37689468   2665908  33109016   8% /
>> none                    257784         0    257784   0% /dev/shm
>> /dev/mapper/VolGroup02-img
>> 505788640      5552 505783088   1% /var/images
>> df: `/Vol_web': Input/output error
>> 
>> I recently installed mod_log_sql on our web servers for central logging.
>> What's strange is that since I've done this, one of my nodes keeps losing
>> one
>> of it's GFS mounts. Web servers have the exact same mount yet aren't
>> losing
>> their mount.
>> 
>> There is no information in /var/log/ messages or any other log file
>> showing
>> what might be going on.
>> 
>> Any thoughts on what I might be looking for?
>> 
>> Mike
> 
> 
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster