[Linux-cluster] gfs_grow
James Chamberlain
jamesc at exa.com
Thu Oct 9 22:53:58 UTC 2008
The gfs_grow did finally complete, but now I've got another problem:
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: fatal:
invalid metadata block
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: bh =
4314413922 (type: exp=5, found=4)
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: function =
gfs_get_meta_buffer
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: file = /
builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/gfs/dio.c, line = 1223
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: time =
1223589349
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: about to
withdraw from the cluster
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: waiting for
outstanding I/O
Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: telling LM
to withdraw
Oct 9 17:55:50 s12n01 kernel: GFS: fsid=s12:scratch13.2: jid=1:
Trying to acquire journal lock...
Oct 9 17:55:50 s12n01 kernel: GFS: fsid=s12:scratch13.2: jid=1: Busy
Oct 9 17:55:50 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1:
Trying to acquire journal lock...
Oct 9 17:55:50 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1:
Looking at journal...
Oct 9 17:55:51 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1:
Acquiring the transaction lock...
Oct 9 17:55:51 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1:
Replaying journal...
Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1:
Replayed 1637 of 3945 blocks
Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1:
replays = 1637, skips = 115, sames = 2193
Oct 9 17:55:52 s12n03 kernel: lock_dlm: withdraw abandoned memory
Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1:
Journal replayed in 2s
Oct 9 17:55:52 s12n03 kernel: GFS: fsid=s12:scratch13.1: withdrawn
Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Done
Oct 9 17:56:26 s12n03 clurgmgrd: [6611]: <err> clusterfs:gfs-
scratch13: Mount point is not accessible!
Oct 9 17:56:26 s12n03 clurgmgrd[6611]: <notice> status on
clusterfs:gfs-scratch13 returned 1 (generic error)
Oct 9 17:56:26 s12n03 clurgmgrd[6611]: <notice> Stopping service
scratch13
Oct 9 17:56:26 s12n03 clurgmgrd: [6611]: <info> Removing IPv4 address
10.14.12.5 from bond0
Oct 9 17:56:36 s12n03 clurgmgrd: [6611]: <err> /scratch13 is not a
directory
Oct 9 17:56:36 s12n03 clurgmgrd[6611]: <notice> stop on nfsclient:nfs-
scratch13 returned 2 (invalid argument(s))
Oct 9 17:56:36 s12n03 clurgmgrd[6611]: <crit> #12: RG scratch13
failed to stop; intervention required
Oct 9 17:56:36 s12n03 clurgmgrd[6611]: <notice> Service scratch13 is
failed
The history here is that a new storage shelf was added to the
chassis. This somehow triggered an error on the chassis - a timeout
of some sort, as I understand it from Site Ops - which I presume
triggered this problem on this file system, since the two events were
coincident. I have run gfs_fsck against this file system, but it
didn't fix the problem - even when I used a newer version of gfs_fsck
from RHEL 5 that had been back-ported to RHEL4. I had done this a
couple of times before running the gfs_grow, and had hoped that the
problem had been taken care of. Apparently not. Does anyone have any
thoughts here? I can make the file system available again by killing
off anything I suspect might be accessing that invalid metadata block,
but that's not a good solution.
Thanks,
James
On Oct 9, 2008, at 11:18 AM, James Chamberlain wrote:
> Thanks Andrew.
>
> What I'm really hoping for is anything I can do to make this
> gfs_grow go faster. It's been running for 19 hours now, I have no
> idea when it'll complete, and the file system I'm trying to grow has
> been all but unusable for the duration. This is a very busy file
> system, and I know it's best to run gfs_grow on a quiet file system,
> but there isn't too much I can do about that. Alternatively, if
> anyone knows of a signal I could send to gfs_grow that would cause
> it to give a status report or increase verbosity, that would be
> helpful, too. I have tried both increasing and decreasing the
> number of NFS threads, but since I can't tell where I am in the
> process or how quickly it's going, I have no idea what effect this
> has on operations.
>
> Thanks,
>
> James
>
> On Oct 8, 2008, at 5:12 PM, Andrew A. Neuschwander wrote:
>
>> James,
>>
>> I have a CentOS 5.2 cluster where I would see the same nfs errors
>> under certain conditions. If I did anything that introduced latency
>> to my gfs operations on the node that served nfs, the nfs threads
>> couldn't service requests faster than they came in from clients.
>> Eventually my nfs threads would all be busy and start dropping nfs
>> requests. I kept an eye on my nfsd thread utilization (/proc/net/
>> rpc/nfsd) and kept bumping up the number of threads until they
>> could handle all the requests while the gfs had a higher latency.
>>
>> In my case, I had EMC Networker streaming data from my gfs
>> filesystems to a local scsi tape device on the same node that
>> served nfs. I eventually separated them onto different nodes.
>>
>> I'm sure gfs_grow would slow down your gfs enough that your nfs
>> threads couldn't keep up. NFS on gfs seems to be very latency
>> sensitive. I have a quick an dirty perl script to generate a
>> historgram image from nfs thread stats if you are interested.
>>
>> -Andrew
>> --
>> Andrew A. Neuschwander, RHCE
>> Linux Systems/Software Engineer
>> College of Forestry and Conservation
>> The University of Montana
>> http://www.ntsg.umt.edu
>> andrew at ntsg.umt.edu - 406.243.6310
>>
>>
>> James Chamberlain wrote:
>>> Hi all,
>>> I'd like to thank Bob Peterson for helping me solve the last
>>> problem I was seeing with my storage cluster. I've got a new one
>>> now. A couple days ago, site ops plugged in a new storage shelf
>>> and this triggered some sort of error in the storage chassis. I
>>> was able to sort that out with gfs_fsck, and have since gotten the
>>> new storage recognized by the cluster. I'd like to make use of
>>> this new storage, and it's here that we run into trouble.
>>> lvextend completed with no trouble, so I ran gfs_grow. gfs_grow
>>> has been running for over an hour now and has not progressed past:
>>> [root at s12n01 ~]# gfs_grow /dev/s12/scratch13
>>> FS: Mount Point: /scratch13
>>> FS: Device: /dev/s12/scratch13
>>> FS: Options: rw,noatime,nodiratime
>>> FS: Size: 4392290302
>>> DEV: Size: 5466032128
>>> Preparing to write new FS information...
>>> The load average on this node has risen from its normal ~30-40 to
>>> 513 (the number of nfsd threads, plus one), and the file system
>>> has become slow-to-inaccessible on client nodes. I am seeing
>>> messages in my log files that indicate things like:
>>> Oct 8 16:26:00 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104
>>> when sending 140 bytes - shutting down socket
>>> Oct 8 16:26:00 s12n01 last message repeated 4 times
>>> Oct 8 16:26:00 s12n01 kernel: nfsd: peername failed (err 107)!
>>> Oct 8 16:26:58 s12n01 kernel: nfsd: peername failed (err 107)!
>>> Oct 8 16:27:56 s12n01 last message repeated 2 times
>>> Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104
>>> when sending 140 bytes - shutting down socket
>>> Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104
>>> when sending 140 bytes - shutting down socket
>>> Oct 8 16:27:56 s12n01 kernel: nfsd: peername failed (err 107)!
>>> Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104
>>> when sending 140 bytes - shutting down socket
>>> Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104
>>> when sending 140 bytes - shutting down socket
>>> Oct 8 16:27:56 s12n01 kernel: nfsd: peername failed (err 107)!
>>> Oct 8 16:28:34 s12n01 last message repeated 2 times
>>> Oct 8 16:30:29 s12n01 last message repeated 2 times
>>> I was seeing similar messages this morning, but those went away
>>> when I mounted this file system on another node in the cluster,
>>> turned on statfs_fast, and then moved the service to that node.
>>> I'm not sure what to do about it given that gfs_grow is running.
>>> Is this something anyone else has seen? Does anyone know what to
>>> do about this? Do I have any option other than to wait until
>>> gfs_grow is done? Given my recent experiences (see
>>> "lm_dlm_cancel" in the list archives), I'm very hesitant to hit ^C
>>> on this gfs_grow. I'm running CentOS 4 for x86-64, kernel
>>> 2.6.9-67.0.20.ELsmp.
>>> Thanks,
>>> James
>>> --
>>> Linux-cluster mailing list
>>> Linux-cluster at redhat.com
>>> https://www.redhat.com/mailman/listinfo/linux-cluster
>>
>> --
>> Linux-cluster mailing list
>> Linux-cluster at redhat.com
>> https://www.redhat.com/mailman/listinfo/linux-cluster
>
> --
> Linux-cluster mailing list
> Linux-cluster at redhat.com
> https://www.redhat.com/mailman/listinfo/linux-cluster
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20081009/b66b9f8f/attachment.htm>
More information about the Linux-cluster
mailing list