<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">The gfs_grow did finally complete, but now I've got another problem:<div><br></div><div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: fatal: invalid metadata block </div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: bh = 4314413922 (type: exp=5, found=4) </div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: function = gfs_get_meta_buffer </div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: file = /builddir/build/BUILD/gfs-kernel-2.6.9-75/smp/src/gfs/dio.c, line = 1223 </div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: time = 1223589349 </div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: about to withdraw from the cluster </div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: waiting for outstanding I/O </div><div>Oct 9 17:55:49 s12n03 kernel: GFS: fsid=s12:scratch13.1: telling LM to withdraw </div><div>Oct 9 17:55:50 s12n01 kernel: GFS: fsid=s12:scratch13.2: jid=1: Trying to acquire journal lock...</div><div>Oct 9 17:55:50 s12n01 kernel: GFS: fsid=s12:scratch13.2: jid=1: Busy</div><div>Oct 9 17:55:50 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Trying to acquire journal lock... </div><div>Oct 9 17:55:50 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Looking at journal... </div><div>Oct 9 17:55:51 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Acquiring the transaction lock... </div><div>Oct 9 17:55:51 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Replaying journal... </div><div>Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Replayed 1637 of 3945 blocks </div><div>Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: replays = 1637, skips = 115, sames = 2193 </div><div>Oct 9 17:55:52 s12n03 kernel: lock_dlm: withdraw abandoned memory </div><div>Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Journal replayed in 2s </div><div>Oct 9 17:55:52 s12n03 kernel: GFS: fsid=s12:scratch13.1: withdrawn </div><div>Oct 9 17:55:52 s12n02 kernel: GFS: fsid=s12:scratch13.0: jid=1: Done </div><div><div>Oct 9 17:56:26 s12n03 clurgmgrd: [6611]: <err> clusterfs:gfs-scratch13: Mount point is not accessible! </div><div>Oct 9 17:56:26 s12n03 clurgmgrd[6611]: <notice> status on clusterfs:gfs-scratch13 returned 1 (generic error) </div><div>Oct 9 17:56:26 s12n03 clurgmgrd[6611]: <notice> Stopping service scratch13 </div><div>Oct 9 17:56:26 s12n03 clurgmgrd: [6611]: <info> Removing IPv4 address 10.14.12.5 from bond0 </div><div>Oct 9 17:56:36 s12n03 clurgmgrd: [6611]: <err> /scratch13 is not a directory </div><div>Oct 9 17:56:36 s12n03 clurgmgrd[6611]: <notice> stop on nfsclient:nfs-scratch13 returned 2 (invalid argument(s)) </div><div>Oct 9 17:56:36 s12n03 clurgmgrd[6611]: <crit> #12: RG scratch13 failed to stop; intervention required </div><div>Oct 9 17:56:36 s12n03 clurgmgrd[6611]: <notice> Service scratch13 is failed </div><div><br></div></div><div>The history here is that a new storage shelf was added to the chassis. This somehow triggered an error on the chassis - a timeout of some sort, as I understand it from Site Ops - which I presume triggered this problem on this file system, since the two events were coincident. I have run gfs_fsck against this file system, but it didn't fix the problem - even when I used a newer version of gfs_fsck from RHEL 5 that had been back-ported to RHEL4. I had done this a couple of times before running the gfs_grow, and had hoped that the problem had been taken care of. Apparently not. Does anyone have any thoughts here? I can make the file system available again by killing off anything I suspect might be accessing that invalid metadata block, but that's not a good solution.</div><div><br></div><div>Thanks,</div><div><br></div><div>James</div><div><br></div><div><div><div>On Oct 9, 2008, at 11:18 AM, James Chamberlain wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Thanks Andrew.<div><br></div><div>What I'm really hoping for is anything I can do to make this gfs_grow go faster. It's been running for 19 hours now, I have no idea when it'll complete, and the file system I'm trying to grow has been all but unusable for the duration. This is a very busy file system, and I know it's best to run gfs_grow on a quiet file system, but there isn't too much I can do about that. Alternatively, if anyone knows of a signal I could send to gfs_grow that would cause it to give a status report or increase verbosity, that would be helpful, too. I have tried both increasing and decreasing the number of NFS threads, but since I can't tell where I am in the process or how quickly it's going, I have no idea what effect this has on operations.</div><div><br></div><div>Thanks,</div><div><br></div><div>James</div><div><br><div><div>On Oct 8, 2008, at 5:12 PM, Andrew A. Neuschwander wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>James,<br><br>I have a CentOS 5.2 cluster where I would see the same nfs errors under certain conditions. If I did anything that introduced latency to my gfs operations on the node that served nfs, the nfs threads couldn't service requests faster than they came in from clients. Eventually my nfs threads would all be busy and start dropping nfs requests. I kept an eye on my nfsd thread utilization (/proc/net/rpc/nfsd) and kept bumping up the number of threads until they could handle all the requests while the gfs had a higher latency.<br><br>In my case, I had EMC Networker streaming data from my gfs filesystems to a local scsi tape device on the same node that served nfs. I eventually separated them onto different nodes.<br><br>I'm sure gfs_grow would slow down your gfs enough that your nfs threads couldn't keep up. NFS on gfs seems to be very latency sensitive. I have a quick an dirty perl script to generate a historgram image from nfs thread stats if you are interested.<br><br>-Andrew<br>--<br>Andrew A. Neuschwander, RHCE<br>Linux Systems/Software Engineer<br>College of Forestry and Conservation<br>The University of Montana<br><a href="http://www.ntsg.umt.edu">http://www.ntsg.umt.edu</a><br><a href="mailto:andrew@ntsg.umt.edu">andrew@ntsg.umt.edu</a> - 406.243.6310<br><br><br>James Chamberlain wrote:<br><blockquote type="cite">Hi all,<br></blockquote><blockquote type="cite">I'd like to thank Bob Peterson for helping me solve the last problem I was seeing with my storage cluster. I've got a new one now. A couple days ago, site ops plugged in a new storage shelf and this triggered some sort of error in the storage chassis. I was able to sort that out with gfs_fsck, and have since gotten the new storage recognized by the cluster. I'd like to make use of this new storage, and it's here that we run into trouble.<br></blockquote><blockquote type="cite">lvextend completed with no trouble, so I ran gfs_grow. gfs_grow has been running for over an hour now and has not progressed past:<br></blockquote><blockquote type="cite">[root@s12n01 ~]# gfs_grow /dev/s12/scratch13<br></blockquote><blockquote type="cite">FS: Mount Point: /scratch13<br></blockquote><blockquote type="cite">FS: Device: /dev/s12/scratch13<br></blockquote><blockquote type="cite">FS: Options: rw,noatime,nodiratime<br></blockquote><blockquote type="cite">FS: Size: 4392290302<br></blockquote><blockquote type="cite">DEV: Size: 5466032128<br></blockquote><blockquote type="cite">Preparing to write new FS information...<br></blockquote><blockquote type="cite">The load average on this node has risen from its normal ~30-40 to 513 (the number of nfsd threads, plus one), and the file system has become slow-to-inaccessible on client nodes. I am seeing messages in my log files that indicate things like:<br></blockquote><blockquote type="cite">Oct 8 16:26:00 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket<br></blockquote><blockquote type="cite">Oct 8 16:26:00 s12n01 last message repeated 4 times<br></blockquote><blockquote type="cite">Oct 8 16:26:00 s12n01 kernel: nfsd: peername failed (err 107)!<br></blockquote><blockquote type="cite">Oct 8 16:26:58 s12n01 kernel: nfsd: peername failed (err 107)!<br></blockquote><blockquote type="cite">Oct 8 16:27:56 s12n01 last message repeated 2 times<br></blockquote><blockquote type="cite">Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket<br></blockquote><blockquote type="cite">Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket<br></blockquote><blockquote type="cite">Oct 8 16:27:56 s12n01 kernel: nfsd: peername failed (err 107)!<br></blockquote><blockquote type="cite">Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket<br></blockquote><blockquote type="cite">Oct 8 16:27:56 s12n01 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket<br></blockquote><blockquote type="cite">Oct 8 16:27:56 s12n01 kernel: nfsd: peername failed (err 107)!<br></blockquote><blockquote type="cite">Oct 8 16:28:34 s12n01 last message repeated 2 times<br></blockquote><blockquote type="cite">Oct 8 16:30:29 s12n01 last message repeated 2 times<br></blockquote><blockquote type="cite">I was seeing similar messages this morning, but those went away when I mounted this file system on another node in the cluster, turned on statfs_fast, and then moved the service to that node. I'm not sure what to do about it given that gfs_grow is running. Is this something anyone else has seen? Does anyone know what to do about this? Do I have any option other than to wait until gfs_grow is done? Given my recent experiences (see "lm_dlm_cancel" in the list archives), I'm very hesitant to hit ^C on this gfs_grow. I'm running CentOS 4 for x86-64, kernel 2.6.9-67.0.20.ELsmp.<br></blockquote><blockquote type="cite">Thanks,<br></blockquote><blockquote type="cite">James<br></blockquote><blockquote type="cite">-- <br></blockquote><blockquote type="cite">Linux-cluster mailing list<br></blockquote><blockquote type="cite"><a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a><br></blockquote><blockquote type="cite"><a href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a><br></blockquote><br>--<br>Linux-cluster mailing list<br><a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a><br><a href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a><br></div></blockquote></div><br></div></div>--<br>Linux-cluster mailing list<br><a href="mailto:Linux-cluster@redhat.com">Linux-cluster@redhat.com</a><br><a href="https://www.redhat.com/mailman/listinfo/linux-cluster">https://www.redhat.com/mailman/listinfo/linux-cluster</a></blockquote></div><br></div></div></body></html>