Btw, all members run on 2.6.9-34.ELsmp, cman-1.0.4-0 and cman-kernel-smp-2.6.9-43.8 with rgmanager-1.9.46-0.<br><br><div><span class="gmail_quote">On 6/16/06, <b class="gmail_sendername"><a href="mailto:aberoham@gmail.com">
aberoham@gmail.com</a></b> <<a href="mailto:aberoham@gmail.com">aberoham@gmail.com</a>> wrote:</span><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><br>If clustat reports rgmanager as online, why would any clusvcadm operation fail with "Try again (resource groups locked)" ?<br><br>Is there any way to recover from that rgmanger failure/error besides resetting the entire cluster?
<br><br>Details --<br><br>Yesterday evening a technician connected a Netgear GS748T switch to my network. The new switch somehow caused a storm of traffic that in turn caused a disruption of network connectivity across the entire LAN, including to all of my CS/GFS cluster nodes, for a few minutes until the new switch was removed from the network.
<br><br>This morning when I finally had a chance to investigate I found that all of the cluster members that are supposed to be online were online and that the cluster was quorate. But rgmanager would not work and services running under rgmanager were hung. (The cluster must have become inquorate and blocked access to the shared GFS volume while the outage was in progress. But some of the services and rgmanager never recovered?)
<br><br>I first tried resetting the "lead" member. (This is a pool of mirrored storage servers where the lead member creates a rsync batch off of a main fileserver and all of the other members then replay the rsync batch that is on a shared filesystem against their local filesystem mirror of the main fileserver)
<br><br>No matter what I did rgmanager would not start. cman_tool services would report code "S-1,80,4" --<br><br>root@gfs05:~<br>(0)>cman_tool services<br>Service Name GID LID State Code
<br>Fence Domain: "default" 1 2 run -<br>[2 1 4 3]<br><br>DLM Lock Space: "clvmd" 2 3 run -<br>[2 1 4 3]<br><br>User: "usrm::manager" 0 4 join S-1,80,4
<br>[]<br><br>Other cluster members would report rgmanager as online, yet when I tried to operate on member services, the operation would fail with "Try again (resource groups locked)".<br><br>root@gfs06:~<br>(1)>clustat
<br>Member Status: Quorate<br><br> Member Name Status<br> ------ ---- ------<br> gfs04 Online, rgmanager<br> gfs05 Online
<br> gfs06 Online, Local, rgmanager<br> gfs07 Online, rgmanager<br> gfs08 Offline<br><br> Service Name Owner (Last) State
<br> ------- ---- ----- ------ -----<br> mapsmirror1 gfs05 started<br> mapsmirror2 gfs06 started<br> mapsmirror3 gfs07 started
<br> mapsmirror4 gfs04 started<br> mapsmirror5 (none) stopped<br>root@gfs06:~<br>(0)>clusvcadm -d mapsmirror1<br>Member gfs06 disabling mapsmirror1...failed: Try again (resource groups locked)
<br><br>Eventually I just gave up and power cycled all cluster members at ounce. Everything, including rgmanger, then came back online OK. <br>
</div></blockquote></div><br>