<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"> <meta name=Generator content="Microsoft Word 12 (filtered medium)"> <style>  </style>  </head> <body lang=IS link=blue vlink=purple> <div class=Section1> Hi,<o:p></o:p> <o:p> </o:p> I have a 2 node cluster running RHEL 5.1 x86_64 and fully patched as of today. <o:p></o:p> <o:p> </o:p> If i cold-boot the cluster (both nodes) everything comes up smoothly and i can migrate services between nodes etc...<o:p></o:p> However when i take one node down i am having difficultys leaving the fence domain. If i kill the fence daemon on the node i am trying to remove gracefully or use cman_tool leave force and reboot it, it comes back up, cman starts and it appears to join the cluster. The CLVMD init script hangs (just sits and hangs) and rgmanager does not start up correctly. Also CLVMD and rgmanager just sit in a zombie state and i have to poweroff or fence the node to get it to reboot....<o:p></o:p> <o:p> </o:p> The cluster never stabilizes it self until i cold boot both nodes. Then it is OK until the next reboot. I have read something about similar cases but did not find any magic solution! ;)<o:p></o:p> <o:p> </o:p> My cluster.conf is attached.<o:p></o:p> <o:p> </o:p> There is no firewall running on the machines in question (chkconfig iptables off;).<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> Various output from the the that is rebooted:<o:p></o:p> <o:p> </o:p> Output from group_tool services:<o:p></o:p> <o:p> </o:p> type level name id state<o:p></o:p> fence 0 default 00000000 JOIN_STOP_WAIT<o:p></o:p> [1 2]<o:p></o:p> dlm 1 rgmanager 00000000 JOIN_STOP_WAIT<o:p></o:p> [1 2]<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Output from group_tool fenced:<o:p></o:p> 1210193027 our_nodeid 1 our_name node-16<o:p></o:p> 1210193027 listen 4 member 5 groupd 7<o:p></o:p> 1210193029 client 3: join default<o:p></o:p> 1210193029 delay post_join 120s post_fail 0s<o:p></o:p> 1210193029 added 2 nodes from ccs<o:p></o:p> 1210193542 client 3: dump<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> Various output from the other node:<o:p></o:p> <o:p> </o:p> Output from group_tool services:<o:p></o:p> type level name id state<o:p></o:p> fence 0 default 00010002 JOIN_START_WAIT<o:p></o:p> [1 2]<o:p></o:p> dlm 1 clvmd 00020002 none<o:p></o:p> [2]<o:p></o:p> dlm 1 rgmanager 00030002 FAIL_ALL_STOPPED<o:p></o:p> [1 2]<o:p></o:p> <o:p> </o:p> Output from group_tool dump fenced:<o:p></o:p> 1210191957 our_nodeid 2 our_name node-17<o:p></o:p> 1210191957 listen 4 member 5 groupd 7<o:p></o:p> 1210191958 client 3: join default<o:p></o:p> 1210191958 delay post_join 120s post_fail 0s<o:p></o:p> 1210191958 added 2 nodes from ccs<o:p></o:p> 1210191958 setid default 65538<o:p></o:p> 1210191958 start default 1 members 2<o:p></o:p> 1210191958 do_recovery stop 0 start 1 finish 0<o:p></o:p> 1210191958 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191958 add first victim node-16<o:p></o:p> 1210191959 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191960 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191961 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191962 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191963 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191964 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191965 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191966 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191967 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191968 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191969 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191970 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191971 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191972 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191973 node "node-16" not a cman member, cn 1<o:p></o:p> 1210191974 reduce victim node-16<o:p></o:p> 1210191974 delay of 16s leaves 0 victims<o:p></o:p> 1210191974 finish default 1<o:p></o:p> 1210191974 stop default<o:p></o:p> 1210191974 start default 2 members 1 2<o:p></o:p> 1210191974 do_recovery stop 1 start 2 finish 1<o:p></o:p> 1210193633 client 3: dump<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Thanks in advanced.<o:p></o:p> <o:p> </o:p> Kær kveðja / Best Regards, Finnur Örn Guðmundsson Network Engineer - Network Operations <a href="mailto:fog@t.is">fog@t.is</a> TM Software Urðarhvarf 6, IS-203 Kópavogur, Iceland Tel: +354 545 3000 - fax +354 545 3610 <a href="http://www.tm-software.is/">www.tm-software.is</a> This e-mail message and any attachments are confidential and may be privileged. TM Software e-mail disclaimer: <a href="http://www.tm-software.is/disclaimer">www.tm-software.is/disclaimer</a> <o:p></o:p> </div> </body> </html>