<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv=Content-Type content="text/html; charset=iso-8859-7"> <meta name=Generator content="Microsoft Word 11 (filtered medium)">  <title>Problem with 2 node cluster - node 2 not starting services</title> <o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"/>  <style>  </style> </head> <body lang=EN-US link=blue vlink=purple> <div class=Section1> Hello All again,<o:p></o:p> <o:p> </o:p> In continuation of my previous e-mail following is the point were I located the problem.<o:p></o:p> <o:p> </o:p> On both nodes I have the default RHEL5.1 /etc/init.d/clvmd script.<o:p></o:p> <o:p> </o:p> Node tweety-1, after starting cman, and after starting rgmanager it succeeds to start the services:<o:p></o:p> <o:p> </o:p> Mar 24 16:46:57 tweety1 clurgmgrd: [10760]: <err> script:CLVMD: stop of /etc/init.d/clvmd failed (returned 143)<o:p></o:p> Mar 24 16:46:57 tweety1 clurgmgrd[10760]: <notice> stop on script "CLVMD" returned 1 (generic error)<o:p></o:p> Mar 24 16:46:57 tweety1 clurgmgrd[10760]: <info> Services Initialized<o:p></o:p> Mar 24 16:46:57 tweety1 clurgmgrd[10760]: <info> State change: Local UP<o:p></o:p> Mar 24 16:47:02 tweety1 clurgmgrd[10760]: <notice> Starting stopped service service:GFS2-t1<o:p></o:p> Mar 24 16:47:02 tweety1 clurgmgrd[10760]: <notice> Starting stopped service service:BOINC-t1<o:p></o:p> …………………..<o:p></o:p> <o:p> </o:p> So what I did with tweety-2 (the one that hangs) was to remove from auto start both cman and rgmanager and after complete boot up of tweety-2, manually start the services (“service start cman” and “service start rgmanager”). This helped me to compare against tweety-1 loggs on the same scripts / services.<o:p></o:p> <o:p> </o:p> So I found out that for some reason I do not understand, on tweety-2, cman starts correctly but rgmanager hangs FOREVER at the point were tweety-1 moves on:<o:p></o:p> <o:p> </o:p> Mar 24 20:02:16 localhost clurgmgrd[5917]: <info> I am node #2<o:p></o:p> Mar 24 20:02:16 localhost clurgmgrd[5917]: <notice> Resource Group Manager Starting<o:p></o:p> Mar 24 20:02:16 localhost clurgmgrd[5917]: <info> Loading Service Data<o:p></o:p> Mar 24 20:02:17 localhost clurgmgrd[5917]: <info> Initializing Services<o:p></o:p> Mar 24 20:02:17 localhost clurgmgrd: [5917]: <err> script:CLVMD: stop of /etc/init.d/clvmd failed (returned 143)<o:p></o:p> Mar 24 20:02:17 localhost clurgmgrd[5917]: <notice> stop on script "CLVMD" returned 1 (generic error)<o:p></o:p> <o:p> </o:p> Also reboot does not work on tweety-2 since at the point were rgmanager should shut down, again it hangs FOREVER:<o:p></o:p> <o:p> </o:p> Mar 24 20:06:06 localhost rgmanager: [8219]: <notice> Shutting down Cluster Service Manager...<o:p></o:p> <o:p> </o:p> This is the last entry on the /var/log/messages after “reboot”. Only poweroff works.<o:p></o:p> <o:p> </o:p> Any ideas someone???<o:p></o:p> <o:p> </o:p> Thank you all for your time,<o:p></o:p> Theophanis Kontogiannis<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <div> <div class=MsoNormal align=center style='margin-left:.5in;text-align:center'> <hr size=2 width="100%" align=center tabindex=-1> </div> From: linux-cluster-bounces@redhat.com [mailto:linux-cluster-bounces@redhat.com] On Behalf Of Theophanis Kontogiannis Sent: Monday, March 24, 2008 4:43 AM To: linux-cluster@redhat.com Subject: [Linux-cluster] Problem with 2 node cluster - node 2 not startingservices<o:p></o:p> </div> <o:p> </o:p> Hello All again,<o:p></o:p> I have a two nodes cluster with the following config<o:p></o:p> <?xml version="1.0"?><o:p></o:p> <cluster alias="tweety" config_version="132" name="tweety"><o:p></o:p> <fence_daemon clean_start="0" post_fail_delay="1" post_join_delay="3"/><o:p></o:p> <clusternodes><o:p></o:p> <clusternode name="tweety-1" nodeid="1" votes="1"><o:p></o:p> <fence><o:p></o:p> <method name="1"><o:p></o:p> <device name="human-fence" nodename="tweety-1"/><o:p></o:p> </method><o:p></o:p> </fence><o:p></o:p> </clusternode><o:p></o:p> <clusternode name="tweety-2" nodeid="2" votes="1"><o:p></o:p> <fence><o:p></o:p> <method name="1"><o:p></o:p> <device name="human-fence" nodename="tweety-2"/><o:p></o:p> </method><o:p></o:p> </fence><o:p></o:p> </clusternode><o:p></o:p> </clusternodes><o:p></o:p> <cman expected_votes="1" two_node="1"/><o:p></o:p> <fencedevices><o:p></o:p> <fencedevice agent="fence_manual" name="human-fence"/><o:p></o:p> </fencedevices><o:p></o:p> <rm log_level="7"><o:p></o:p> <failoverdomains><o:p></o:p> <failoverdomain name="tweety1" ordered="0" restricted="1"><o:p></o:p> <failoverdomainnode name="tweety-1" priority="1"/><o:p></o:p> </failoverdomain><o:p></o:p> <failoverdomain name="tweety2" ordered="0" restricted="1"><o:p></o:p> <failoverdomainnode name="tweety-2" priority="1"/><o:p></o:p> </failoverdomain><o:p></o:p> <failoverdomain name="tweety-cluster" ordered="1" restricted="1"><o:p></o:p> <failoverdomainnode name="tweety-2" priority="1"/><o:p></o:p> <failoverdomainnode name="tweety-1" priority="1"/><o:p></o:p> </failoverdomain><o:p></o:p> <failoverdomain name="tweety-1-2" ordered="1" restricted="1"><o:p></o:p> <failoverdomainnode name="tweety-1" priority="1"/><o:p></o:p> <failoverdomainnode name="tweety-2" priority="2"/><o:p></o:p> </failoverdomain><o:p></o:p> <failoverdomain name="tweety-2-1" ordered="1" restricted="1"><o:p></o:p> <failoverdomainnode name="tweety-1" priority="2"/><o:p></o:p> <failoverdomainnode name="tweety-2" priority="1"/><o:p></o:p> </failoverdomain><o:p></o:p> </failoverdomains><o:p></o:p> <resources><o:p></o:p> <script file="/etc/init.d/clvmd" name="clvmd"/><o:p></o:p> <script file="/etc/init.d/gfs2" name="GFS2"/><o:p></o:p> <script file="/etc/init.d/boinc" name="BOINC"/><o:p></o:p> <script file="/etc/init.d/gfs2-check" name="GFS2-Control"/><o:p></o:p> </resources><o:p></o:p> <service autostart="1" domain="tweety1" name="LV-tweety1"><o:p></o:p> <script ref="clvmd"><o:p></o:p> <script ref="GFS2"/><o:p></o:p> </script><o:p></o:p> </service><o:p></o:p> <service autostart="1" domain="tweety2" name="LV-tweety2"><o:p></o:p> <script ref="clvmd"><o:p></o:p> <script ref="GFS2"/><o:p></o:p> </script><o:p></o:p> </service><o:p></o:p> <service autostart="1" domain="tweety1" name="BOINC-t1"><o:p></o:p> <script ref="BOINC"/><o:p></o:p> </service><o:p></o:p> <service autostart="1" domain="tweety2" exclusive="0" name="BOINC-t2" recovery="restart"><o:p></o:p> <script ref="BOINC"/><o:p></o:p> </service><o:p></o:p> </rm><o:p></o:p> </cluster><o:p></o:p> <o:p> </o:p> Tweety-1 boots up smoothly and brings up all the services<o:p></o:p> Tweety-2 boots up smoothly and brings up no services unless I manually do “service clvmd start” and “service gfs2 start”<o:p></o:p> The log on tweety-2 is:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service handler 'openais distributed locking service B.01.01'<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service handler 'openais message service B.01.01'<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service handler 'openais configuration service'<o:p></o:p> Mar 24 04:30:18 localhost ccsd[2672]: Cluster is not quorate. Refusing connection.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service handler 'openais cluster closed process group service v1.01'<o:p></o:p> Mar 24 04:30:18 localhost ccsd[2672]: Error while processing connect: Connection refused<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service handler 'openais CMAN membership service 2.01'<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CMAN ] CMAN 2.0.73 (built Nov 29 2007 18:40:32) started<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SYNC ] Not using a virtual synchrony filter.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Creating commit token because I am the rep.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Saving state aru 0 high seq received 0<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Storing new sequence id for ring 41c<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering COMMIT state.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering RECOVERY state.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] position [0] member 10.254.254.254:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] previous ring seq 1048 rep 10.254.254.254<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] aru 0 high delivered 0 received flag 1<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Did not need to originate any messages in recovery.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Sending initial ORF token<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0) ip(10.254.254.254)<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0) ip(10.254.254.254)<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SYNC ] This node is within the primary component and will provide service.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering OPERATIONAL state.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CMAN ] quorum regained, resuming activity<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] got nodejoin message 10.254.254.254<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering GATHER state from 11.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Saving state aru 9 high seq received 9<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Storing new sequence id for ring 420<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering COMMIT state.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering RECOVERY state.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] position [0] member 10.254.254.253:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] previous ring seq 1052 rep 10.254.254.253<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] aru c high delivered c received flag 1<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] position [1] member 10.254.254.254:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] previous ring seq 1052 rep 10.254.254.254<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] aru 9 high delivered 9 received flag 1<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Did not need to originate any messages in recovery.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0) ip(10.254.254.254)<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0) ip(10.254.254.253)<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0) ip(10.254.254.254)<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0) ip(10.254.254.253)<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [SYNC ] This node is within the primary component and will provide service.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering OPERATIONAL state.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [<st1:place w:st="on">MAIN</st1:place> ] Received message has invalid digest... ignoring.<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [MAIN ] Invalid packet data<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] got nodejoin message 10.254.254.253<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CLM ] got nodejoin message 10.254.254.254<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CPG ] got joinlist message from node 2<o:p></o:p> Mar 24 04:30:18 localhost openais[2681]: [CPG ] got joinlist message from node 1<o:p></o:p> Mar 24 04:30:18 localhost ccsd[2672]: Initial status:: Quorate<o:p></o:p> Mar 24 04:30:44 localhost modclusterd: startup succeeded<o:p></o:p> Mar 24 04:30:45 localhost kernel: dlm: Using TCP for communications<o:p></o:p> Mar 24 04:30:45 localhost kernel: dlm: connecting to 1<o:p></o:p> Mar 24 04:30:45 localhost kernel: dlm: got connection from 1<o:p></o:p> Mar 24 04:30:46 localhost clurgmgrd[3200]: <notice> Resource Group Manager Starting<o:p></o:p> Mar 24 04:30:46 localhost clurgmgrd[3200]: <info> Loading Service Data<o:p></o:p> Mar 24 04:30:55 localhost clurgmgrd[3200]: <info> Initializing Services<o:p></o:p> Mar 24 04:30:58 localhost clurgmgrd: [3200]: <err> script:clvmd: stop of /etc/init.d/clvmd failed (returned 143)<o:p></o:p> Mar 24 04:30:58 localhost clurgmgrd[3200]: <notice> stop on script "clvmd" returned 1 (generic error)<o:p></o:p> AND THAT’s IT ALL.<o:p></o:p> However on tweety-1 the log goes further then were tweety-2 stops:<o:p></o:p> Mar 24 04:23:39 tweety1 clurgmgrd[3379]: <info> Services Initialized<o:p></o:p> Mar 24 04:23:39 tweety1 clurgmgrd[3379]: <info> State change: Local UP<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Starting stopped service service:LV-tweety1<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Starting stopped service service:BOINC-t1<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd: [3379]: <err> script:BOINC: start of /etc/init.d/boinc failed (returned 1)<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> start on script "BOINC" returned 1 (generic error)<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <warning> #68: Failed to start service:BOINC-t1; return value: 1<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Stopping service service:BOINC-t1<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Service service:BOINC-t1 is recovering<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <warning> #71: Relocating failed service service:BOINC-t1<o:p></o:p> Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Stopping service service:BOINC-t1<o:p></o:p> Mar 24 04:23:46 tweety1 clurgmgrd[3379]: <notice> Service service:BOINC-t1 is stopped<o:p></o:p> Mar 24 04:23:46 tweety1 clvmd: Cluster LVM daemon started - connected to CMAN<o:p></o:p> Mar 24 04:23:48 tweety1 kernel: GFS2: fsid=: Trying to join cluster "lock_dlm", "tweety:gfs0"<o:p></o:p> Mar 24 04:23:48 tweety1 kernel: GFS2: fsid=tweety:gfs0.0: Joined cluster. Now mounting FS...<o:p></o:p> Mar 24 04:23:49 tweety1 clurgmgrd[3379]: <notice> Service service:LV-tweety1 started<o:p></o:p> Mar 24 04:24:42 tweety1 kernel: dlm: closing connection to node 2<o:p></o:p> Mar 24 04:25:21 tweety1 kernel: dlm: closing connection to node 2<o:p></o:p> Mar 24 04:27:32 tweety1 kernel: dlm: closing connection to node 2<o:p></o:p> Can someone give food for thoughts as to what the problem might be? Do I need to provide more information?<o:p></o:p> Thank you all for your time<o:p></o:p> Theophanis Kontogiannis<o:p></o:p> <o:p> </o:p> </div> </body> </html>