[Linux-cluster] Problem with 2 node cluster - node 2 not starting services
Theophanis Kontogiannis
theophanis_kontogiannis at yahoo.gr
Mon Mar 24 02:42:34 UTC 2008
Hello All again,
I have a two nodes cluster with the following config
<?xml version="1.0"?>
<cluster alias="tweety" config_version="132" name="tweety">
<fence_daemon clean_start="0" post_fail_delay="1"
post_join_delay="3"/>
<clusternodes>
<clusternode name="tweety-1" nodeid="1" votes="1">
<fence>
<method name="1">
<device name="human-fence"
nodename="tweety-1"/>
</method>
</fence>
</clusternode>
<clusternode name="tweety-2" nodeid="2" votes="1">
<fence>
<method name="1">
<device name="human-fence"
nodename="tweety-2"/>
</method>
</fence>
</clusternode>
</clusternodes>
<cman expected_votes="1" two_node="1"/>
<fencedevices>
<fencedevice agent="fence_manual" name="human-fence"/>
</fencedevices>
<rm log_level="7">
<failoverdomains>
<failoverdomain name="tweety1" ordered="0"
restricted="1">
<failoverdomainnode name="tweety-1"
priority="1"/>
</failoverdomain>
<failoverdomain name="tweety2" ordered="0"
restricted="1">
<failoverdomainnode name="tweety-2"
priority="1"/>
</failoverdomain>
<failoverdomain name="tweety-cluster" ordered="1"
restricted="1">
<failoverdomainnode name="tweety-2"
priority="1"/>
<failoverdomainnode name="tweety-1"
priority="1"/>
</failoverdomain>
<failoverdomain name="tweety-1-2" ordered="1"
restricted="1">
<failoverdomainnode name="tweety-1"
priority="1"/>
<failoverdomainnode name="tweety-2"
priority="2"/>
</failoverdomain>
<failoverdomain name="tweety-2-1" ordered="1"
restricted="1">
<failoverdomainnode name="tweety-1"
priority="2"/>
<failoverdomainnode name="tweety-2"
priority="1"/>
</failoverdomain>
</failoverdomains>
<resources>
<script file="/etc/init.d/clvmd" name="clvmd"/>
<script file="/etc/init.d/gfs2" name="GFS2"/>
<script file="/etc/init.d/boinc" name="BOINC"/>
<script file="/etc/init.d/gfs2-check"
name="GFS2-Control"/>
</resources>
<service autostart="1" domain="tweety1" name="LV-tweety1">
<script ref="clvmd">
<script ref="GFS2"/>
</script>
</service>
<service autostart="1" domain="tweety2" name="LV-tweety2">
<script ref="clvmd">
<script ref="GFS2"/>
</script>
</service>
<service autostart="1" domain="tweety1" name="BOINC-t1">
<script ref="BOINC"/>
</service>
<service autostart="1" domain="tweety2" exclusive="0"
name="BOINC-t2" recovery="restart">
<script ref="BOINC"/>
</service>
</rm>
</cluster>
Tweety-1 boots up smoothly and brings up all the services
Tweety-2 boots up smoothly and brings up no services unless I manually do
"service clvmd start" and "service gfs2 start"
The log on tweety-2 is:
Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service
handler 'openais distributed locking service B.01.01'
Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service
handler 'openais message service B.01.01'
Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service
handler 'openais configuration service'
Mar 24 04:30:18 localhost ccsd[2672]: Cluster is not quorate. Refusing
connection.
Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service
handler 'openais cluster closed process group service v1.01'
Mar 24 04:30:18 localhost ccsd[2672]: Error while processing connect:
Connection refused
Mar 24 04:30:18 localhost openais[2681]: [SERV ] Initialising service
handler 'openais CMAN membership service 2.01'
Mar 24 04:30:18 localhost openais[2681]: [CMAN ] CMAN 2.0.73 (built Nov 29
2007 18:40:32) started
Mar 24 04:30:18 localhost openais[2681]: [SYNC ] Not using a virtual
synchrony filter.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Creating commit token
because I am the rep.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Saving state aru 0 high seq
received 0
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Storing new sequence id for
ring 41c
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering COMMIT state.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering RECOVERY state.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] position [0] member
10.254.254.254:
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] previous ring seq 1048 rep
10.254.254.254
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] aru 0 high delivered 0
received flag 1
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Did not need to originate
any messages in recovery.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Sending initial ORF token
Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE
Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE
Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0)
ip(10.254.254.254)
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0)
ip(10.254.254.254)
Mar 24 04:30:18 localhost openais[2681]: [SYNC ] This node is within the
primary component and will provide service.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering OPERATIONAL state.
Mar 24 04:30:18 localhost openais[2681]: [CMAN ] quorum regained, resuming
activity
Mar 24 04:30:18 localhost openais[2681]: [CLM ] got nodejoin message
10.254.254.254
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering GATHER state from
11.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Saving state aru 9 high seq
received 9
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Storing new sequence id for
ring 420
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering COMMIT state.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering RECOVERY state.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] position [0] member
10.254.254.253:
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] previous ring seq 1052 rep
10.254.254.253
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] aru c high delivered c
received flag 1
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] position [1] member
10.254.254.254:
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] previous ring seq 1052 rep
10.254.254.254
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] aru 9 high delivered 9
received flag 1
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] Did not need to originate
any messages in recovery.
Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE
Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0)
ip(10.254.254.254)
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] CLM CONFIGURATION CHANGE
Mar 24 04:30:18 localhost openais[2681]: [CLM ] New Configuration:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0)
ip(10.254.254.253)
Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0)
ip(10.254.254.254)
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Left:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] Members Joined:
Mar 24 04:30:18 localhost openais[2681]: [CLM ] r(0)
ip(10.254.254.253)
Mar 24 04:30:18 localhost openais[2681]: [SYNC ] This node is within the
primary component and will provide service.
Mar 24 04:30:18 localhost openais[2681]: [TOTEM] entering OPERATIONAL state.
Mar 24 04:30:18 localhost openais[2681]: [MAIN ] Received message has
invalid digest... ignoring.
Mar 24 04:30:18 localhost openais[2681]: [MAIN ] Invalid packet data
Mar 24 04:30:18 localhost openais[2681]: [CLM ] got nodejoin message
10.254.254.253
Mar 24 04:30:18 localhost openais[2681]: [CLM ] got nodejoin message
10.254.254.254
Mar 24 04:30:18 localhost openais[2681]: [CPG ] got joinlist message from
node 2
Mar 24 04:30:18 localhost openais[2681]: [CPG ] got joinlist message from
node 1
Mar 24 04:30:18 localhost ccsd[2672]: Initial status:: Quorate
Mar 24 04:30:44 localhost modclusterd: startup succeeded
Mar 24 04:30:45 localhost kernel: dlm: Using TCP for communications
Mar 24 04:30:45 localhost kernel: dlm: connecting to 1
Mar 24 04:30:45 localhost kernel: dlm: got connection from 1
Mar 24 04:30:46 localhost clurgmgrd[3200]: <notice> Resource Group Manager
Starting
Mar 24 04:30:46 localhost clurgmgrd[3200]: <info> Loading Service Data
Mar 24 04:30:55 localhost clurgmgrd[3200]: <info> Initializing Services
Mar 24 04:30:58 localhost clurgmgrd: [3200]: <err> script:clvmd: stop of
/etc/init.d/clvmd failed (returned 143)
Mar 24 04:30:58 localhost clurgmgrd[3200]: <notice> stop on script "clvmd"
returned 1 (generic error)
AND THAT's IT ALL.
However on tweety-1 the log goes further then were tweety-2 stops:
Mar 24 04:23:39 tweety1 clurgmgrd[3379]: <info> Services Initialized
Mar 24 04:23:39 tweety1 clurgmgrd[3379]: <info> State change: Local UP
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Starting stopped service
service:LV-tweety1
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Starting stopped service
service:BOINC-t1
Mar 24 04:23:45 tweety1 clurgmgrd: [3379]: <err> script:BOINC: start of
/etc/init.d/boinc failed (returned 1)
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> start on script "BOINC"
returned 1 (generic error)
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <warning> #68: Failed to start
service:BOINC-t1; return value: 1
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Stopping service
service:BOINC-t1
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Service service:BOINC-t1
is recovering
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <warning> #71: Relocating failed
service service:BOINC-t1
Mar 24 04:23:45 tweety1 clurgmgrd[3379]: <notice> Stopping service
service:BOINC-t1
Mar 24 04:23:46 tweety1 clurgmgrd[3379]: <notice> Service service:BOINC-t1
is stopped
Mar 24 04:23:46 tweety1 clvmd: Cluster LVM daemon started - connected to
CMAN
Mar 24 04:23:48 tweety1 kernel: GFS2: fsid=: Trying to join cluster
"lock_dlm", "tweety:gfs0"
Mar 24 04:23:48 tweety1 kernel: GFS2: fsid=tweety:gfs0.0: Joined cluster.
Now mounting FS...
Mar 24 04:23:49 tweety1 clurgmgrd[3379]: <notice> Service service:LV-tweety1
started
Mar 24 04:24:42 tweety1 kernel: dlm: closing connection to node 2
Mar 24 04:25:21 tweety1 kernel: dlm: closing connection to node 2
Mar 24 04:27:32 tweety1 kernel: dlm: closing connection to node 2
Can someone give food for thoughts as to what the problem might be? Do I
need to provide more information?
Thank you all for your time
Theophanis Kontogiannis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20080324/e59ca186/attachment.htm>
More information about the Linux-cluster
mailing list