[Linux-cluster] CS5 & qdisk : details about problem "Node is undead" (contd.)
Alain.Moulle
Alain.Moulle at bull.net
Mon Mar 2 14:12:48 UTC 2009
Hi,
I have a more precise sequence in the syslog on the node entering the
infernal loop "Node is undead",
I think it could give some more indication about the problem, if someone
could help me ...
Thanks a lot
Regards
Alain
[root at node3 ~]# tail -f /var/log/syslog /var/log/daemons/*&
[1] 12017
[root at node3 ~]# ==> /var/log/syslog <==
Mar 2 14:07:29 s_sys at node3 qdiskd[11100]: <info> Initial score 1/1
Mar 2 14:07:29 s_sys at node3 qdiskd[11100]: <info> Initialization complete
Mar 2 14:07:29 s_sys at node3 openais[11209]: [CMAN ] quorum device registered
Mar 2 14:07:29 s_sys at node3 qdiskd[11100]: <notice> Score sufficient for master operation (1/1; required=1); upgrading
Mar 2 14:07:38 s_sys at node3 qdiskd[11100]: <info> Node 1 is the master
*===> Here is the poweroff -nf on node2*
Mar 2 14:07:59 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (2/16)
Mar 2 14:08:00 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (3/16)
Mar 2 14:08:01 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (4/16)
Mar 2 14:08:02 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (5/16)
Mar 2 14:08:03 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (6/16)
==> /var/log/daemons/errors <==
Mar 2 13:18:24 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:18:24 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:18:25 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:18:25 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:46:31 s_sys at node3 mountd[16139]: Caught signal 15, un-registering and exiting.
Mar 2 13:46:44 s_sys at node3 dlm_controld[15234]: cluster is down, exiting
Mar 2 13:46:44 s_sys at node3 gfs_controld[15240]: cluster is down, exiting
Mar 2 13:46:44 s_sys at node3 fenced[15228]: cluster is down, exiting
Mar 2 13:46:44 s_sys at node3 qdiskd[15156]: <err> cman_dispatch: Host is down
Mar 2 13:46:44 s_sys at node3 qdiskd[15156]: <err> Halting qdisk operations
==> /var/log/daemons/info <==
Mar 2 14:06:58 s_sys at node3 openais[11209]: [MAIN ] Using default multicast address of 239.192.0.0
Mar 2 14:06:58 s_sys at node3 openais[11209]: [CMAN ] CMAN 2.0.98 (built Jan 27 2009 10:53:06) started
Mar 2 14:06:59 s_sys at node3 openais[11209]: [CMAN ] quorum regained, resuming activity
Mar 2 14:07:12 s_sys at node3 qdiskd[11100]: <info> Quorum Partition: /dev/disk/by-id/scsi-3600601607b801c00343ab689eb03de11 Label: QDISK_0_0
Mar 2 14:07:12 s_sys at node3 qdiskd[11100]: <info> Quorum Daemon Initializing
Mar 2 14:07:13 s_sys at node3 qdiskd[11100]: <info> Heuristic: 'ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W3 -c1 -t3 12.11.2.1; RES=$?; fi; fi; echo $RES | grep -w 0 ' UP
Mar 2 14:07:29 s_sys at node3 qdiskd[11100]: <info> Initial score 1/1
Mar 2 14:07:29 s_sys at node3 qdiskd[11100]: <info> Initialization complete
Mar 2 14:07:29 s_sys at node3 openais[11209]: [CMAN ] quorum device registered
Mar 2 14:07:38 s_sys at node3 qdiskd[11100]: <info> Node 1 is the master
==> /var/log/daemons/warnings <==
Mar 2 10:25:54 s_sys at node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar 2 10:25:54 s_sys at node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar 2 10:26:14 s_sys at node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar 2 10:26:14 s_sys at node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar 2 10:26:24 s_sys at node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar 2 10:26:24 s_sys at node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar 2 10:30:31 s_sys at node3 avahi-daemon[25488]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar 2 11:11:15 s_sys at node3 avahi-daemon[25474]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar 2 11:28:24 s_sys at node3 avahi-daemon[25466]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar 2 13:56:47 s_sys at node3 avahi-daemon[25408]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
[root at node3 ~]#
==> /var/log/syslog <==
Mar 2 14:08:04 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (7/16)
Mar 2 14:08:05 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (8/16)
Mar 2 14:08:06 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (9/16)
Mar 2 14:08:07 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (10/16)
Mar 2 14:08:08 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (11/16)
Mar 2 14:08:09 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (12/16)
Mar 2 14:08:10 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (13/16)
Mar 2 14:08:11 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (14/16)
Mar 2 14:08:12 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (15/16)
Mar 2 14:08:13 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (16/16)
Mar 2 14:08:14 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (17/16)
Mar 2 14:08:14 s_sys at node3 qdiskd[11100]: <debug> Node 1 DOWN
Mar 2 14:08:14 s_sys at node3 qdiskd[11100]: <debug> Making bid for master
Mar 2 14:08:15 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (18/16)
Mar 2 14:08:16 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (19/16)
Mar 2 14:08:17 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (20/16)
Mar 2 14:08:18 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (21/16)
Mar 2 14:08:19 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (22/16)
Mar 2 14:08:20 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (23/16)
Mar 2 14:08:21 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (24/16)
Mar 2 14:08:21 s_sys at node3 qdiskd[11100]: <info> Assuming master role
==> /var/log/daemons/info <==
Mar 2 14:08:21 s_sys at node3 qdiskd[11100]: <info> Assuming master role
==> /var/log/syslog <==
Mar 2 14:08:22 s_sys at node3 qdiskd[11100]: <debug> Node 1 missed an update (25/16)
Mar 2 14:08:22 s_sys at node3 qdiskd[11100]: <notice> Writing eviction notice for node 1
Mar 2 14:08:22 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:23 s_sys at node3 qdiskd[11100]: <notice> Node 1 evicted
Mar 2 14:08:24 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:24 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:24 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:25 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:25 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:25 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:26 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:26 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:26 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:27 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:27 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:27 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:28 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:28 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:28 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:29 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:29 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:29 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:29 s_sys at node3 openais[11209]: [TOTEM] The token was lost in the OPERATIONAL state.
Mar 2 14:08:29 s_sys at node3 openais[11209]: [TOTEM] Receive multicast socket recv buffer size (288000 bytes).
Mar 2 14:08:29 s_sys at node3 openais[11209]: [TOTEM] Transmit multicast socket send buffer size (262142 bytes).
Mar 2 14:08:29 s_sys at node3 openais[11209]: [TOTEM] entering GATHER state from 2.
Mar 2 14:08:30 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:30 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:30 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:31 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:31 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:31 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:32 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:32 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:32 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:33 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:33 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:33 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] entering GATHER state from 0.
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] Creating commit token because I am the rep.
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] Saving state aru 49 high seq received 49
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] Storing new sequence id for ring 27c
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] entering COMMIT state.
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] entering RECOVERY state.
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] position [0] member 12.11.2.4:
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] previous ring seq 632 rep 12.11.2.3
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] aru 49 high delivered 49 received flag 1
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] Did not need to originate any messages in recovery.
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] Sending initial ORF token
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] CLM CONFIGURATION CHANGE
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] New Configuration:
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] r(0) ip(12.11.2.4)
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] Members Left:
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] r(0) ip(12.11.2.3)
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] Members Joined:
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0641] confchg, low nodeid=2, us = 2
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0651] confchg, build downlist: 1 nodes
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] CLM CONFIGURATION CHANGE
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] New Configuration:
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] r(0) ip(12.11.2.4)
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] Members Left:
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] Members Joined:
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0662] confchg, sent downlist
Mar 2 14:08:34 s_sys at node3 openais[11209]: [SYNC ] This node is within the primary component and will provide service.
Mar 2 14:08:34 s_sys at node3 openais[11209]: [TOTEM] entering OPERATIONAL state.
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0785] downlist left_list: 1
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0393] Sending new joinlist (1 elements) to clients
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CLM ] got nodejoin message 12.11.2.4
Mar 2 14:08:34 s_kernel at node3 kernel: dlm: closing connection to node 1
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:0959] sending joinlist to cluster
Mar 2 14:08:34 s_sys at node3 openais[11209]: [CPG ] got joinlist message from node 2
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0
Mar 2 14:08:34 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:34 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:34 s_sys at node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0
Mar 2 14:08:34 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:34 s_sys at node3 fenced[11229]: node2 not a cluster member after 0 sec post_fail_delay
Mar 2 14:08:34 s_sys at node3 fenced[11229]: fencing node "node2"
==> /var/log/daemons/info <==
Mar 2 14:08:34 s_sys at node3 fenced[11229]: node2 not a cluster member after 0 sec post_fail_delay
Mar 2 14:08:34 s_sys at node3 fenced[11229]: fencing node "node2"
==> /var/log/syslog <==
Mar 2 14:08:35 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:35 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:35 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:36 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:36 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:36 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:37 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:37 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:37 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:38 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:38 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:38 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
[root at node3 ~]# Mar 2 14:08:39 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:39 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:39 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:39 s_sys at node3 fenced[11229]: fence "node2" success
Mar 2 14:08:39 s_sys at node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0
Mar 2 14:08:39 s_sys at node3 openais[11209]: [cpg.c:1114] got mcast request on 0x128a57e0
==> /var/log/daemons/info <==
Mar 2 14:08:39 s_sys at node3 fenced[11229]: fence "node2" success
fg
==> /var/log/syslog <==
Mar 2 14:08:40 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:40 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:40 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
tail -f /var/log/syslog /var/log/daemons/*
[root at node3 ~]# tail -f /var/log/syslog /var/log/daemons/*&
[1] 12044
[root at node3 ~]# ==> /var/log/syslog <==
Mar 2 14:08:41 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:42 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:42 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:42 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:43 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:43 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:43 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:44 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:44 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:44 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
==> /var/log/daemons/errors <==
Mar 2 13:18:24 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:18:24 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:18:25 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:18:25 s_sys at node3 ccsd[15202]: Error while processing connect: Connection refused
Mar 2 13:46:31 s_sys at node3 mountd[16139]: Caught signal 15, un-registering and exiting.
Mar 2 13:46:44 s_sys at node3 dlm_controld[15234]: cluster is down, exiting
Mar 2 13:46:44 s_sys at node3 gfs_controld[15240]: cluster is down, exiting
Mar 2 13:46:44 s_sys at node3 fenced[15228]: cluster is down, exiting
Mar 2 13:46:44 s_sys at node3 qdiskd[15156]: <err> cman_dispatch: Host is down
Mar 2 13:46:44 s_sys at node3 qdiskd[15156]: <err> Halting qdisk operations
==> /var/log/daemons/info <==
Mar 2 14:07:12 s_sys at node3 qdiskd[11100]: <info> Quorum Daemon Initializing
Mar 2 14:07:13 s_sys at node3 qdiskd[11100]: <info> Heuristic: 'ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W1 -c1 -t3 12.11.2.1; RES=$?; if [ $RES -ne 0 ]; then ping -W3 -c1 -t3 12.11.2.1; RES=$?; fi; fi; echo $RES | grep -w 0 ' UP
Mar 2 14:07:29 s_sys at node3 qdiskd[11100]: <info> Initial score 1/1
Mar 2 14:07:29 s_sys at node3 qdiskd[11100]: <info> Initialization complete
Mar 2 14:07:29 s_sys at node3 openais[11209]: [CMAN ] quorum device registered
Mar 2 14:07:38 s_sys at node3 qdiskd[11100]: <info> Node 1 is the master
Mar 2 14:08:21 s_sys at node3 qdiskd[11100]: <info> Assuming master role
Mar 2 14:08:34 s_sys at node3 fenced[11229]: node2 not a cluster member after 0 sec post_fail_delay
Mar 2 14:08:34 s_sys at node3 fenced[11229]: fencing node "node2"
Mar 2 14:08:39 s_sys at node3 fenced[11229]: fence "node2" success
==> /var/log/daemons/warnings <==
Mar 2 10:25:54 s_sys at node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar 2 10:25:54 s_sys at node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar 2 10:26:14 s_sys at node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar 2 10:26:14 s_sys at node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar 2 10:26:24 s_sys at node3 clurgmgrd: [5635]: <warning> Link for eth0: Not detected
Mar 2 10:26:24 s_sys at node3 clurgmgrd: [5635]: <warning> No link on eth0...
Mar 2 10:30:31 s_sys at node3 avahi-daemon[25488]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar 2 11:11:15 s_sys at node3 avahi-daemon[25474]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar 2 11:28:24 s_sys at node3 avahi-daemon[25466]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Mar 2 13:56:47 s_sys at node3 avahi-daemon[25408]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
[root at node3 ~]#
[root at node3 ~]#
[root at node3 ~]#
[root at node3 ~]#
==> /var/log/syslog <==
Mar 2 14:08:45 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:45 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:45 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
tail -f /var/log/syslog /var/log/daemons/Mar 2 14:08:46 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:46 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:46 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN Mar 2 14:08:47 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:47 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:47 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:48 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:48 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:48 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:49 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:49 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:49 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:50 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:50 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:50 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:51 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:51 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:51 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:52 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:52 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:52 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:53 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:53 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:53 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:54 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:54 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:54 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:55 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:55 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:55 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:56 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:56 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:56 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:57 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:57 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:57 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:58 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:58 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:58 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:08:59 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:08:59 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:08:59 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:09:00 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:09:00 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:09:00 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:09:01 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:09:01 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:09:01 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:09:02 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:09:02 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
Mar 2 14:09:02 s_sys at node3 qdiskd[11100]: <debug> Telling CMAN to kill the node
Mar 2 14:09:03 s_sys at node3 qdiskd[11100]: <crit> Node 1 is undead.
Mar 2 14:09:03 s_sys at node3 qdiskd[11100]: <alert> Writing eviction notice for node 1
M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20090302/2466bc74/attachment.htm>
More information about the Linux-cluster
mailing list