[Linux-cluster] share experience migrating cluster suite from centos 5.3 to centos 5.4
Gianluca Cecchi
gianluca.cecchi at gmail.com
Mon Nov 2 14:09:26 UTC 2009
Hello,
sorry for the long e-mail in advance.
trying to do on a test environment what in subject and I think it could be
useful for others too, both in RH EL and in CentOS.
I have configured two ip+fs services and HA-LVM
Starting point is CentOS 5.3 updated at these components:
cman-2.0.98-1.el5_3.1
openais-0.80.3-22.el5_3.4
rgmanager-2.0.46-1.el5.centos.3
luci-0.12.1-7.3.el5.centos.1
ricci-0.12.1-7.3.el5.centos.1
lvm2-2.02.40-6.el5
device-mapper-multipath-0.4.7-23.el5_3.4
Target would be:
cman-2.0.115-1.el5_4.3
openais-0.80.6-8.el5_4.1
rgmanager-2.0.52-1.el5.centos.2
luci-0.12.2-6.el5.centos
ricci-0.12.2-6.el5.centos
lvm2-2.02.46-8.el5_4.1
device-mapper-multipath-0.4.7-30.el5_4.2
they are guests in Qemu-KVM environment and I have a backup of the starting
situation, so that I can reply and change eventually order of operations.
node1 is mork, node2 is mindy
Attempt of approach:
- services are on node2 (mindy)
- shutdown ad restart node1 in single user mode
- activate network and update node1 with:
yum clean all
yum update glibc\*
yum update yum\* rpm\* python\*
yum clean all
yum update
shutdown -r now and start in single user mode to check correct start and
so on
- init 3 for node1 and join to cluster
QUESTION1: are there any incompatibilities in this first join of the
cluster, based on the different components' versions?
Would it be better in your opinion to make a shutdown of node2 and then have
node1 start alone and take the services and then upgrade node2 and have the
first contemporary two-nodes join with aligned versions of clusterware
software?
Now, following my approach, after the init 3 on node1 all was ok with
cluster join, but I forgot to do a touch of the initrd file of the updated
kernel,
due to de-optimized check in HA-LVM service comparing timestamp of initrd of
running kernel and lvm.conf
So clurgmgrd complains having
-rw-r--r-- 1 root root 16433 Nov 2 12:28 /etc/lvm/lvm.conf
newer than initrd that is dated end of September..... (see below)
Nov 2 12:41:00 mork kernel: DLM (built Sep 30 2009 12:53:28) installed
Nov 2 12:41:00 mork kernel: GFS2 (built Sep 30 2009 12:54:10) installed
Nov 2 12:41:00 mork kernel: Lock_DLM (built Sep 30 2009 12:54:16) installed
Nov 2 12:41:00 mork ccsd[2290]: Starting ccsd 2.0.115:
Nov 2 12:41:00 mork ccsd[2290]: Built: Oct 26 2009 22:01:34
Nov 2 12:41:00 mork ccsd[2290]: Copyright (C) Red Hat, Inc. 2004 All
rights reserved.
Nov 2 12:41:00 mork ccsd[2290]: cluster.conf (cluster name = clumm, version
= 5) found.
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate
node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate
node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate
node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork ccsd[2290]: Remote copy of cluster.conf is from quorate
node.
Nov 2 12:41:00 mork ccsd[2290]: Local version # : 5
Nov 2 12:41:00 mork ccsd[2290]: Remote version #: 5
Nov 2 12:41:00 mork openais[2302]: [MAIN ] AIS Executive Service RELEASE
'subrev 1887 version 0.80.6'
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Copyright (C) 2002-2006
MontaVista Software, Inc and contributors.
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Nov 2 12:41:00 mork openais[2302]: [MAIN ] AIS Executive Service: started
and ready to provide service.
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Using default multicast address
of 239.192.12.183
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Token Timeout (162000 ms)
retransmit timeout (8019 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] token hold (6405 ms) retransmits
before loss (20 retrans)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] join (60 ms) send_join (0 ms)
consensus (4800 ms) merge (200 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] downcheck (1000 ms) fail to recv
const (50 msgs)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] seqno unchanged const (30
rotations) Maximum network MTU 1500
s)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] send threads (0 threads)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP token expired timeout (8019
ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP token problem counter (2000
ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP threshold (10 problem count)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] RRP mode set to none.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] heartbeat_failures_allowed (0)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] max_network_delay (50 ms)
Nov 2 12:41:00 mork openais[2302]: [TOTEM] HeartBeat is Disabled. To enable
set heartbeat_failures_allowed > 0
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Receive multicast socket recv
buffer size (262142 bytes).
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Transmit multicast socket send
buffer size (262142 bytes).
Nov 2 12:41:00 mork openais[2302]: [TOTEM] The network interface
[172.16.0.11] is now up.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Created or loaded sequence id
336.172.16.0.11 for this ring.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering GATHER state from 15.
Nov 2 12:41:00 mork openais[2302]: [CMAN ] CMAN 2.0.115 (built Oct 26 2009
22:01:42) started
Nov 2 12:41:00 mork openais[2302]: [MAIN ] Service initialized 'openais
CMAN membership service 2.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
extended virtual synchrony service'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
cluster membership service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
availability management framework B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
checkpoint service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
event service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
distributed locking service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
message service B.01.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
configuration service'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
cluster closed process group service v1.01'
Nov 2 12:41:00 mork openais[2302]: [SERV ] Service initialized 'openais
cluster config database access v1.01'
Nov 2 12:41:00 mork openais[2302]: [SYNC ] Not using a virtual synchrony
filter.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Creating commit token because I
am the rep.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Saving state aru 0 high seq
received 0
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Storing new sequence id for ring
154
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering COMMIT state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering RECOVERY state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:41:00 mork openais[2302]: [TOTEM] previous ring seq 336 rep
172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [TOTEM] aru 0 high delivered 0 received
flag 1
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Did not need to originate any
messages in recovery.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Sending initial ORF token
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [SYNC ] This node is within the primary
component and will provide service.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:41:00 mork openais[2302]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering GATHER state from 11.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Creating commit token because I
am the rep.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Saving state aru a high seq
received a
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Storing new sequence id for ring
158
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering COMMIT state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering RECOVERY state.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:41:00 mork openais[2302]: [TOTEM] previous ring seq 340 rep
172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [TOTEM] aru a high delivered a received
flag 1
Nov 2 12:41:00 mork openais[2302]: [TOTEM] position [1] member 172.16.0.12:
Nov 2 12:41:00 mork openais[2302]: [TOTEM] previous ring seq 340 rep
172.16.0.12
Nov 2 12:41:00 mork openais[2302]: [TOTEM] aru d high delivered d received
flag 1
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Did not need to originate any
messages in recovery.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] Sending initial ORF token
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:41:00 mork openais[2302]: [CLM ] New Configuration:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Left:
Nov 2 12:41:00 mork openais[2302]: [CLM ] Members Joined:
Nov 2 12:41:00 mork openais[2302]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:41:00 mork openais[2302]: [SYNC ] This node is within the primary
component and will provide service.
Nov 2 12:41:00 mork openais[2302]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:41:00 mork openais[2302]: [CMAN ] quorum regained, resuming
activity
Nov 2 12:41:00 mork openais[2302]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:41:00 mork openais[2302]: [CLM ] got nodejoin message 172.16.0.12
Nov 2 12:41:00 mork openais[2302]: [CPG ] got joinlist message from node 2
Nov 2 12:41:01 mork ccsd[2290]: Initial status:: Quorate
uorum
Nov 2 12:41:01 mork qdiskd[2331]: <info> Quorum Daemon Initializing
Nov 2 12:41:02 mork qdiskd[2331]: <info> Heuristic: 'ping -c1 -w1
192.168.122.1' UP
Nov 2 12:41:12 mork modclusterd: startup succeeded
Nov 2 12:41:12 mork kernel: dlm: Using TCP for communications
Nov 2 12:41:12 mork kernel: dlm: connecting to 2
Nov 2 12:41:12 mork kernel: dlm: got connection from 2
Nov 2 12:41:12 mork clurgmgrd[2886]: <notice> Resource Group Manager
Starting
Nov 2 12:41:13 mork oddjobd: oddjobd startup succeeded
Nov 2 12:41:13 mork saslauthd[3338]: detach_tty : master pid is: 3338
Nov 2 12:41:13 mork saslauthd[3338]: ipc_init : listening on socket:
/var/run/saslauthd/mux
Nov 2 12:41:14 mork ricci: startup succeeded
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> HA LVM: Improper setup
detected
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> HA LVM: Improper setup
detected
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> - initrd image needs to be
newer than lvm.conf
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> - initrd image needs to be
newer than lvm.conf
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> WARNING: An improper setup can
cause data corruption!
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> WARNING: An improper setup can
cause data corruption!
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> node2 owns vg_cl1/lv_cl1
unable to stop
Nov 2 12:41:14 mork clurgmgrd: [2886]: <err> node2 owns vg_cl2/lv_cl2
unable to stop
Nov 2 12:41:14 mork clurgmgrd[2886]: <notice> stop on lvm "CL2" returned 1
(generic error)
Nov 2 12:41:14 mork clurgmgrd[2886]: <notice> stop on lvm "CL1" returned 1
(generic error)
Nov 2 12:41:31 mork qdiskd[2331]: <info> Node 2 is the master
Nov 2 12:42:21 mork qdiskd[2331]: <info> Initial score 1/1
Nov 2 12:42:21 mork qdiskd[2331]: <info> Initialization complete
Nov 2 12:42:21 mork openais[2302]: [CMAN ] quorum device registered
Nov 2 12:42:21 mork qdiskd[2331]: <notice> Score sufficient for master
operation (1/1; required=1); upgrading
Note that a clustat of both nodes gives correct results (in the sens of
nodes taking part in the cluster and rgmanager active on both and quorum
disk).
At this point, after touching initrd file, I think to do a shutdown -r of
mork again and see if all goes well.
It seems so, as I get again:
...
Nov 2 12:46:23 mork openais[2278]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:46:23 mork openais[2278]: [CLM ] New Configuration:
Nov 2 12:46:23 mork openais[2278]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:46:23 mork openais[2278]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:46:23 mork openais[2278]: [CLM ] Members Left:
Nov 2 12:46:23 mork openais[2278]: [CLM ] Members Joined:
Nov 2 12:46:23 mork openais[2278]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:46:23 mork openais[2278]: [SYNC ] This node is within the primary
component and will provide service.
Nov 2 12:46:23 mork openais[2278]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:46:23 mork openais[2278]: [CMAN ] quorum regained, resuming
activity
Nov 2 12:46:23 mork openais[2278]: [CLM ] got nodejoin message 172.16.0.11
Nov 2 12:46:23 mork openais[2278]: [CLM ] got nodejoin message 172.16.0.12
Nov 2 12:46:23 mork openais[2278]: [CPG ] got joinlist message from node 2
Nov 2 12:46:24 mork ccsd[2267]: Initial status:: Quorate
uorum
Nov 2 12:46:25 mork qdiskd[2310]: <info> Quorum Daemon Initializing
Nov 2 12:46:26 mork qdiskd[2310]: <info> Heuristic: 'ping -c1 -w1
192.168.122.1' UP
...
Nov 2 12:46:35 mork modclusterd: startup succeeded
Nov 2 12:46:35 mork kernel: dlm: Using TCP for communications
Nov 2 12:46:35 mork kernel: dlm: connecting to 2
Nov 2 12:46:36 mork oddjobd: oddjobd startup succeeded
Nov 2 12:46:36 mork saslauthd[2990]: detach_tty : master pid is: 2990
Nov 2 12:46:36 mork saslauthd[2990]: ipc_init : listening on socket:
/var/run/saslauthd/mux
Nov 2 12:46:36 mork ricci: startup succeeded
Nov 2 12:46:55 mork qdiskd[2310]: <info> Node 2 is the master
Nov 2 12:47:45 mork qdiskd[2310]: <info> Initial score 1/1
Nov 2 12:47:45 mork qdiskd[2310]: <info> Initialization complete
Nov 2 12:47:45 mork openais[2278]: [CMAN ] quorum device registered
Nov 2 12:47:45 mork qdiskd[2310]: <notice> Score sufficient for master
operation (1/1; required=1); upgrading
but instead, on mindy I get this error and the node goes out of memory and I
have to power off it....
Nov 2 12:47:54 mindy kernel: dlm: connect from non cluster node
Donna if the problem with cluster is the cause or the effect of the
problem....
In particular, these are messages on mindy, during the first join of the
cluster and the reboot of mork:
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering GATHER state from 11.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] Saving state aru d high seq
received d
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] Storing new sequence id for
ring 158
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering COMMIT state.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering RECOVERY state.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] previous ring seq 340 rep
172.16.0.11
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] aru a high delivered a received
flag 1
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] position [1] member 172.16.0.12:
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] previous ring seq 340 rep
172.16.0.12
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] aru d high delivered d received
flag 1
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] Did not need to originate any
messages in recovery.
Nov 2 12:42:20 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:42:20 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:42:20 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:42:20 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:42:20 mindy openais[2465]: [SYNC ] This node is within the primary
component and will provide service.
Nov 2 12:42:20 mindy openais[2465]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:42:20 mindy openais[2465]: [CLM ] got nodejoin message
172.16.0.11
Nov 2 12:42:20 mindy openais[2465]: [CLM ] got nodejoin message
172.16.0.12
Nov 2 12:42:20 mindy openais[2465]: [CPG ] got joinlist message from node
2
Nov 2 12:42:32 mindy kernel: dlm: connecting to 1
Nov 2 12:42:32 mindy kernel: dlm: got connection from 1
Nov 2 12:46:16 mindy clurgmgrd[3101]: <notice> Member 1 shutting down
Nov 2 12:46:26 mindy qdiskd[2508]: <info> Node 1 shutdown
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering GATHER state from 12.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] Saving state aru 3e high seq
received 3e
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] Storing new sequence id for
ring 160
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering COMMIT state.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering RECOVERY state.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] position [0] member 172.16.0.11:
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] previous ring seq 348 rep
172.16.0.11
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] aru a high delivered a received
flag 1
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] position [1] member 172.16.0.12:
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] previous ring seq 344 rep
172.16.0.11
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] aru 3e high delivered 3e
received flag 1
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] Did not need to originate any
messages in recovery.
Nov 2 12:47:43 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:47:43 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:47:43 mindy kernel: dlm: closing connection to node 1
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] CLM CONFIGURATION CHANGE
Nov 2 12:47:43 mindy openais[2465]: [CLM ] New Configuration:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.11)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] r(0) ip(172.16.0.12)
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Left:
Nov 2 12:47:43 mindy openais[2465]: [CLM ] Members Joined:
Nov 2 12:47:43 mindy openais[2465]: [SYNC ] This node is within the primary
component and will provide service.
Nov 2 12:47:43 mindy openais[2465]: [TOTEM] entering OPERATIONAL state.
Nov 2 12:47:43 mindy openais[2465]: [CLM ] got nodejoin message
172.16.0.11
Nov 2 12:47:43 mindy openais[2465]: [CLM ] got nodejoin message
172.16.0.12
Nov 2 12:47:43 mindy openais[2465]: [CPG ] got joinlist message from node
2
Nov 2 12:47:54 mindy kernel: dlm: connect from non cluster node
Nov 2 12:59:48 mindy kernel: dlm_send invoked oom-killer: gfp_mask=0xd0,
order=1, oomkilladj=0
Nov 2 12:59:48 mindy kernel:
Nov 2 12:59:48 mindy kernel: Call Trace:
Nov 2 12:59:48 mindy kernel: [<ffffffff800c3a6a>] out_of_memory+0x8e/0x2f5
Nov 2 12:59:48 mindy kernel: [<ffffffff8009dba4>]
autoremove_wake_function+0x0/0x2e
Nov 2 12:59:48 mindy kernel: [<ffffffff8000f2eb>]
__alloc_pages+0x245/0x2ce
Nov 2 12:59:48 mindy kernel: [<ffffffff8000f10b>] __alloc_pages+0x65/0x2ce
Nov 2 12:59:48 mindy kernel: [<ffffffff80017493>] cache_grow+0x137/0x395
Nov 2 12:59:48 mindy kernel: [<ffffffff8005bbf7>]
cache_alloc_refill+0x136/0x186
Nov 2 12:59:48 mindy kernel: [<ffffffff8000a96e>]
kmem_cache_alloc+0x6c/0x76
Nov 2 12:59:48 mindy kernel: [<ffffffff80043ae3>] sk_alloc+0x2e/0xf3
Nov 2 12:59:48 mindy kernel: [<ffffffff80059676>] inet_create+0x137/0x267
Nov 2 12:59:49 mindy kernel: [<ffffffff8004c9af>]
__sock_create+0x170/0x27c
Nov 2 12:59:49 mindy kernel: [<ffffffff8839086e>]
:dlm:process_send_sockets+0x0/0x179
Nov 2 12:59:49 mindy kernel: [<ffffffff883902f4>]
:dlm:tcp_connect_to_sock+0x70/0x1de
Nov 2 12:59:49 mindy kernel: [<ffffffff80063097>] thread_return+0x62/0xfe
Nov 2 12:59:49 mindy kernel: [<ffffffff8839088e>]
:dlm:process_send_sockets+0x20/0x179
Nov 2 12:59:49 mindy kernel: [<ffffffff8839086e>]
:dlm:process_send_sockets+0x0/0x179
Nov 2 12:59:49 mindy kernel: [<ffffffff8004d159>] run_workqueue+0x94/0xe4
Nov 2 12:59:49 mindy kernel: [<ffffffff800499da>] worker_thread+0x0/0x122
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>]
keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff80049aca>] worker_thread+0xf0/0x122
Nov 2 12:59:49 mindy kernel: [<ffffffff8008a4b3>]
default_wake_function+0x0/0xe
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>]
keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>]
keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff80032380>] kthread+0xfe/0x132
Nov 2 12:59:49 mindy kernel: [<ffffffff8005dfb1>] child_rip+0xa/0x11
Nov 2 12:59:49 mindy kernel: [<ffffffff8009d98c>]
keventd_create_kthread+0x0/0xc4
Nov 2 12:59:49 mindy kernel: [<ffffffff8804e024>]
:ext3:ext3_journal_dirty_data+0x0/0x34
Nov 2 12:59:49 mindy kernel: [<ffffffff80032282>] kthread+0x0/0x132
Nov 2 12:59:49 mindy kernel: [<ffffffff8005dfa7>] child_rip+0x0/0x11
Nov 2 12:59:49 mindy kernel:
Both nodes are Qemu-KVM x86_64 guests, each one assigned 1Gb of ram and 2
cpus
I can send copy of cluster.conf eventually
Thanks in advance for your comments.
Gianluca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/linux-cluster/attachments/20091102/50c0755e/attachment.htm>
More information about the Linux-cluster
mailing list