[Linux-cluster] quorum lost in spite of 'leave remove'
Kadlecsik Jozsi
kadlec at sunserv.kfki.hu
Sat Sep 1 19:42:33 UTC 2007
On Fri, 31 Aug 2007, Kadlecsik Jozsi wrote:
> On Fri, 31 Aug 2007, Kadlecsik Jozsi wrote:
>
> > > > '/etc/init.d/cman stop' was issued and executed successfully on the tree
> > > > other nodes.
> > >
> > > It looks like a bug to me. Congratulations - you're the first person to spot that!
I collected some debug logs on a node when at another one the command
'cman stop' was issued and attached it. I hope it helps.
Best regards,
Jozsef
--
E-mail : kadlec at sunserv.kfki.hu, kadlec at blackhole.kfki.hu
PGP key: http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address: KFKI Research Institute for Particle and Nuclear Physics
H-1525 Budapest 114, POB. 49, Hungary
-------------- next part --------------
waiting for aisexec to start
waiting for aisexec to start
[MAIN ] AIS Executive Service RELEASE 'subrev 1358 version 0.80.3'
[MAIN ] Copyright (C) 2002-2006 MontaVista Software, Inc and contributors.
[MAIN ] Copyright (C) 2006 Red Hat, Inc.
[MAIN ] AIS Executive Service: started and ready to provide service.
[MAIN ] Using override node name lxserv0-gfs
[MAIN ] openais component openais_cpg loaded.
[MAIN ] Registering service handler 'openais cluster closed process group service v1.01'
[MAIN ] openais component openais_cfg loaded.
[MAIN ] Registering service handler 'openais configuration service'
[MAIN ] openais component openais_msg loaded.
[MAIN ] Registering service handler 'openais message service B.01.01'
[MAIN ] openais component openais_lck loaded.
[MAIN ] Registering service handler 'openais distributed locking service B.01.01'
[MAIN ] openais component openais_evt loaded.
[MAIN ] Registering service handler 'openais event service B.01.01'
[MAIN ] openais component openais_ckpt loaded.
[MAIN ] Registering service handler 'openais checkpoint service B.01.01'
[MAIN ] openais component openais_amf loaded.
[MAIN ] Registering service handler 'openais availability management framework B.01.01'
[MAIN ] openais component openais_clm loaded.
[MAIN ] Registering service handler 'openais cluster membership service B.01.01'
[MAIN ] openais component openais_evs loaded.
[MAIN ] Registering service handler 'openais extended virtual synchrony service'
[MAIN ] openais component openais_cman loaded.
[MAIN ] Registering service handler 'openais CMAN membership service 2.01'
[TOTEM] Token Timeout (10000 ms) retransmit timeout (495 ms)
[TOTEM] token hold (386 ms) retransmits before loss (20 retrans)
[TOTEM] join (60 ms) send_join (0 ms) consensus (4800 ms) merge (200 ms)
[TOTEM] downcheck (1000 ms) fail to recv const (50 msgs)
[TOTEM] seqno unchanged const (30 rotations) Maximum network MTU 1500
[TOTEM] window size per rotation (50 messages) maximum messages per rotation (17 messages)
[TOTEM] send threads (0 threads)
[TOTEM] RRP token expired timeout (495 ms)
[TOTEM] RRP token problem counter (2000 ms)
[TOTEM] RRP threshold (10 problem count)
[TOTEM] RRP mode set to none.
[TOTEM] heartbeat_failures_allowed (0)
[TOTEM] max_network_delay (50 ms)
[TOTEM] HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
[TOTEM] Receive multicast socket recv buffer size (288000 bytes).
[TOTEM] Transmit multicast socket send buffer size (262142 bytes).
[TOTEM] The network interface [192.168.192.15] is now up.
[TOTEM] Created or loaded sequence id 944.192.168.192.15 for this ring.
[TOTEM] entering GATHER state from 15.
[SERV ] Initialising service handler 'openais extended virtual synchrony service'
[SERV ] Initialising service handler 'openais cluster membership service B.01.01'
[SERV ] Initialising service handler 'openais availability management framework B.01.01'
[SERV ] Initialising service handler 'openais checkpoint service B.01.01'
[SERV ] Initialising service handler 'openais event service B.01.01'
[SERV ] Initialising service handler 'openais distributed locking service B.01.01'
[SERV ] Initialising service handler 'openais message service B.01.01'
[SERV ] Initialising service handler 'openais configuration service'
[SERV ] Initialising service handler 'openais cluster closed process group service v1.01'
[SERV ] Initialising service handler 'openais CMAN membership service 2.01'
[CMAN ] CMAN 2.01.00 (built Aug 23 2007 12:19:58) started
[SYNC ] Not using a virtual synchrony filter.
[TOTEM] Creating commit token because I am the rep.
[TOTEM] Saving state aru 0 high seq received 0
[TOTEM] entering COMMIT state.
[TOTEM] entering RECOVERY state.
[TOTEM] position [0] member 192.168.192.15:
[TOTEM] previous ring seq 944 rep 192.168.192.15
[TOTEM] aru 0 high delivered 0 received flag 0
[TOTEM] Did not need to originate any messages in recovery.
[TOTEM] Storing new sequence id for ring 3b4
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is b
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is b
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 4000000b to fd 10
[TOTEM] Sending initial ORF token
[logging.c:0090] daemon: read 0 bytes from fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 7
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 7
[logging.c:0090] memb: command return code is -2
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000007 to fd 10
[CLM ] CLM CONFIGURATION CHANGE
[CLM ] New Configuration:
[CLM ] Members Left:
[CLM ] Members Joined:
[logging.c:0090] ais: confchg_fn called type = 1, seq=948
[SYNC ] This node is within the primary component and will provide service.
[CLM ] CLM CONFIGURATION CHANGE
[CLM ] New Configuration:
[CLM ] r(0) ip(192.168.192.15)
[CLM ] Members Left:
[CLM ] Members Joined:
[CLM ] r(0) ip(192.168.192.15)
[logging.c:0090] ais: confchg_fn called type = 0, seq=948
[logging.c:0090] ais: last memb_count = 0, current = 1
[logging.c:0090] memb: sending TRANSITION message. cluster_name = kfki
[logging.c:0090] ais: comms send message 0xbfadf2dc len = 65
[logging.c:0090] daemon: sending reply 103 to fd 10
[SYNC ] This node is within the primary component and will provide service.
[TOTEM] entering OPERATIONAL state.
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 1, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 1
[logging.c:0090] memb: add_ais_node ID=1, incarnation = 948
[CLM ] got nodejoin message 192.168.192.15
[TOTEM] entering GATHER state from 11.
[TOTEM] Saving state aru 9 high seq received 9
[TOTEM] entering COMMIT state.
[TOTEM] entering RECOVERY state.
[TOTEM] position [0] member 192.168.192.6:
[TOTEM] previous ring seq 952 rep 192.168.192.6
[TOTEM] aru 2d high delivered 2d received flag 0
[TOTEM] position [1] member 192.168.192.7:
[TOTEM] previous ring seq 952 rep 192.168.192.6
[TOTEM] aru 2d high delivered 2d received flag 0
[TOTEM] position [2] member 192.168.192.15:
[TOTEM] previous ring seq 948 rep 192.168.192.15
[TOTEM] aru 9 high delivered 9 received flag 0
[TOTEM] position [3] member 192.168.192.17:
[TOTEM] previous ring seq 952 rep 192.168.192.6
[TOTEM] aru 2d high delivered 2d received flag 0
[TOTEM] position [4] member 192.168.192.18:
[TOTEM] previous ring seq 952 rep 192.168.192.6
[TOTEM] aru 2d high delivered 2d received flag 0
[TOTEM] Did not need to originate any messages in recovery.
[TOTEM] Storing new sequence id for ring 3bc
[CLM ] CLM CONFIGURATION CHANGE
[CLM ] New Configuration:
[CLM ] r(0) ip(192.168.192.15)
[CLM ] Members Left:
[CLM ] Members Joined:
[logging.c:0090] ais: confchg_fn called type = 1, seq=956
[SYNC ] This node is within the primary component and will provide service.
[CLM ] CLM CONFIGURATION CHANGE
[CLM ] New Configuration:
[CLM ] r(0) ip(192.168.192.6)
[CLM ] r(0) ip(192.168.192.7)
[CLM ] r(0) ip(192.168.192.15)
[CLM ] r(0) ip(192.168.192.17)
[CLM ] r(0) ip(192.168.192.18)
[CLM ] Members Left:
[CLM ] Members Joined:
[CLM ] r(0) ip(192.168.192.6)
[CLM ] r(0) ip(192.168.192.7)
[CLM ] r(0) ip(192.168.192.17)
[CLM ] r(0) ip(192.168.192.18)
[logging.c:0090] ais: confchg_fn called type = 0, seq=956
[logging.c:0090] ais: last memb_count = 1, current = 5
[logging.c:0090] memb: sending TRANSITION message. cluster_name = kfki
[logging.c:0090] ais: comms send message 0xbfadf2dc len = 65
[logging.c:0090] daemon: sending reply 103 to fd 10
[SYNC ] This node is within the primary component and will provide service.
[TOTEM] entering OPERATIONAL state.
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 2, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 2
[logging.c:0090] memb: add_ais_node ID=2, incarnation = 956
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 1, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 1
[logging.c:0090] memb: add_ais_node ID=1, incarnation = 956
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 3, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 3
[logging.c:0090] memb: add_ais_node ID=3, incarnation = 956
[CMAN ] quorum regained, resuming activity
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 5, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 5
[logging.c:0090] memb: add_ais_node ID=5, incarnation = 956
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 4, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 4
[logging.c:0090] memb: add_ais_node ID=4, incarnation = 956
[CLM ] got nodejoin message 192.168.192.6
[CLM ] got nodejoin message 192.168.192.7
[CLM ] got nodejoin message 192.168.192.15
[CLM ] got nodejoin message 192.168.192.17
[CLM ] got nodejoin message 192.168.192.18
[CPG ] got joinlist message from node 2
[CPG ] got joinlist message from node 3
[CPG ] got joinlist message from node 5
[CPG ] got joinlist message from node 4
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is 1
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 1
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000001 to fd 11
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is 5
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 5
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000005 to fd 11
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is 7
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 7
[logging.c:0090] memb: get_all_members: allocated new buffer (retsize=1024)
[logging.c:0090] memb: get_all_members: retlen = 2120
[logging.c:0090] memb: command return code is 5
[logging.c:0090] daemon: Returning command data. length = 2120
[logging.c:0090] daemon: sending reply 40000007 to fd 11
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is 7
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 7
[logging.c:0090] memb: get_all_members: allocated new buffer (retsize=1024)
[logging.c:0090] memb: get_all_members: retlen = 2120
[logging.c:0090] memb: command return code is 5
[logging.c:0090] daemon: Returning command data. length = 2120
[logging.c:0090] daemon: sending reply 40000007 to fd 11
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 7
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 7
[logging.c:0090] memb: get_all_members: allocated new buffer (retsize=1024)
[logging.c:0090] memb: get_all_members: retlen = 2120
[logging.c:0090] memb: command return code is 5
[logging.c:0090] daemon: Returning command data. length = 2120
[logging.c:0090] daemon: sending reply 40000007 to fd 10
[logging.c:0090] daemon: read 0 bytes from fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 91
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 91
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 24
[logging.c:0090] daemon: sending reply 40000091 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 9
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 9
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 16
[logging.c:0090] daemon: sending reply 40000009 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 92
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 92
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 320
[logging.c:0090] daemon: sending reply 40000092 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 5
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 5
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000005 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is d
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is d
[logging.c:0090] memb: command return code is 2
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 4000000d to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 90
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 90
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 424
[logging.c:0090] daemon: sending reply 40000090 to fd 10
[logging.c:0090] daemon: read 0 bytes from fd 10
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 20, source nodeid = 2, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 7 (len = 4)
[logging.c:0090] memb: got LEAVE from node 2, reason = 3
[TOTEM] The token was lost in the OPERATIONAL state.
[TOTEM] Receive multicast socket recv buffer size (288000 bytes).
[TOTEM] Transmit multicast socket send buffer size (262142 bytes).
[TOTEM] entering GATHER state from 2.
[TOTEM] entering GATHER state from 11.
[TOTEM] Saving state aru 43 high seq received 43
[TOTEM] entering COMMIT state.
[TOTEM] entering RECOVERY state.
[TOTEM] position [0] member 192.168.192.6:
[TOTEM] previous ring seq 956 rep 192.168.192.6
[TOTEM] aru 43 high delivered 43 received flag 0
[TOTEM] position [1] member 192.168.192.15:
[TOTEM] previous ring seq 956 rep 192.168.192.6
[TOTEM] aru 43 high delivered 43 received flag 0
[TOTEM] position [2] member 192.168.192.17:
[TOTEM] previous ring seq 956 rep 192.168.192.6
[TOTEM] aru 43 high delivered 43 received flag 0
[TOTEM] position [3] member 192.168.192.18:
[TOTEM] previous ring seq 956 rep 192.168.192.6
[TOTEM] aru 43 high delivered 43 received flag 0
[TOTEM] Did not need to originate any messages in recovery.
[TOTEM] Storing new sequence id for ring 3c4
[CLM ] CLM CONFIGURATION CHANGE
[CLM ] New Configuration:
[CLM ] r(0) ip(192.168.192.6)
[CLM ] r(0) ip(192.168.192.15)
[CLM ] r(0) ip(192.168.192.17)
[CLM ] r(0) ip(192.168.192.18)
[CLM ] Members Left:
[CLM ] r(0) ip(192.168.192.7)
[CLM ] Members Joined:
[logging.c:0090] ais: confchg_fn called type = 1, seq=964
[logging.c:0090] memb: del_ais_node 2
[logging.c:0090] daemon: sending reply 102 to fd 11
[SYNC ] This node is within the primary component and will provide service.
[CLM ] CLM CONFIGURATION CHANGE
[CLM ] New Configuration:
[CLM ] r(0) ip(192.168.192.6)
[CLM ] r(0) ip(192.168.192.15)
[CLM ] r(0) ip(192.168.192.17)
[CLM ] r(0) ip(192.168.192.18)
[CLM ] Members Left:
[CLM ] Members Joined:
[logging.c:0090] ais: confchg_fn called type = 0, seq=964
[logging.c:0090] ais: last memb_count = 5, current = 4
[logging.c:0090] memb: sending TRANSITION message. cluster_name = kfki
[logging.c:0090] ais: comms send message 0xbfadf2dc len = 65
[SYNC ] This node is within the primary component and will provide service.
[TOTEM] entering OPERATIONAL state.
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is 5
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 5
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000005 to fd 11
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is 7
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 7
[logging.c:0090] memb: get_all_members: allocated new buffer (retsize=1024)
[logging.c:0090] memb: get_all_members: retlen = 2120
[logging.c:0090] memb: command return code is 5
[logging.c:0090] daemon: Returning command data. length = 2120
[logging.c:0090] daemon: sending reply 40000007 to fd 11
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is 7
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 7
[logging.c:0090] memb: get_all_members: allocated new buffer (retsize=1024)
[logging.c:0090] memb: get_all_members: retlen = 2120
[logging.c:0090] memb: command return code is 5
[logging.c:0090] daemon: Returning command data. length = 2120
[logging.c:0090] daemon: sending reply 40000007 to fd 11
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 1, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 1
[logging.c:0090] memb: add_ais_node ID=1, incarnation = 964
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 3, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 3
[logging.c:0090] memb: add_ais_node ID=3, incarnation = 964
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 5, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 5
[logging.c:0090] memb: add_ais_node ID=5, incarnation = 964
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 81, source nodeid = 4, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 5 (len = 65)
[logging.c:0090] memb: got TRANSITION from node 4
[logging.c:0090] memb: add_ais_node ID=4, incarnation = 964
[CLM ] got nodejoin message 192.168.192.6
[CLM ] got nodejoin message 192.168.192.15
[CLM ] got nodejoin message 192.168.192.17
[CLM ] got nodejoin message 192.168.192.18
[CPG ] got joinlist message from node 3
[CPG ] got joinlist message from node 5
[CPG ] got joinlist message from node 4
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 91
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 91
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 24
[logging.c:0090] daemon: sending reply 40000091 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 9
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 9
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 16
[logging.c:0090] daemon: sending reply 40000009 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 92
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 92
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 320
[logging.c:0090] daemon: sending reply 40000092 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 5
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 5
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000005 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is d
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is d
[logging.c:0090] memb: command return code is 2
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 4000000d to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 90
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 90
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 424
[logging.c:0090] daemon: sending reply 40000090 to fd 10
[logging.c:0090] daemon: read 0 bytes from fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 91
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 91
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 24
[logging.c:0090] daemon: sending reply 40000091 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 9
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 9
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 16
[logging.c:0090] daemon: sending reply 40000009 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 92
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 92
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 320
[logging.c:0090] daemon: sending reply 40000092 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 5
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 5
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000005 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is d
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is d
[logging.c:0090] memb: command return code is 2
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 4000000d to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 90
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 90
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 424
[logging.c:0090] daemon: sending reply 40000090 to fd 10
[logging.c:0090] daemon: read 0 bytes from fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 91
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 91
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 24
[logging.c:0090] daemon: sending reply 40000091 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 9
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 9
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 16
[logging.c:0090] daemon: sending reply 40000009 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 92
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 92
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 320
[logging.c:0090] daemon: sending reply 40000092 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 5
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 5
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 40000005 to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is d
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is d
[logging.c:0090] memb: command return code is 2
[logging.c:0090] daemon: Returning command data. length = 0
[logging.c:0090] daemon: sending reply 4000000d to fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 90
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 90
[logging.c:0090] memb: command return code is 0
[logging.c:0090] daemon: Returning command data. length = 424
[logging.c:0090] daemon: sending reply 40000090 to fd 10
[logging.c:0090] daemon: read 0 bytes from fd 10
[logging.c:0090] daemon: read 20 bytes from fd 10
[logging.c:0090] daemon: client command is 800000bb
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is 800000bb
[logging.c:0090] daemon: sending reply 102 to fd 11
[logging.c:0090] memb: command return code is -11
[logging.c:0090] daemon: read 20 bytes from fd 11
[logging.c:0090] daemon: client command is bc
[logging.c:0090] daemon: About to process command
[logging.c:0090] memb: command to process is bc
[logging.c:0090] memb: Shutdown reply is 1
[logging.c:0090] memb: Sending LEAVE, reason 3
[logging.c:0090] ais: comms send message 0xbfae5bdc len = 4
[logging.c:0090] memb: shutdown decision is: 0 (yes=1, no=0) flags=2
[logging.c:0090] memb: command return code is -11
[logging.c:0090] ais: deliver_fn called, iov_len = 1, iov[0].len = 20, source nodeid = 1, conversion reqd=0
[logging.c:0090] memb: Message on port 0 is 7 (len = 4)
[logging.c:0090] memb: got LEAVE from node 1, reason = 3
[logging.c:0090] daemon: send status return: 0
[logging.c:0090] daemon: sending reply c00000bb to fd 10
More information about the Linux-cluster
mailing list