<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.16.3"> </HEAD> <BODY> We have a premium subscription ticket open on this already, but I wanted to throw the question out there to this development list to possibly hear from its software engineers and make this scenario more clear to its users: 1) When one node detects 'missed too many heartbeats', what decision-making process goes into effect towards the final outcome of fencing the node? 2) If a few nodes are down for maintenance, and they left the cluster with "remove" for adjustment of 'quorum' count, but not 'expected' count, how might this affect question #1? It would be even more excellent If the responses could apply using our RHEL AS 4.5 11-node cluster as example: <TT>$ </TT><TT>cman_tool nodes</TT> <TT>Node Votes Exp Sts Name</TT> <TT> 1 1 19 M db2</TT> <TT> 2 5 19 M net1</TT> <TT> 3 5 19 M net2</TT> <TT> 4 1 19 M db4</TT> <TT> 5 1 19 M db1</TT> <TT> 6 1 19 M db5</TT> <TT> 7 1 19 X app3</TT> <TT> 8 1 19 X app2</TT> <TT> 9 1 19 M app6</TT> <TT> 10 1 19 M db3</TT> <TT> 11 1 19 X net3</TT> LVS network tier: net1 (5-votes), net2 (5-votes), net3 (remove) Application tier: app2 (remove), app3 (remove), app6 Database tier: db1, db2, db3, db4, db5 Expected: 19, Quorum: 9, Total votes: 16 FYI: the nodes net3, app2, app3 left this cluster with "remove" to do some isolated testing of RHEL AS 4.6 update, but only net3 was left powered on. It was in this state for over a week. As seen in syslog messages from each member that net1 went 'dark': Mar 15 16:20:28 net2 kernel: CMAN: node net1 has been removed from the cluster : Missed too many heartbeats Mar 15 16:20:29 net2 fenced[19273]: fencing deferred to db2 Mar 15 16:23:05 net2 clurgmgrd[20012]: <info> Magma Event: Membership Change Mar 15 16:23:05 net2 clurgmgrd[20012]: <info> State change: net1 DOWN Mar 15 12:29:16 app6 kernel: CMAN: node net1 has been removed from the cluster : Missed too many heartbeats Mar 15 12:29:17 app6 fenced[19015]: fencing deferred to db2 Mar 15 12:31:53 app6 clurgmgrd[21831]: <info> Magma Event: Membership Change Mar 15 12:31:53 app6 clurgmgrd[21831]: <info> State change: net1 DOWN Mar 15 16:29:19 db1 kernel: CMAN: node net1 has been removed from the cluster : Missed too many heartbeats Mar 15 16:29:20 db1 fenced[19297]: fencing deferred to db2 Mar 15 16:31:56 db1 clurgmgrd[21436]: <info> Magma Event: Membership Change Mar 15 16:31:56 db1 clurgmgrd[21436]: <info> State change: net1 DOWN Mar 15 16:29:19 db2 kernel: CMAN: removing node net1 from the cluster : Missed too many heartbeats Mar 15 16:29:20 db2 fenced[14778]: net1 not a cluster member after 0 sec post_fail_delay Mar 15 16:29:20 db2 fenced[14778]: fencing node "net1" Mar 15 16:31:48 db2 ccsd[14677]: Attempt to close an unopened CCS descriptor (151704870). Mar 15 16:31:48 db2 ccsd[14677]: Error while processing disconnect: Invalid request descriptor Mar 15 16:31:48 db2 fenced[14778]: fence "net1" success Mar 15 16:29:19 db3 kernel: CMAN: node net1 has been removed from the cluster : Missed too many heartbeats Mar 15 16:29:20 db3 fenced[19097]: fencing deferred to db2 Mar 15 16:31:56 db3 clurgmgrd[21315]: <info> Magma Event: Membership Change Mar 15 16:31:56 db3 clurgmgrd[21315]: <info> State change: net1 DOWN Mar 15 16:29:19 db4 kernel: CMAN: node net1 has been removed from the cluster : Missed too many heartbeats Mar 15 16:29:20 db4 fenced[19126]: fencing deferred to db2 Mar 15 16:31:56 db4 clurgmgrd[21182]: <info> Magma Event: Membership Change Mar 15 16:31:56 db4 clurgmgrd[21182]: <info> State change: net1 DOWN Mar 15 16:29:19 db5 kernel: CMAN: node net1 has been removed from the cluster : Missed too many heartbeats Mar 15 16:29:20 db5 fenced[14508]: fencing deferred to db2 Mar 15 16:31:56 db5 clurgmgrd[17187]: <info> Magma Event: Membership Change Mar 15 16:31:56 db5 clurgmgrd[17187]: <info> State change: net1 DOWN It may be of no consequence, but also note that there was clock drift on net2, because of a failed NTP server; and also app6 because its clock was not calibrated after being down for a motherboard swapout and memory upgrade for a few weeks. <TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%"> <TR> <TD> <TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%"> <TR> <TD> Robert Hurst, Sr. Caché Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ∙ Fax: 617-754-8730 ∙ Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. </TD> </TR> </TABLE> </TD> </TR> </TABLE> </BODY> </HTML>