<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.10.2"> </HEAD> <BODY> Troubling, this behavior has not occurred prior to our Mar 2nd up2date on our RHEL GFS/CS subscription. I rebooted an application server (app1) in an 11-node cluster, and from viewing its console, it 'hung' on a service cman stop. Consequently, ALL GFS I/O got blocked on ALL nodes. All servers are configured the same: AMD64 dual CPU/duo core HP DL385, 8GB RAM, dual hba (PowerPath) # uname -r 2.6.9-42.0.10.ELsmp ccs-1.0.7-0 cman-1.0.11-0 dlm-1.0.1-1 fence-1.32.25-1 GFS-6.1.6-1 magma-1.0.6-0 magma-plugins-1.0.9-0 rgmanager-1.9.54-1 My central syslog server showed that all nodes registered the membership change, yet the service continued to hang. <TT>Mar 20 11:06:18 app1 shutdown: shutting down for system reboot </TT> <TT>Mar 20 11:06:18 app1 init: Switching to runlevel: 6 </TT> <TT>Mar 20 11:06:19 app1 rgmanager: [1873]: <notice> Shutting down Cluster Service Manager... </TT> <TT>Mar 20 11:06:20 app1 clurgmgrd[11220]: <notice> Shutting down </TT> <TT>Mar 20 11:06:20 net2 clurgmgrd[30893]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 app3 clurgmgrd[11092]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 db1 clurgmgrd[8351]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 db3 clurgmgrd[8279]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 db2 clurgmgrd[10875]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 app6 clurgmgrd[10959]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 app4 clurgmgrd[11146]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 app2 clurgmgrd[10835]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 app5 clurgmgrd[11198]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:06:20 net1 clurgmgrd[12689]: <info> State change: app1 DOWN </TT> <TT>Mar 20 11:12:26 net2 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats</TT> <TT>Mar 20 11:12:26 db2 kernel: CMAN: removing node app1 from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 db3 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 app4 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 app5 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 app6 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 net1 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 app3 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 db1 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:26 app2 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats </TT> <TT>Mar 20 11:12:32 net1 fenced[10510]: app1 not a cluster member after 0 sec post_fail_delay </TT> <TT>Mar 20 11:12:32 net1 fenced[10510]: fencing node "app1" </TT> <TT>Mar 20 11:13:42 net1 fenced[10510]: fence "app1" success </TT> I issued a 'power reset' on its HP ILO management port to hardware reboot the server around 11:12. That is when the net1 server attempted to fence app1, after it was missing. Here's net1's syslog entries on that event: <TT>Mar 20 11:06:20 net1 clurgmgrd[12689]: <info> Magma Event: Membership Change </TT> <TT>Mar 20 11:06:20 net1 clurgmgrd[12689]: <info> State change: app1 DOWN</TT> <TT>Mar 20 11:12:26 net1 kernel: CMAN: node app1 has been removed from the cluster : Missed too many heartbeats</TT> <TT>Mar 20 11:12:32 net1 fenced[10510]: app1 not a cluster member after 0 sec post_fail_delay</TT> <TT>Mar 20 11:12:32 net1 fenced[10510]: fencing node "app1"</TT> <TT>Mar 20 11:13:42 net1 fenced[10510]: fence "app1" success</TT> <TT>Mar 20 11:15:45 net1 kernel: CMAN: node app1 rejoining</TT> <TT>Mar 20 11:18:05 net1 clurgmgrd[12689]: <info> Magma Event: Membership Change </TT> <TT>Mar 20 11:18:05 net1 clurgmgrd[12689]: <info> State change: app1 UP </TT> <TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%"> <TR> <TD> Robert Hurst, Sr. Caché Administrator Beth Israel Deaconess Medical Center 1135 Tremont Street, REN-7 Boston, Massachusetts 02120-2140 617-754-8754 ∙ Fax: 617-754-8730 ∙ Cell: 401-787-3154 Any technology distinguishable from magic is insufficiently advanced. </TD> </TR> </TABLE> </BODY> </HTML>