<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=koi8-r"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style></style></head><body lang=RU link="#0563C1" vlink="#954F72"><div class=WordSection1>Dear colleagues,<o:p></o:p><o:p> </o:p>Your advices will be greatly appreciated.<o:p></o:p><o:p> </o:p>I have another small GFS2 cluster. 2 nodes connected to the same iSCSI-target.<o:p></o:p><o:p> </o:p>Tonight something has happen and now both nodes can’t work with the mounted filesystem anymore.<o:p></o:p><o:p> </o:p>Processes that opened files on the filesystem are keeping files opened and working with them, but I can’t open new files, I even can’t get the list of files on the mountpoint by “ls” command.<o:p></o:p><o:p> </o:p>Both nodes are joined:<o:p></o:p><o:p> </o:p>Nodeš Stsšš Incšš Joinedšššššššššššššš Name<o:p></o:p>šš 1šš Mššš 388šš 2013-11-26 03:43:01š ***<o:p></o:p>šš 2šš Mššš 360šš 2013-11-11 07:39:22š ***<o:p></o:p><o:p> </o:p>That’s what “gfs_control dump” says:<o:p></o:p><o:p> </o:p>1384148367 logging mode 3 syslog f 160 p 6 logfile p 6 /var/log/cluster/gfs_controld.log<o:p></o:p>1384148367 gfs_controld 3.0.12.1 started<o:p></o:p>1384148367 cluster node 1 added seq 364<o:p></o:p>1384148367 cluster node 2 added seq 364<o:p></o:p>1384148367 logging mode 3 syslog f 160 p 6 logfile p 6 /var/log/cluster/gfs_controld.log<o:p></o:p>1384148367 group_mode 3 compat 0<o:p></o:p>1384148367 setup_cpg_daemon 14<o:p></o:p>1384148367 gfs:controld conf 2 1 0 memb 1 2 join 2 left<o:p></o:p>1384148367 run protocol from nodeid 1<o:p></o:p>1384148367 daemon run 1.1.1 max 1.1.1 kernel run 1.1.1 max 1.1.1<o:p></o:p>1384148372 client connection 5 fd 16<o:p></o:p>1384148372 join: /mnt/psv4 gfs2 lock_dlm ckvm1_pod1:psv4 rw,noatime,nodiratime /dev/dm-0<o:p></o:p>1384148372 psv4 join: cluster name matches: ckvm1_pod1<o:p></o:p>1384148372 psv4 process_dlmcontrol register 0<o:p></o:p>1384148372 gfs:mount:psv4 conf 2 1 0 memb 1 2 join 2 left<o:p></o:p>1384148372 psv4 add_change cg 1 joined nodeid 2<o:p></o:p>1384148372 psv4 add_change cg 1 we joined<o:p></o:p>1384148372 psv4 add_change cg 1 counts member 2 joined 1 remove 0 failed 0<o:p></o:p>1384148372 psv4 wait_conditions skip for zero started_count<o:p></o:p>1384148372 psv4 send_start cg 1 id_count 2 om 0 nm 2 oj 0 nj 0<o:p></o:p>1384148372 psv4 receive_start 2:1 len 104<o:p></o:p>1384148372 psv4 match_change 2:1 matches cg 1<o:p></o:p>1384148372 psv4 wait_messages cg 1 need 1 of 2<o:p></o:p>1384148372 psv4 receive_start 1:2 len 104<o:p></o:p>1384148372 psv4 match_change 1:2 matches cg 1<o:p></o:p>1384148372 psv4 wait_messages cg 1 got all 2<o:p></o:p>1384148372 psv4 pick_first_recovery_master old 1<o:p></o:p>1384148372 psv4 sync_state first_recovery_needed master 1<o:p></o:p>1384148372 psv4 create_old_nodes 1 jid 0 ro 0 spect 0 kernel_mount_done 0 error 0<o:p></o:p>1384148372 psv4 create_new_nodes 2 ro 0 spect 0<o:p></o:p>1384148372 psv4 create_new_journals 2 gets jid 1<o:p></o:p>1384148373 psv4 receive_first_recovery_done from 1 master 1 mount_client_notified 0<o:p></o:p>1384148373 psv4 start_kernel cg 1 member_count 2<o:p></o:p>1384148373 psv4 set /sys/fs/gfs2/ckvm1_pod1:psv4/lock_module/block to 0<o:p></o:p>1384148373 psv4 set open /sys/fs/gfs2/ckvm1_pod1:psv4/lock_module/block error -1 2<o:p></o:p>1384148373 psv4 client_reply_join_full ci 5 result 0 hostdata=jid=1:id=2447518500:first=0<o:p></o:p>1384148373 client_reply_join psv4 ci 5 result 0<o:p></o:p>1384148373 psv4 wait_recoveries done<o:p></o:p>1384148373 uevent add gfs2 /fs/gfs2/ckvm1_pod1:psv4<o:p></o:p>1384148373 psv4 ping_kernel_mount 0<o:p></o:p>1384148373 psv4 receive_mount_done from 1 result 0<o:p></o:p>1384148373 psv4 wait_recoveries done<o:p></o:p>1384148373 uevent change gfs2 /fs/gfs2/ckvm1_pod1:psv4<o:p></o:p>1384148373 psv4 recovery_uevent jid 1 ignore<o:p></o:p>1384148373 uevent online gfs2 /fs/gfs2/ckvm1_pod1:psv4<o:p></o:p>1384148373 psv4 ping_kernel_mount 0<o:p></o:p>1384148373 mount_done: psv4 result 0<o:p></o:p>1384148373 psv4 receive_mount_done from 2 result 0<o:p></o:p>1384148373 psv4 wait_recoveries done<o:p></o:p>1385430013 cluster node 1 removed seq 368<o:p></o:p>1385430013 gfs:controld conf 1 0 1 memb 2 join left 1<o:p></o:p>1385430013 gfs:mount:psv4 conf 1 0 1 memb 2 join left 1<o:p></o:p>1385430013 psv4 add_change cg 2 remove nodeid 1 reason 3<o:p></o:p>1385430013 psv4 add_change cg 2 counts member 1 joined 0 remove 1 failed 1<o:p></o:p>1385430013 psv4 stop_kernel<o:p></o:p>1385430013 psv4 set /sys/fs/gfs2/ckvm1_pod1:psv4/lock_module/block to 1<o:p></o:p>1385430013 psv4 check_dlm_notify nodeid 1 begin<o:p></o:p>1385430013 psv4 process_dlmcontrol notified nodeid 1 result -11<o:p></o:p>1385430013 psv4 check_dlm_notify result -11 will retry nodeid 1<o:p></o:p>1385430013 psv4 check_dlm_notify nodeid 1 begin<o:p></o:p>1385430013 psv4 process_dlmcontrol notified nodeid 1 result 0<o:p></o:p>1385430013 psv4 check_dlm_notify done<o:p></o:p>1385430013 psv4 send_start cg 2 id_count 2 om 1 nm 0 oj 0 nj 1<o:p></o:p>1385430013 psv4 receive_start 2:2 len 104<o:p></o:p>1385430013 psv4 match_change 2:2 matches cg 2<o:p></o:p>1385430013 psv4 wait_messages cg 2 got all 1<o:p></o:p>1385430013 psv4 sync_state first_recovery_msg<o:p></o:p>1385430013 psv4 set_failed_journals jid 0 nodeid 1<o:p></o:p>1385430013 psv4 wait_recoveries jid 0 nodeid 1 unrecovered<o:p></o:p>1385430013 psv4 start_journal_recovery jid 0<o:p></o:p>1385430013 psv4 set /sys/fs/gfs2/ckvm1_pod1:psv4/lock_module/recover to 0<o:p></o:p>1385430044 cluster node 1 added seq 372<o:p></o:p>1385430044 gfs:mount:psv4 conf 2 1 0 memb 1 2 join 1 left<o:p></o:p>1385430044 psv4 add_change cg 3 joined nodeid 1<o:p></o:p>1385430044 psv4 add_change cg 3 counts member 2 joined 1 remove 0 failed 0<o:p></o:p>1385430044 psv4 check_dlm_notify done<o:p></o:p>1385430044 psv4 send_start cg 3 id_count 3 om 1 nm 1 oj 1 nj 0<o:p></o:p>1385430044 cpg_mcast_joined retried 1 start<o:p></o:p>1385430044 gfs:controld conf 2 1 0 memb 1 2 join 1 left<o:p></o:p>1385430044 psv4 receive_start 2:3 len 116<o:p></o:p>1385430044 psv4 match_change 2:3 matches cg 3<o:p></o:p>1385430044 psv4 wait_messages cg 3 need 1 of 2<o:p></o:p>1385430044 psv4 receive_start 1:4 len 116<o:p></o:p>1385430044 psv4 match_change 1:4 matches cg 3<o:p></o:p>1385430044 receive_start 1:4 add node with started_count 3<o:p></o:p>1385430044 psv4 wait_messages cg 3 need 1 of 2<o:p></o:p>1385430088 cluster node 1 removed seq 376<o:p></o:p>1385430088 gfs:controld conf 1 0 1 memb 2 join left 1<o:p></o:p>1385430088 gfs:mount:psv4 conf 1 0 1 memb 2 join left 1<o:p></o:p>1385430088 psv4 add_change cg 4 remove nodeid 1 reason 3<o:p></o:p>1385430088 psv4 add_change cg 4 counts member 1 joined 0 remove 1 failed 1<o:p></o:p>1385430088 psv4 check_dlm_notify nodeid 1 begin<o:p></o:p>1385430088 psv4 process_dlmcontrol notified nodeid 1 result 0<o:p></o:p>1385430088 psv4 check_dlm_notify done<o:p></o:p>1385430088 psv4 send_start cg 4 id_count 2 om 1 nm 0 oj 1 nj 0<o:p></o:p>1385430088 psv4 receive_start 2:4 len 104<o:p></o:p>1385430088 psv4 match_change 2:4 skip 3 already start<o:p></o:p>1385430088 psv4 match_change 2:4 matches cg 4<o:p></o:p>1385430088 psv4 wait_messages cg 4 got all 1<o:p></o:p>1385430088 psv4 sync_state first_recovery_msg<o:p></o:p>1385430088 psv4 set_failed_journals no journal for nodeid 1<o:p></o:p>1385430088 psv4 wait_recoveries jid 0 nodeid 1 unrecovered<o:p></o:p>1385430092 cluster node 1 added seq 380<o:p></o:p>1385430092 gfs:mount:psv4 conf 2 1 0 memb 1 2 join 1 left<o:p></o:p>1385430092 psv4 add_change cg 5 joined nodeid 1<o:p></o:p>1385430092 psv4 add_change cg 5 counts member 2 joined 1 remove 0 failed 0<o:p></o:p>1385430092 psv4 check_dlm_notify done<o:p></o:p>1385430092 psv4 send_start cg 5 id_count 3 om 1 nm 1 oj 1 nj 0<o:p></o:p>1385430092 cpg_mcast_joined retried 1 start<o:p></o:p>1385430092 gfs:controld conf 2 1 0 memb 1 2 join 1 left<o:p></o:p>1385430092 psv4 receive_start 2:5 len 116<o:p></o:p>1385430092 psv4 match_change 2:5 matches cg 5<o:p></o:p>1385430092 psv4 wait_messages cg 5 need 1 of 2<o:p></o:p>1385430092 psv4 receive_start 1:6 len 116<o:p></o:p>1385430092 psv4 match_change 1:6 matches cg 5<o:p></o:p>1385430092 receive_start 1:6 add node with started_count 4<o:p></o:p>1385430092 psv4 wait_messages cg 5 need 1 of 2<o:p></o:p>1385430143 cluster node 1 removed seq 384<o:p></o:p>1385430143 gfs:mount:psv4 conf 1 0 1 memb 2 join left 1<o:p></o:p>1385430143 psv4 add_change cg 6 remove nodeid 1 reason 3<o:p></o:p>1385430143 psv4 add_change cg 6 counts member 1 joined 0 remove 1 failed 1<o:p></o:p>1385430143 psv4 check_dlm_notify nodeid 1 begin<o:p></o:p>1385430143 gfs:controld conf 1 0 1 memb 2 join left 1<o:p></o:p>1385430143 psv4 process_dlmcontrol notified nodeid 1 result 0<o:p></o:p>1385430143 psv4 check_dlm_notify done<o:p></o:p>1385430143 psv4 send_start cg 6 id_count 2 om 1 nm 0 oj 1 nj 0<o:p></o:p>1385430143 psv4 receive_start 2:6 len 104<o:p></o:p>1385430143 psv4 match_change 2:6 skip 5 already start<o:p></o:p>1385430143 psv4 match_change 2:6 matches cg 6<o:p></o:p>1385430143 psv4 wait_messages cg 6 got all 1<o:p></o:p>1385430143 psv4 sync_state first_recovery_msg<o:p></o:p>1385430143 psv4 set_failed_journals no journal for nodeid 1<o:p></o:p>1385430143 psv4 wait_recoveries jid 0 nodeid 1 unrecovered<o:p></o:p>1385430181 cluster node 1 added seq 388<o:p></o:p>1385430181 gfs:mount:psv4 conf 2 1 0 memb 1 2 join 1 left<o:p></o:p>1385430181 psv4 add_change cg 7 joined nodeid 1<o:p></o:p>1385430181 psv4 add_change cg 7 counts member 2 joined 1 remove 0 failed 0<o:p></o:p>1385430181 psv4 check_dlm_notify done<o:p></o:p>1385430181 psv4 send_start cg 7 id_count 3 om 1 nm 1 oj 1 nj 0<o:p></o:p>1385430181 cpg_mcast_joined retried 1 start<o:p></o:p>1385430181 gfs:controld conf 2 1 0 memb 1 2 join 1 left<o:p></o:p>1385430181 psv4 receive_start 2:7 len 116<o:p></o:p>1385430181 psv4 match_change 2:7 matches cg 7<o:p></o:p>1385430181 psv4 wait_messages cg 7 need 1 of 2<o:p></o:p>1385430181 psv4 receive_start 1:8 len 116<o:p></o:p>1385430181 psv4 match_change 1:8 matches cg 7<o:p></o:p>1385430181 receive_start 1:8 add node with started_count 5<o:p></o:p>1385430181 psv4 wait_messages cg 7 need 1 of 2<o:p></o:p><o:p> </o:p>I can’t reboot nodes, they’re pretty busy, but, of course, I’d like to make that GFS2-filesystem working again.<o:p></o:p><o:p> </o:p>There’s what I’d got in the log-file when that happened:<o:p></o:p><o:p> </o:p>Nov 26 03:40:11 host2 corosync[2596]:šš [TOTEM ] A processor failed, forming new configuration.<o:p></o:p>Nov 26 03:40:12 host2 kernel: connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 5576348348, last ping 5576353348, now 5576358348<o:p></o:p>Nov 26 03:40:12 host2 kernel: connection1:0: detected conn error (1011)<o:p></o:p>Nov 26 03:40:13 host2 iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)<o:p></o:p>Nov 26 03:40:13 host2 corosync[2596]:šš [CMANš ] quorum lost, blocking activity<o:p></o:p>Nov 26 03:40:13 host2 corosync[2596]:šš [QUORUM] This node is within the non-primary component and will NOT provide any services.<o:p></o:p>Nov 26 03:40:13 host2 corosync[2596]:šš [QUORUM] Members[1]: 2<o:p></o:p>Nov 26 03:40:13 host2 corosync[2596]:šš [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p>Nov 26 03:40:13 host2 corosync[2596]:šš [CPGšš ] chosen downlist: sender r(0) ip(192.168.1.2) ; members(old:2 left:1)<o:p></o:p>Nov 26 03:40:13 host2 corosync[2596]:šš [MAINš ] Completed service synchronization, ready to provide service.<o:p></o:p>Nov 26 03:40:13 host2 kernel: dlm: closing connection to node 1<o:p></o:p>Nov 26 03:40:13 host2 kernel: GFS2: fsid=ckvm1_pod1:psv4.1: jid=0: Trying to acquire journal lock...<o:p></o:p>Nov 26 03:40:44 host2 iscsid: connection1:0 is operational after recovery (3 attempts)<o:p></o:p>Nov 26 03:40:44 host2 corosync[2596]:šš [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p>Nov 26 03:40:44 host2 corosync[2596]:šš [CMANš ] quorum regained, resuming activity<o:p></o:p>Nov 26 03:40:44 host2 corosync[2596]:šš [QUORUM] This node is within the primary component and will provide service.<o:p></o:p>Nov 26 03:40:44 host2 corosync[2596]:šš [QUORUM] Members[2]: 1 2<o:p></o:p>Nov 26 03:40:44 host2 corosync[2596]:šš [QUORUM] Members[2]: 1 2<o:p></o:p>Nov 26 03:40:44 host2 corosync[2596]:šš [CPGšš ] chosen downlist: sender r(0) ip(192.168.1.1) ; members(old:1 left:0)<o:p></o:p>Nov 26 03:40:44 host2 corosync[2596]:šš [MAINš ] Completed service synchronization, ready to provide service.<o:p></o:p>Nov 26 03:40:44 host2 gfs_controld[2727]: receive_start 1:4 add node with started_count 3<o:p></o:p>Nov 26 03:40:44 host2 fenced[2652]: receive_start 1:4 add node with started_count 2<o:p></o:p>Nov 26 03:41:26 host2 corosync[2596]:šš [TOTEM ] A processor failed, forming new configuration.<o:p></o:p>Nov 26 03:41:28 host2 corosync[2596]:šš [CMANš ] quorum lost, blocking activity<o:p></o:p>Nov 26 03:41:28 host2 corosync[2596]:šš [QUORUM] This node is within the non-primary component and will NOT provide any services.<o:p></o:p>Nov 26 03:41:28 host2 corosync[2596]:šš [QUORUM] Members[1]: 2<o:p></o:p>Nov 26 03:41:28 host2 corosync[2596]:šš [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p>Nov 26 03:41:28 host2 corosync[2596]:šš [CPGšš ] chosen downlist: sender r(0) ip(192.168.1.2) ; members(old:2 left:1)<o:p></o:p>Nov 26 03:41:28 host2 corosync[2596]:š š[MAINš ] Completed service synchronization, ready to provide service.<o:p></o:p>Nov 26 03:41:28 host2 kernel: dlm: closing connection to node 1<o:p></o:p>Nov 26 03:41:29 host2 kernel: connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 5576425428, last ping 5576430428, now 5576435428<o:p></o:p>Nov 26 03:41:29 host2 kernel: connection1:0: detected conn error (1011)<o:p></o:p>Nov 26 03:41:30 host2 iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)<o:p></o:p>Nov 26 03:41:32 host2 corosync[2596]:šš [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p>Nov 26 03:41:32 host2 corosync[2596]:šš [CMANš ] quorum regained, resuming activity<o:p></o:p>Nov 26 03:41:32 host2 corosync[2596]:šš [QUORUM] This node is within the primary component and will provide service.<o:p></o:p>Nov 26 03:41:32 host2 corosync[2596]:šš [QUORUM] Members[2]: 1 2<o:p></o:p>Nov 26 03:41:32 host2 corosync[2596]:šš [QUORUM] Members[2]: 1 2<o:p></o:p>Nov 26 03:41:32 host2 corosync[2596]:šš [CPGšš ] chosen downlist: sender r(0) ip(192.168.1.1) ; members(old:1 left:0)<o:p></o:p>Nov 26 03:41:32 host2 corosync[2596]:šš [MAINš ] Completed service synchronization, ready to provide service.<o:p></o:p>Nov 26 03:41:32 host2 fenced[2652]: receive_start 1:6 add node with started_count 2<o:p></o:p>Nov 26 03:41:32 host2 gfs_controld[2727]: receive_start 1:6 add node with started_count 4<o:p></o:p>Nov 26 03:41:37 host2 iscsid: connection1:0 is operational after recovery (1 attempts)<o:p></o:p>Nov 26 03:42:19 host2 kernel: connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 5576475399, last ping 5576480399, now 5576485399<o:p></o:p>Nov 26 03:42:19 host2 kernel: connection1:0: detected conn error (1011)<o:p></o:p>Nov 26 03:42:20 host2 iscsid: Kernel reported iSCSI connection 1:0 error (1011 - ISCSI_ERR_CONN_FAILED: iSCSI connection failed) state (3)<o:p></o:p>Nov 26 03:42:21 host2 corosync[2596]:šš [TOTEM ] A processor failed, forming new configuration.<o:p></o:p>Nov 26 03:42:23 host2 corosync[2596]:šš [CMANš ] quorum lost, blocking activity<o:p></o:p>Nov 26 03:42:23 host2 corosync[2596]:šš [QUORUM] This node is within the non-primary component and will NOT provide any services. Nov 26 03:42:23 host2 corosync[2596]:šš [QUORUM] Members[1]: 2<o:p></o:p>Nov 26 03:42:23 host2 corosync[2596]:šš [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p>Nov 26 03:42:23 host2 corosync[2596]:šš [CPGšš ] chosen downlist: sender r(0) ip(192.168.1.2) ; members(old:2 left:1)<o:p></o:p>Nov 26 03:42:23 host2 corosync[2596]:šš [MAINš ] Completed service synchronization, ready to provide service.<o:p></o:p>Nov 26 03:42:23 host2 kernel: dlm: closing connection to node 1<o:p></o:p>Nov 26 03:42:41 host2 kernel: INFO: task kslowd001:2942 blocked for more than 120 seconds.<o:p></o:p>Nov 26 03:42:41 host2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<o:p></o:p>Nov 26 03:42:41 host2 kernel: kslowd001šššš D 000000000000000bšššš 0š 2942ššššš 2 0x00000080<o:p></o:p>Nov 26 03:42:41 host2 kernel: ffff88086b29d958 0000000000000046 0000000000000102 0000005000000002<o:p></o:p>Nov 26 03:42:41 host2 kernel: fffffffffffffffc 000000000000010e 0000003f00000002 fffffffffffffffc<o:p></o:p>Nov 26 03:42:41 host2 kernel: ffff88086b29bab8 ffff88086b29dfd8 000000000000fb88 ffff88086b29bab8<o:p></o:p>Nov 26 03:42:41 host2 kernel: Call Trace:<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff814ffec5>] rwsem_down_failed_common+0x95/0x1d0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81500056>] rwsem_down_read_failed+0x26/0x30<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8127e634>] call_rwsem_down_read_failed+0x14/0x30<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff814ff554>] ? down_read+0x24/0x30<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa06046d2>] dlm_lock+0x62/0x1e0 [dlm]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8127cd04>] ? vsnprintf+0x484/0x5f0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa06564e1>] gdlm_lock+0xf1/0x130 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa06565f0>] ? gdlm_ast+0x0/0xe0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa0656520>] ? gdlm_bast+0x0/0x50 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa063a385>] do_xmote+0x1a5/0x280 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8127cf14>] ? snprintf+0x34/0x40<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa063a551>] run_queue+0xf1/0x1d0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa063a8de>] gfs2_glock_nq+0x21e/0x3d0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa063ac71>] gfs2_glock_nq_num+0x61/0xa0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa064eca3>] gfs2_recover_work+0x93/0x7b0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8105b483>] ? perf_event_task_sched_out+0x33/0x80<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa063ac69>] ? gfs2_glock_nq_num+0x59/0xa0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8106335b>] ? enqueue_task_fair+0xfb/0x100<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81108093>] slow_work_execute+0x233/0x310<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff811082c7>] slow_work_thread+0x157/0x360<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81108170>] ? slow_work_thread+0x0/0x360<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81091d66>] kthread+0x96/0xa0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81091cd0>] ? kthread+0x0/0xa0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20<o:p></o:p>Nov 26 03:42:41 host2 kernel: INFO: task gfs2_quotad:2950 blocked for more than 120 seconds.<o:p></o:p>Nov 26 03:42:41 host2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<o:p></o:p>Nov 26 03:42:41 host2 kernel: gfs2_quotadšš D 0000000000000001šššš 0š 2950ššššš 2 0x00000080<o:p></o:p>Nov 26 03:42:41 host2 kernel: ffff88086afdfc20 0000000000000046 0000000000000000 ffffffffa0605f4d<o:p></o:p>Nov 26 03:42:41 host2 kernel: 0000000000000000 ffff88106c505800 ffff88086afdfc50 ffffffffa0604708<o:p></o:p>Nov 26 03:42:41 host2 kernel: ffff88086afddaf8 ffff88086afdffd8 000000000000fb88 ffff88086afddaf8<o:p></o:p>Nov 26 03:42:41 host2 kernel: Call Trace:<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa0605f4d>] ? dlm_put_lockspace+0x1d/0x40 [dlm]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa0604708>] ? dlm_lock+0x98/0x1e0 [dlm]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa0637570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa063757e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff814feaaf>] __wait_on_bit+0x5f/0x90<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa0637570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff814feb58>] out_of_line_wait_on_bit+0x78/0x90<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa06394f5>] gfs2_glock_wait+0x45/0x90 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa063a8f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8107eabb>] ? try_to_del_timer_sync+0x7b/0xe0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa0653658>] gfs2_statfs_sync+0x58/0x1b0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff814fe75a>] ? schedule_timeout+0x19a/0x2e0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa0653650>] ? gfs2_statfs_sync+0x50/0x1b0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa064b9d7>] quotad_check_timeo+0x57/0xb0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa064bc64>] gfs2_quotad+0x234/0x2b0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffffa064ba30>] ? gfs2_quotad+0x0/0x2b0 [gfs2]<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81091d66>] kthread+0x96/0xa0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff81091cd0>] ? kthread+0x0/0xa0<o:p></o:p>Nov 26 03:42:41 host2 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20<o:p></o:p>Nov 26 03:42:54 host2 iscsid: connect to 192.168.1.161:3260 failed (No route to host)<o:p></o:p>Nov 26 03:43:00 host2 iscsid: connect to 192.168.1.161:3260 failed (No route to host)<o:p></o:p>Nov 26 03:43:01 host2 corosync[2596]:šš [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p>Nov 26 03:43:01 host2 corosync[2596]:šš [CMANš ] quorum regained, resuming activity<o:p></o:p>Nov 26 03:43:01 host2 corosync[2596]:šš [QUORUM] This node is within the primary component and will provide service.<o:p></o:p>Nov 26 03:43:01 host2 corosync[2596]:šš [QUORUM] Members[2]: 1 2<o:p></o:p>Nov 26 03:43:01 host2 corosync[2596]:šš [QUORUM] Members[2]: 1 2<o:p></o:p>Nov 26 03:43:01 host2 corosync[2596]:šš [CPGšš ] chosen downlist: sender r(0) ip(192.168.1.1) ; members(old:1 left:0)<o:p></o:p>Nov 26 03:43:01 host2 corosync[2596]:šš [MAINš ] Completed service synchronization, ready to provide service.<o:p></o:p>Nov 26 03:43:01 host2 gfs_controld[2727]: receive_start 1:8 add node with started_count 5<o:p></o:p>Nov 26 03:43:01 host2 fenced[2652]: receive_start 1:8 add node with started_count 2<o:p></o:p>Nov 26 03:43:03 host2 iscsid: connection1:0 is operational after recovery (5 attempts)<o:p></o:p>Nov 26 03:44:41 host2 kernel: INFO: task kslowd001:2942 blocked for more than 120 seconds.<o:p></o:p>Nov 26 03:44:41 host2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<o:p></o:p>Nov 26 03:44:41 host2 kernel: kslowd001šššš D 000000000000000bšššš 0š 2942ššššš 2 0x00000080<o:p></o:p>Nov 26 03:44:41 host2 kernel: ffff88086b29d958 0000000000000046 0000000000000102 0000005000000002<o:p></o:p>Nov 26 03:44:41 host2 kernel: fffffffffffffffc 000000000000010e 0000003f00000002 fffffffffffffffc<o:p></o:p>Nov 26 03:44:41 host2 kernel: ffff88086b29bab8 ffff88086b29dfd8 000000000000fb88 ffff88086b29bab8<o:p></o:p>Nov 26 03:44:41 host2 kernel: Call Trace:<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff814ffec5>] rwsem_down_failed_common+0x95/0x1d0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81500056>] rwsem_down_read_failed+0x26/0x30<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8127e634>] call_rwsem_down_read_failed+0x14/0x30<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff814ff554>] ? down_read+0x24/0x30<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa06046d2>] dlm_lock+0x62/0x1e0 [dlm]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8127cd04>] ? vsnprintf+0x484/0x5f0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa06564e1>] gdlm_lock+0xf1/0x130 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa06565f0>] ? gdlm_ast+0x0/0xe0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa0656520>] ? gdlm_bast+0x0/0x50 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa063a385>] do_xmote+0x1a5/0x280 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8127cf14>] ? snprintf+0x34/0x40<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa063a551>] run_queue+0xf1/0x1d0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa063a8de>] gfs2_glock_nq+0x21e/0x3d0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa063ac71>] gfs2_glock_nq_num+0x61/0xa0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa064eca3>] gfs2_recover_work+0x93/0x7b0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8105b483>] ? perf_event_task_sched_out+0x33/0x80<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff810096f0>] ? __switch_to+0xd0/0x320<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa063ac69>] ? gfs2_glock_nq_num+0x59/0xa0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8106335b>] ? enqueue_task_fair+0xfb/0x100<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81108093>] slow_work_execute+0x233/0x310<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff811082c7>] slow_work_thread+0x157/0x360<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81108170>] ? slow_work_thread+0x0/0x360<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81091d66>] kthread+0x96/0xa0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81091cd0>] ? kthread+0x0/0xa0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20<o:p></o:p>Nov 26 03:44:41 host2 kernel: INFO: task gfs2_quotad:2950 blocked for more than 120 seconds.<o:p></o:p>Nov 26 03:44:41 host2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.<o:p></o:p>Nov 26 03:44:41 host2 kernel: gfs2_quotadšš D 0000000000000001šššš 0š 2950ššššš 2 0x00000080<o:p></o:p>Nov 26 03:44:41 host2 kernel: ffff88086afdfc20 0000000000000046 0000000000000000 ffffffffa0605f4d<o:p></o:p>Nov 26 03:44:41 host2 kernel: 0000000000000000 ffff88106c505800 ffff88086afdfc50 ffffffffa0604708<o:p></o:p>Nov 26 03:44:41 host2 kernel: ffff88086afddaf8 ffff88086afdffd8 000000000000fb88 ffff88086afddaf8<o:p></o:p>Nov 26 03:44:41 host2 kernel: Call Trace:<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa0605f4d>] ? dlm_put_lockspace+0x1d/0x40 [dlm]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa0604708>] ? dlm_lock+0x98/0x1e0 [dlm]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa0637570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa063757e>] gfs2_glock_holder_wait+0xe/0x20 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff814feaaf>] __wait_on_bit+0x5f/0x90<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa0637570>] ? gfs2_glock_holder_wait+0x0/0x20 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff814feb58>] out_of_line_wait_on_bit+0x78/0x90<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81092110>] ? wake_bit_function+0x0/0x50<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa06394f5>] gfs2_glock_wait+0x45/0x90 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa063a8f7>] gfs2_glock_nq+0x237/0x3d0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8107eabb>] ? try_to_del_timer_sync+0x7b/0xe0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa0653658>] gfs2_statfs_sync+0x58/0x1b0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff814fe75a>] ? schedule_timeout+0x19a/0x2e0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa0653650>] ? gfs2_statfs_sync+0x50/0x1b0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa064b9d7>] quotad_check_timeo+0x57/0xb0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa064bc64>] gfs2_quotad+0x234/0x2b0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffffa064ba30>] ? gfs2_quotad+0x0/0x2b0 [gfs2]<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81091d66>] kthread+0x96/0xa0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff81091cd0>] ? kthread+0x0/0xa0<o:p></o:p>Nov 26 03:44:41 host2 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20<o:p></o:p><o:p> </o:p>What would you do in the same case? Is it possible to restart GFS2 without rebooting nodes?<o:p></o:p><o:p> </o:p>Thank you very much for any help.<o:p></o:p><o:p> </o:p>-- <o:p></o:p>V.Melnik<o:p></o:p></div></body></html>