[Linux-cluster] kernel panic - help!

[German Staltari]

David Teigland wrote:
> Any messages before this?  The best you could hope for with a SAN failure
> is that all the cluster nodes withdraw gfs, allowing you to reboot them
> without the panic.  So, the end result wouldn't be all that different than
> the panics.
>   
>
This is the log just before the panic:

Jun 21 14:59:17 qmail-be-04 kernel: CMAN: removing node qmail-be-02 from 
the cluster : Missed too many heartbeats
Jun 21 14:59:23 qmail-be-04 kernel: CMAN: removing node qmail-be-01 from 
the cluster : No response to messages
Jun 21 14:59:29 qmail-be-04 kernel: CMAN: removing node qmail-be-06 from 
the cluster : No response to messages
Jun 21 14:59:39 qmail-be-04 kernel: CMAN: removing node qmail-be-03 from 
the cluster : No response to messages
Jun 21 14:59:46 qmail-be-04 kernel: CMAN: removing node qmail-be-05 from 
the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: quorum lost, blocking activity
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: node qmail-be-04 has been 
removed from the cluster : No response to messages
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: killed by NODEDOWN message
Jun 21 14:59:52 qmail-be-04 kernel: CMAN: we are leaving the cluster. No 
response to messages
Jun 21 14:59:52 qmail-be-04 kernel: WARNING: dlm_emergency_shutdown
Jun 21 14:59:52 qmail-be-04 fenced[17897]: process_events: service get 
event failed
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 1000041 from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 3
Jun 21 14:59:53 qmail-be-04 last message repeated 7 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003d from 3 req 3
Jun 21 14:59:53 qmail-be-04 last message repeated 6 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 1000041 from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 3
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 1000041 from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 1000041 from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 1
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003b from 3 req 5
Jun 21 14:59:53 qmail-be-04 last message repeated 5 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003d from 3 req 5
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003f from 3 req 5
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003d from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003d from 3 req 5
Jun 21 14:59:53 qmail-be-04 last message repeated 3 times
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003d from 3 req 9
Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
lockspace 100003d from 3 req 5
Jun 21 14:59:54 qmail-be-04 last message repeated 20 times
Jun 21 14:59:54 qmail-be-04 kernel: dlm: dlm_unlock: lkid 3b013d 
lockspace not found
Jun 21 14:59:54 qmail-be-04 kernel: store004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-001 add_to_requestq cmd 3 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-002 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
Jun 21 14:59:54 qmail-be-04 kernel:  type 2 event 282 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_start 282 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start last_stop 273 
last_start 283 last_finish 273
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start count 5 type 2 event 
283 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start 283 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start last_stop 283 
last_start 285 last_finish 283
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start count 6 type 2 event 
285 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start 285 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start last_stop 116 
last_start 287 last_finish 116
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start count 4 type 2 event 
287 flags 21a
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start 287 done 1
Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28975 rereq 2,1cec36 id e029f 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start last_stop 282 
last_start 289 last_finish 282
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start count 5 type 2 event 
289 flags 21a
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start 289 done 1
Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start last_stop 287 
last_start 291 last_finish 287
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start count 5 type 2 event 
291 flags 21a
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start 291 done 1
Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_finish flags 1a
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fe4b id a001e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,3fd13 id 7007a 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faaf id 90009 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fdd6 id c0135 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,4fc8f id c023b 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,34a id 8011d 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fe4b id b03c3 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faaf id b001e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,34a id 11000e 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faa8 id f0016 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faa8 id 1001a8 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fbd9 id f00e9 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fb4d id 802ac 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,5fbd9 id f0026 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,3fd13 id c009b 3,0
Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fdd6 id 8001d 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,4fc8f id c0367 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,2fe40 id a01fd 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start last_stop 289 
last_start 293 last_finish 289
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start count 6 type 2 event 
293 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,7fa97 id 1502b3 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,3fcea id 702f3 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,5fc1e id 6015f 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fc1e id c01c1 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,2fdfa id c0362 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,2fdfa id c02ad 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fb4d id f01ce 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,7fa97 id d0293 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,3fcea id d02ad 3,0
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start 293 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 118 
last_start 295 last_finish 118
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start count 4 type 2 event 
295 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start 295 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start last_stop 291 
last_start 297 last_finish 291
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start count 6 type 2 event 
297 flags 21a
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start 297 done 1
Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_finish flags 1a
Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 295 
last_start 299 last_finish 295
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start count 5 type 2 event 
299 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start 299 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start last_stop 299 
last_start 301 last_finish 299
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start count 6 type 2 event 
301 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start 301 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 120 
last_start 303 last_finish 120
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 4 type 2 event 
303 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 303 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 122 
last_start 305 last_finish 122
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 4 type 2 event 
305 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 305 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start last_stop 124 
last_start 308 last_finish 124
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start count 4 type 2 event 
308 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start 308 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29457 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start last_stop 303 
last_start 309 last_finish 303
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start count 5 type 2 event 
309 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start 309 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 305 
last_start 311 last_finish 305
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 5 type 2 event 
311 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 311 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 309 
last_start 313 last_finish 309
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 6 type 2 event 
313 flags 21a
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 313 done 1
Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 308 
last_start 315 last_finish 308
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 5 type 2 event 
315 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 315 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start last_stop 311 
last_start 317 last_finish 311
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start count 6 type 2 event 
317 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start 317 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29408 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 315 
last_start 319 last_finish 315
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 6 type 2 event 
319 flags 21a
Jun 21 14:59:58 qmail-be-04 kernel: 29457 rereq 2,2bd7b5 id 801e5 3,0
Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 319 done 1
Jun 21 14:59:58 qmail-be-04 kernel: 29457 pr_finish flags 1a
Jun 21 14:59:58 qmail-be-04 kernel:
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  Assertion failed on line 
357 of file /soft/kernel/cluster-1.02.00/gfs-kernel/src/dlm/lock.c
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  assertion:  "!error"
Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  time = 2512697
Jun 21 14:59:58 qmail-be-04 kernel: mstore008-002: error=-22 
num=2,7798a8 lkf=10001 flags=84

Is the second panic today :(
Thanks again
German