[Linux-cluster] kernel panic - help!

[David Teigland]

On Wed, Jun 21, 2006 at 03:41:58PM -0300, German Staltari wrote:
> Jun 21 14:59:17 qmail-be-04 kernel: CMAN: removing node qmail-be-02 from 
> the cluster : Missed too many heartbeats
> Jun 21 14:59:23 qmail-be-04 kernel: CMAN: removing node qmail-be-01 from 
> the cluster : No response to messages
> Jun 21 14:59:29 qmail-be-04 kernel: CMAN: removing node qmail-be-06 from 
> the cluster : No response to messages
> Jun 21 14:59:39 qmail-be-04 kernel: CMAN: removing node qmail-be-03 from 
> the cluster : No response to messages
> Jun 21 14:59:46 qmail-be-04 kernel: CMAN: removing node qmail-be-05 from 
> the cluster : No response to messages
> Jun 21 14:59:52 qmail-be-04 kernel: CMAN: quorum lost, blocking activity
> Jun 21 14:59:52 qmail-be-04 kernel: CMAN: node qmail-be-04 has been 
> removed from the cluster : No response to messages
> Jun 21 14:59:52 qmail-be-04 kernel: CMAN: killed by NODEDOWN message
> Jun 21 14:59:52 qmail-be-04 kernel: CMAN: we are leaving the cluster. No 
> response to messages

This is what led to the gfs panic, the cluster shut down when it lost
contact with all the other nodes.

Dave


> Jun 21 14:59:52 qmail-be-04 kernel: WARNING: dlm_emergency_shutdown
> Jun 21 14:59:52 qmail-be-04 fenced[17897]: process_events: service get 
> event failed
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 1000041 from 3 req 1
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 3
> Jun 21 14:59:53 qmail-be-04 last message repeated 7 times
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003d from 3 req 3
> Jun 21 14:59:53 qmail-be-04 last message repeated 6 times
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 1000041 from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 3
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 1
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 1000041 from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 1000041 from 3 req 1
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 1
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 1
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003b from 3 req 5
> Jun 21 14:59:53 qmail-be-04 last message repeated 5 times
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003d from 3 req 5
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003f from 3 req 5
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003d from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003d from 3 req 5
> Jun 21 14:59:53 qmail-be-04 last message repeated 3 times
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003d from 3 req 9
> Jun 21 14:59:53 qmail-be-04 kernel: dlm: process_cluster_request invalid 
> lockspace 100003d from 3 req 5
> Jun 21 14:59:54 qmail-be-04 last message repeated 20 times
> Jun 21 14:59:54 qmail-be-04 kernel: dlm: dlm_unlock: lkid 3b013d 
> lockspace not found
> Jun 21 14:59:54 qmail-be-04 kernel: store004-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore001-001 add_to_requestq cmd 3 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore003-004 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore002-004 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore002-002 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore001-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore002-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore003-002 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-004 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore004-001 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel: mstore003-003 add_to_requestq cmd 5 fr 3
> Jun 21 14:59:54 qmail-be-04 kernel:  type 2 event 282 flags 21a
> Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_start 282 done 1
> Jun 21 14:59:54 qmail-be-04 kernel: 28975 pr_finish flags 1a
> Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start last_stop 273 
> last_start 283 last_finish 273
> Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start count 5 type 2 event 
> 283 flags 21a
> Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_start 283 done 1
> Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
> Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start last_stop 283 
> last_start 285 last_finish 283
> Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start count 6 type 2 event 
> 285 flags 21a
> Jun 21 14:59:54 qmail-be-04 kernel: 28957 pr_start 285 done 1
> Jun 21 14:59:54 qmail-be-04 kernel: 28956 pr_finish flags 1a
> Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start last_stop 116 
> last_start 287 last_finish 116
> Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start count 4 type 2 event 
> 287 flags 21a
> Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_start 287 done 1
> Jun 21 14:59:54 qmail-be-04 kernel: 28992 pr_finish flags 1a
> Jun 21 14:59:55 qmail-be-04 kernel: 28975 rereq 2,1cec36 id e029f 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start last_stop 282 
> last_start 289 last_finish 282
> Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start count 5 type 2 event 
> 289 flags 21a
> Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_start 289 done 1
> Jun 21 14:59:55 qmail-be-04 kernel: 28975 pr_finish flags 1a
> Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start last_stop 287 
> last_start 291 last_finish 287
> Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start count 5 type 2 event 
> 291 flags 21a
> Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_start 291 done 1
> Jun 21 14:59:55 qmail-be-04 kernel: 28992 pr_finish flags 1a
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fe4b id a001e 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,3fd13 id 7007a 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faaf id 90009 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fdd6 id c0135 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,4fc8f id c023b 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,34a id 8011d 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,2fe4b id b03c3 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faaf id b001e 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,34a id 11000e 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,6faa8 id f0016 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,6faa8 id 1001a8 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fbd9 id f00e9 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,5fb4d id 802ac 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,5fbd9 id f0026 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 5,3fd13 id c009b 3,0
> Jun 21 14:59:55 qmail-be-04 kernel: 28974 rereq 2,2fdd6 id 8001d 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,4fc8f id c0367 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28974 rereq 2,2fe40 id a01fd 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start last_stop 289 
> last_start 293 last_finish 289
> Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start count 6 type 2 event 
> 293 flags 21a
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,7fa97 id 1502b3 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,3fcea id 702f3 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,5fc1e id 6015f 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fc1e id c01c1 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,2fdfa id c0362 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,2fdfa id c02ad 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,5fb4d id f01ce 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 2,7fa97 id d0293 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28975 rereq 5,3fcea id d02ad 3,0
> Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_start 293 done 1
> Jun 21 14:59:56 qmail-be-04 kernel: 28974 pr_finish flags 1a
> Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 118 
> last_start 295 last_finish 118
> Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start count 4 type 2 event 
> 295 flags 21a
> Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start 295 done 1
> Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_finish flags 1a
> Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start last_stop 291 
> last_start 297 last_finish 291
> Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start count 6 type 2 event 
> 297 flags 21a
> Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_start 297 done 1
> Jun 21 14:59:56 qmail-be-04 kernel: 28991 pr_finish flags 1a
> Jun 21 14:59:56 qmail-be-04 kernel: 29174 pr_start last_stop 295 
> last_start 299 last_finish 295
> Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start count 5 type 2 event 
> 299 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_start 299 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29174 pr_finish flags 1a
> Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start last_stop 299 
> last_start 301 last_finish 299
> Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start count 6 type 2 event 
> 301 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_start 301 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29175 pr_finish flags 1a
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 120 
> last_start 303 last_finish 120
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 4 type 2 event 
> 303 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 303 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 122 
> last_start 305 last_finish 122
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 4 type 2 event 
> 305 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 305 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
> Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start last_stop 124 
> last_start 308 last_finish 124
> Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start count 4 type 2 event 
> 308 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29458 pr_start 308 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29457 pr_finish flags 1a
> Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start last_stop 303 
> last_start 309 last_finish 303
> Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start count 5 type 2 event 
> 309 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29191 pr_start 309 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start last_stop 305 
> last_start 311 last_finish 305
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start count 5 type 2 event 
> 311 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_start 311 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29408 pr_finish flags 1a
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start last_stop 309 
> last_start 313 last_finish 309
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start count 6 type 2 event 
> 313 flags 21a
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_start 313 done 1
> Jun 21 14:59:57 qmail-be-04 kernel: 29192 pr_finish flags 1a
> Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 308 
> last_start 315 last_finish 308
> Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 5 type 2 event 
> 315 flags 21a
> Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 315 done 1
> Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_finish flags 1a
> Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start last_stop 311 
> last_start 317 last_finish 311
> Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start count 6 type 2 event 
> 317 flags 21a
> Jun 21 14:59:58 qmail-be-04 kernel: 29409 pr_start 317 done 1
> Jun 21 14:59:58 qmail-be-04 kernel: 29408 pr_finish flags 1a
> Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start last_stop 315 
> last_start 319 last_finish 315
> Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start count 6 type 2 event 
> 319 flags 21a
> Jun 21 14:59:58 qmail-be-04 kernel: 29457 rereq 2,2bd7b5 id 801e5 3,0
> Jun 21 14:59:58 qmail-be-04 kernel: 29458 pr_start 319 done 1
> Jun 21 14:59:58 qmail-be-04 kernel: 29457 pr_finish flags 1a
> Jun 21 14:59:58 qmail-be-04 kernel:
> Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  Assertion failed on line 
> 357 of file /soft/kernel/cluster-1.02.00/gfs-kernel/src/dlm/lock.c
> Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  assertion:  "!error"
> Jun 21 14:59:58 qmail-be-04 kernel: lock_dlm:  time = 2512697
> Jun 21 14:59:58 qmail-be-04 kernel: mstore008-002: error=-22 
> num=2,7798a8 lkf=10001 flags=84
> 
> Is the second panic today :(
> Thanks again
> German