[Linux-cluster] different subnets/ manual fencing

Mon Apr 24 17:18:30 UTC 2006

> Yes, but you'll need to configure it for multicas rather than broadcast -
> and make sure that any intervening routers are good enough.

That is good news. So we have the head node (dream) with two ethernet cards. 
We want it to serve a GFS partition to two different subnets. I guess this is 
than also doable with multicast, right?

> I'd like to see those please.
>
Thanks! I attached a text file (myoops.txt). I hope it helps. I can't tell, 
because I never really had to deal with kernel panics before...

Thanks again,

wolfgang

P.S.: Apr 23 16:15:10 node15 kernel: Linux version 2.6.15-1.1833_FC4smp 
(bhcompile at hs20-bc1-1.build.redhat.com) (gcc version 4.0.2 20051125 (Red Hat 
4.0.2-8)) #1 SMP Wed Mar 1 23:56:51 EST 2006
-------------- next part --------------
Apr 23 14:03:59 node15 ccsd[2367]: Starting ccsd 1.0.0:
Apr 23 14:03:59 node15 ccsd[2367]:  Built: Jun 16 2005 10:45:39
Apr 23 14:03:59 node15 ccsd[2367]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved.
Apr 23 14:04:00 node15 kernel: CMAN 2.6.11.5-20050601.152643.FC4.23 (built Mar  7 2006 15:36:41) installed
Apr 23 14:04:00 node15 kernel: NET: Registered protocol family 30
Apr 23 14:04:00 node15 ccsd[2367]: cluster.conf (cluster name = oreilly_cluster, version = 33) found.
Apr 23 14:04:03 node15 kernel: CMAN: Waiting to join or form a Linux-cluster
Apr 23 14:04:03 node15 ccsd[2367]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.2
Apr 23 14:04:03 node15 ccsd[2367]: Initial status:: Inquorate
Apr 23 14:04:32 node15 kernel: CMAN: sending membership request
Apr 23 14:04:53 node15 last message repeated 19 times
Apr 23 14:04:54 node15 kernel: CMAN: got node node27
Apr 23 14:04:54 node15 kernel: CMAN: got node node17
Apr 23 14:04:54 node15 kernel: CMAN: got node node16
Apr 23 14:04:54 node15 kernel: CMAN: got node node24
Apr 23 14:04:54 node15 kernel: CMAN: got node node1
Apr 23 14:04:54 node15 kernel: CMAN: got node node2
Apr 23 14:04:54 node15 kernel: CMAN: got node node23
Apr 23 14:04:54 node15 kernel: CMAN: got node node6
Apr 23 14:04:54 node15 kernel: CMAN: got node node10
Apr 23 14:04:54 node15 kernel: CMAN: got node node9
Apr 23 14:05:01 node15 kernel: CMAN: quorum regained, resuming activity
Apr 23 14:05:01 node15 ccsd[2367]: Cluster is quorate.  Allowing connections.
Apr 23 14:05:01 node15 kernel: dlm: no version for "struct_module" found: kernel tainted.
Apr 23 14:05:01 node15 kernel: DLM 2.6.11.5-20050601.152643.FC4.22 (built Mar  7 2006 15:42:37) installed
Apr 23 14:07:28 node15 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Apr 23 14:07:28 node15 kernel:  printing eip:
Apr 23 14:07:28 node15 kernel: f8acfa39
Apr 23 14:07:28 node15 kernel: *pde = 37d1f001
Apr 23 14:07:28 node15 kernel: Oops: 0000 [#1]
Apr 23 14:07:28 node15 kernel: SMP
Apr 23 14:07:28 node15 kernel: last sysfs file: /class/misc/dlm-control/dev
Apr 23 14:07:28 node15 kernel: Modules linked in: dlm(U) cman(U) ipv6 parport_pc lp parport autofs4 nfs lockd nfs_acl sunrpc dm_mod eepro100 uhci_hcd hw_ra
ndom i8xx_tco i2c_i801 i2c_core e1000 e100 mii floppy ext3 jbd
Apr 23 14:07:28 node15 kernel: CPU:    0
Apr 23 14:07:28 node15 kernel: EIP:    0060:[<f8acfa39>]    Tainted: GF     VLI
Apr 23 14:07:28 node15 kernel: EFLAGS: 00010207   (2.6.15-1.1833_FC4smp)
Apr 23 14:07:28 node15 kernel: EIP is at memcpy_fromkvec+0x2e/0x4f [cman]
Apr 23 14:07:28 node15 kernel: eax: 00000046   ebx: c1d7aeb7   ecx: 00000011   edx: f68b0fa0
Apr 23 14:07:28 node15 kernel: esi: 00000000   edi: c1d7aeb7   ebp: 00000046   esp: f68b0ec8
Apr 23 14:07:28 node15 kernel: ds: 007b   es: 007b   ss: 0068
Apr 23 14:07:28 node15 kernel: Process cman_comms (pid: 2392, threadinfo=f68b0000 task=f7d85550)
Apr 23 14:07:28 node15 kernel: Stack: badc0ded f6846380 00000000 f7386400 f68b0f74 f8acfbb3 00000100 00000002
Apr 23 14:07:28 node15 kernel:        00000040 f731e000 f7decb40 f731e001 00000001 00000001 f6cea440 f8ad002d
Apr 23 14:07:28 node15 kernel:        f68b0f90 00000001 000002fd c1b091e0 f68b0f90 f68b0f74 f7decb40 c1eb4940
Apr 23 14:07:28 node15 kernel: Call Trace:
Apr 23 14:07:28 node15 kernel:  [<f8acfbb3>] send_to_user_port+0x159/0x3cc [cman]     [<f8ad002d>] process_incoming_packet+0x207/0x26c [cman]
Apr 23 14:07:29 node15 kernel:  [<f8acf188>] receive_message+0xb7/0xe0 [cman]     [<f8acf33c>] cluster_kthread+0x18b/0x39f [cman]
Apr 23 14:07:29 node15 kernel:  [<c01205ee>] default_wake_function+0x0/0xc     [<f8acf1b1>] cluster_kthread+0x0/0x39f [cman]
Apr 23 14:07:29 node15 kernel:  [<c010243d>] kernel_thread_helper+0x5/0xb
Apr 23 14:07:29 node15 kernel: Code: 53 89 c3 89 cd 85 c9 7e 3e 83 c2 08 eb 07 83 c2 08 85 ed 7e 32 8b 42 fc 85 c0 74 f2 39 e8 0f 47 c5 89 c1 c1 e9 02 8b 7
2 f8 89 df <f3> a5 89 c1 83 e1 03 74 02 f3 a4 29 c5 01 c3 01 42 f8 29 42 fc
Apr 23 14:07:29 node15 kernel: Continuing in 120 seconds. ^MContinuing in 119 seconds. ^MContinuing in 118 seconds. ^MContinuing in 117 seconds. ^MContinui
ng in 116 seconds. ^MContinuing in 115 seconds. ^MContinuing in 114 seconds. ^MContinuing in 113 seconds. ^MContinuing in 112 seconds. ^MContinuing in 111
seconds. ^MContinuing in 110 seconds. ^MContinuing in 109 seconds. ^MContinuing in 108 seconds. ^MContinuing in 107 seconds. ^MContinuing in 106 seconds.
Continuing in 105 seconds. ^MContinuing in 104 seconds. ^MContinuing in 103 seconds. ^MContinuing in 102 seconds. ^MContinuing in 101 seconds. ^MContinuing
 in 100 seconds. ^M<4>hda: dma_timer_expiry: dma status == 0x24
Apr 23 14:07:29 node15 kernel: Continuing in 99 seconds. ^MContinuing in 98 seconds. ^MContinuing in 97 seconds. ^MContinuing in 96 seconds. ^MContinuing i
n 95 seconds. ^MContinuing in 94 seconds. ^MContinuing in 93 seconds. ^MContinuing in 92 seconds. ^MContinuing in 91 seconds. ^MContinuing in 90 seconds.
hda: DMA interrupt recovery
Apr 23 14:07:29 node15 kernel: hda: lost interrupt
Apr 23 14:07:29 node15 kernel: Continuing in 89 seconds. ^MContinuing in 88 seconds. ^MContinuing in 87 seconds. ^MContinuing in 86 seconds. ^MContinuing i
n 85 seconds. ^MContinuing in 84 seconds. ^MContinuing in 83 seconds. ^MContinuing in 82 seconds. ^MContinuing in 81 seconds. ^MContinuing in 80 seconds.
Continuing in 79 seconds. ^MContinuing in 78 seconds. ^MContinuing in 77 seconds. ^MContinuing in 76 seconds. ^MContinuing in 75 seconds. ^MContinuing in 7
4 seconds. ^MContinuing in 73 seconds. ^MContinuing in 72 seconds. ^MContinuing in 71 seconds. ^MContinuing in 70 seconds. ^MContinuing in 69 seconds. ^M<4
>hda: dma_timer_expiry: dma status == 0x24
Apr 23 14:07:29 node15 kernel: Continuing in 68 seconds. ^MContinuing in 67 seconds. ^MContinuing in 66 seconds. ^MContinuing in 65 seconds. ^MContinuing i
n 64 seconds. ^MContinuing in 63 seconds. ^MContinuing in 62 seconds. ^MContinuing in 61 seconds. ^M<6>ide-cd: cmd 0x3 timed out
Apr 23 14:07:29 node15 kernel: hdc: lost interrupt
Apr 23 14:07:29 node15 kernel: Continuing in 60 seconds. ^MContinuing in 59 seconds. ^Mhda: DMA interrupt recovery
Apr 23 14:07:29 node15 kernel: hda: lost interrupt
Apr 23 14:07:29 node15 kernel: Continuing in 58 seconds. ^MContinuing in 57 seconds. ^MContinuing in 56 seconds. ^MContinuing in 55 seconds. ^MContinuing i
n 54 seconds. ^MContinuing in 53 seconds. ^MContinuing in 52 seconds. ^MContinuing in 51 seconds. ^MContinuing in 50 seconds. ^MContinuing in 49 seconds.
Continuing in 48 seconds. ^MContinuing in 47 seconds. ^MContinuing in 46 seconds. ^MContinuing in 45 seconds. ^MContinuing in 44 seconds. ^MContinuing in 4
3 seconds. ^MContinuing in 42 seconds. ^MContinuing in 41 seconds. ^MContinuing in 40 seconds. ^MContinuing in 39 seconds. ^M<4>hda: dma_timer_expiry: dma
status == 0x24
Apr 23 14:07:29 node15 kernel: Continuing in 38 seconds. ^MContinuing in 37 seconds. ^MContinuing in 36 seconds. ^MContinuing in 35 seconds. ^MContinuing i
n 34 seconds. ^MContinuing in 33 seconds. ^MContinuing in 32 seconds. ^MContinuing in 31 seconds. ^MContinuing in 30 seconds. ^MContinuing in 29 seconds.
hda: DMA interrupt recovery
Apr 23 14:07:29 node15 kernel: hda: lost interrupt
Apr 23 14:07:29 node15 kernel: Continuing in 28 seconds. ^MContinuing in 27 seconds. ^MContinuing in 26 seconds. ^MContinuing in 25 seconds. ^MContinuing i
n 24 seconds. ^MContinuing in 23 seconds. ^MContinuing in 22 seconds. ^MContinuing in 21 seconds. ^MContinuing in 20 seconds. ^MContinuing in 19 seconds.
Continuing in 18 seconds. ^MContinuing in 17 seconds. ^MContinuing in 16 seconds. ^MContinuing in 15 seconds. ^MContinuing in 14 seconds. ^MContinuing in 1
3 seconds. ^MContinuing in 12 seconds. ^MContinuing in 11 seconds. ^MContinuing in 10 seconds. ^MContinuing in 9 seconds. ^M<4>hda: dma_timer_expiry: dma s
tatus == 0x24
Apr 23 14:07:29 node15 kernel: Continuing in 8 seconds. ^MContinuing in 7 seconds. ^MContinuing in 6 seconds. ^MContinuing in 5 seconds. ^MContinuing in 4
seconds. ^MContinuing in 3 seconds. ^MContinuing in 2 seconds. ^MContinuing in 1 seconds.
Apr 23 14:07:29 node15 kernel:  <0>Fatal exception: panic in 5 seconds

# -----------------------------------------------

Apr 23 16:07:04 node15 ccsd[2373]: Starting ccsd 1.0.0:
Apr 23 16:07:04 node15 ccsd[2373]:  Built: Jun 16 2005 10:45:39
Apr 23 16:07:04 node15 ccsd[2373]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved.
Apr 23 16:07:05 node15 kernel: CMAN 2.6.11.5-20050601.152643.FC4.23 (built Mar  7 2006 15:36:41) installed
Apr 23 16:07:05 node15 kernel: NET: Registered protocol family 30
Apr 23 16:07:05 node15 ccsd[2373]: cluster.conf (cluster name = oreilly_cluster, version = 33) found.
Apr 23 16:07:14 node15 kernel: CMAN: Waiting to join or form a Linux-cluster
Apr 23 16:07:14 node15 ccsd[2373]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.2
Apr 23 16:07:14 node15 ccsd[2373]: Initial status:: Inquorate
Apr 23 16:07:17 node15 kernel: CMAN: sending membership request
Apr 23 16:07:37 node15 last message repeated 27 times
Apr 23 16:07:38 node15 kernel: CMAN: got node node2
Apr 23 16:07:38 node15 kernel: CMAN: got node node26
Apr 23 16:07:38 node15 kernel: CMAN: got node node6
Apr 23 16:07:38 node15 kernel: CMAN: got node node27
Apr 23 16:07:38 node15 kernel: CMAN: got node node4
Apr 23 16:07:38 node15 kernel: CMAN: got node node5
Apr 23 16:07:38 node15 kernel: CMAN: got node node17
Apr 23 16:07:38 node15 kernel: CMAN: got node node3
Apr 23 16:07:38 node15 kernel: CMAN: got node node18
Apr 23 16:07:38 node15 kernel: CMAN: got node node16
Apr 23 16:07:38 node15 kernel: CMAN: got node node23
Apr 23 16:07:38 node15 kernel: CMAN: got node node12
Apr 23 16:07:38 node15 kernel: CMAN: got node node7
Apr 23 16:07:38 node15 ccsd[2373]: Cluster is quorate.  Allowing connections.
Apr 23 16:07:38 node15 kernel: CMAN: got node dream
Apr 23 16:07:38 node15 kernel: CMAN: got node node20
Apr 23 16:07:38 node15 kernel: CMAN: quorum regained, resuming activity
Apr 23 16:07:38 node15 kernel: dlm: no version for "struct_module" found: kernel tainted.
Apr 23 16:07:38 node15 kernel: DLM 2.6.11.5-20050601.152643.FC4.22 (built Mar  7 2006 15:42:37) installed
Apr 23 16:10:06 node15 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Apr 23 16:10:06 node15 kernel:  printing eip:
Apr 23 16:10:06 node15 kernel: f8a85a39
Apr 23 16:10:06 node15 kernel: *pde = 363b4001
Apr 23 16:10:06 node15 kernel: Oops: 0000 [#1]
Apr 23 16:10:06 node15 kernel: SMP
Apr 23 16:10:06 node15 kernel: last sysfs file: /class/misc/dlm-control/dev
Apr 23 16:10:06 node15 kernel: Modules linked in: dlm(U) cman(U) ipv6 parport_pc lp parport autofs4 sunrpc dm_mod eepro100 uhci_hcd hw_random i8xx_tco i2c_
i801 i2c_core e1000 e100 mii floppy ext3 jbd
Apr 23 16:10:06 node15 kernel: CPU:    0
Apr 23 16:10:06 node15 kernel: EIP:    0060:[<f8a85a39>]    Tainted: GF     VLI
Apr 23 16:10:06 node15 kernel: EFLAGS: 00010202   (2.6.15-1.1833_FC4smp)
Apr 23 16:10:06 node15 kernel: EIP is at memcpy_fromkvec+0x2e/0x4f [cman]
Apr 23 16:10:06 node15 kernel: eax: 00000040   ebx: c1db9eba   ecx: 00000010   edx: f6344fa0
Apr 23 16:10:06 node15 kernel: esi: 00000000   edi: c1db9eba   ebp: 00000040   esp: f6344ec8
Apr 23 16:10:06 node15 kernel: ds: 007b   es: 007b   ss: 0068
Apr 23 16:10:06 node15 kernel: Process cman_comms (pid: 2399, threadinfo=f6344000 task=c1e73aa0)
Apr 23 16:10:06 node15 kernel: Stack: badc0ded f63d5a80 00000000 f6000a00 f6344f74 f8a85bb3 00000100 00000002
Apr 23 16:10:06 node15 kernel:        00000040 f66a9800 f6319a40 f66a9801 00000001 00000001 f6383cc0 f8a8602d
Apr 23 16:10:06 node15 kernel:        f6344f90 00000001 000002fa c1b091e0 f6344f90 f6344f74 f6319a40 f7dc1b80
Apr 23 16:10:06 node15 kernel: Call Trace:
Apr 23 16:10:06 node15 kernel:  [<f8a85bb3>] send_to_user_port+0x159/0x3cc [cman]     [<f8a8602d>] process_incoming_packet+0x207/0x26c [cman]
Apr 23 16:10:06 node15 kernel:  [<f8a85188>] receive_message+0xb7/0xe0 [cman]     [<f8a8533c>] cluster_kthread+0x18b/0x39f [cman]
Apr 23 16:10:06 node15 kernel:  [<c01205ee>] default_wake_function+0x0/0xc     [<f8a851b1>] cluster_kthread+0x0/0x39f [cman]
Apr 23 16:10:06 node15 kernel:  [<c010243d>] kernel_thread_helper+0x5/0xb
Apr 23 16:10:06 node15 kernel: Code: 53 89 c3 89 cd 85 c9 7e 3e 83 c2 08 eb 07 83 c2 08 85 ed 7e 32 8b 42 fc 85 c0 74 f2 39 e8 0f 47 c5 89 c1 c1 e9 02 8b 7
2 f8 89 df <f3> a5 89 c1 83 e1 03 74 02 f3 a4 29 c5 01 c3 01 42 f8 29 42 fc
Apr 23 16:10:06 node15 kernel:  <0>Fatal exception: panic in 5 seconds

# -----------------------------------------------

Apr 23 18:05:33 node15 ccsd[3356]: Starting ccsd 1.0.0:
Apr 23 18:05:33 node15 ccsd[3356]:  Built: Jun 16 2005 10:45:39
Apr 23 18:05:33 node15 ccsd[3356]:  Copyright (C) Red Hat, Inc.  2004  All rights reserved.
Apr 23 18:05:34 node15 kernel: CMAN 2.6.11.5-20050601.152643.FC4.23 (built Mar  7 2006 15:36:41) installed
Apr 23 18:05:34 node15 kernel: NET: Registered protocol family 30
Apr 23 18:05:34 node15 ccsd[3356]: cluster.conf (cluster name = oreilly_cluster, version = 35) found.
Apr 23 18:05:35 node15 kernel: CMAN: Waiting to join or form a Linux-cluster
Apr 23 18:05:36 node15 ccsd[3356]: Connected to cluster infrastruture via: CMAN/SM Plugin v1.1.2
Apr 23 18:05:36 node15 ccsd[3356]: Initial status:: Inquorate
Apr 23 18:05:36 node15 kernel: CMAN: sending membership request
Apr 23 18:05:36 node15 kernel: CMAN: got node dream
Apr 23 18:06:13 node15 kernel: CMAN: quorum regained, resuming activity
Apr 23 18:06:13 node15 ccsd[3356]: Cluster is quorate.  Allowing connections.
Apr 23 18:06:13 node15 kernel: dlm: no version for "struct_module" found: kernel tainted.
Apr 23 18:06:13 node15 kernel: DLM 2.6.11.5-20050601.152643.FC4.22 (built Mar  7 2006 15:42:37) installed
Apr 23 18:06:23 node15 kernel: CMAN: node node1 rejoining
Apr 23 18:06:28 node15 last message repeated 3 times
Apr 23 18:06:32 node15 kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000000
Apr 23 18:06:32 node15 kernel:  printing eip:
Apr 23 18:06:32 node15 kernel: f8a85a39
Apr 23 18:06:32 node15 kernel: *pde = 37e89001
Apr 23 18:06:32 node15 kernel: Oops: 0000 [#1]
Apr 23 18:06:32 node15 kernel: SMP
Apr 23 18:06:32 node15 kernel: last sysfs file: /class/misc/dlm-control/dev
Apr 23 18:06:32 node15 kernel: Modules linked in: dlm(U) cman(U) nfs lockd nfs_acl ipv6 parport_pc lp parport autofs4 sunrpc dm_mod eepro100 uhci_hcd hw_ra
ndom i8xx_tco i2c_i801 i2c_core e1000 e100 mii floppy ext3 jbd
Apr 23 18:06:32 node15 kernel: CPU:    0
Apr 23 18:06:32 node15 kernel: EIP:    0060:[<f8a85a39>]    Tainted: GF     VLI
Apr 23 18:06:32 node15 kernel: EFLAGS: 00010203   (2.6.15-1.1833_FC4smp)
Apr 23 18:06:32 node15 kernel: EIP is at memcpy_fromkvec+0x2e/0x4f [cman]
Apr 23 18:06:32 node15 kernel: eax: 00000042   ebx: c1ffcab9   ecx: 00000010   edx: f5c22fa0
Apr 23 18:06:32 node15 kernel: esi: 00000000   edi: c1ffcab9   ebp: 00000042   esp: f5c22ec8
Apr 23 18:06:32 node15 kernel: ds: 007b   es: 007b   ss: 0068
Apr 23 18:06:32 node15 kernel: Process cman_comms (pid: 3387, threadinfo=f5c22000 task=c1e50000)
Apr 23 18:06:33 node15 kernel: Stack: badc0ded f7fd6d80 00000000 f6462000 f5c22f74 f8a85bb3 00000100 00000002
Apr 23 18:06:33 node15 kernel:        00000040 f66b1000 f5caf9c0 f66b1001 00000001 00000001 f5caf840 f8a8602d
Apr 23 18:06:33 node15 kernel:        f5c22f90 00000001 000002fb c1b091e0 f5c22f90 f5c22f74 f5caf9c0 c1e6d100
Apr 23 18:06:33 node15 kernel: Call Trace:
Apr 23 18:06:33 node15 kernel:  [<f8a85bb3>] send_to_user_port+0x159/0x3cc [cman]     [<f8a8602d>] process_incoming_packet+0x207/0x26c [cman]
Apr 23 18:06:33 node15 kernel:  [<f8a85188>] receive_message+0xb7/0xe0 [cman]     [<f8a8533c>] cluster_kthread+0x18b/0x39f [cman]
Apr 23 18:06:33 node15 kernel:  [<c01205ee>] default_wake_function+0x0/0xc     [<f8a851b1>] cluster_kthread+0x0/0x39f [cman]
Apr 23 18:06:33 node15 kernel:  [<c010243d>] kernel_thread_helper+0x5/0xb
Apr 23 18:06:33 node15 kernel: Code: 53 89 c3 89 cd 85 c9 7e 3e 83 c2 08 eb 07 83 c2 08 85 ed 7e 32 8b 42 fc 85 c0 74 f2 39 e8 0f 47 c5 89 c1 c1 e9 02 8b 7
2 f8 89 df <f3> a5 89 c1 83 e1 03 74 02 f3 a4 29 c5 01 c3 01 42 f8 29 42 fc
Apr 23 18:06:33 node15 kernel: Continuing in 120 seconds. ^MContinuing in 119 seconds. ^MContinuing in 118 seconds. ^MContinuing in 117 seconds. ^MContinui
ng in 116 seconds. ^MContinuing in 115 seconds. ^MContinuing in 114 seconds. ^MContinuing in 113 seconds. ^MContinuing in 112 seconds. ^MContinuing in 111
seconds. ^MContinuing in 110 seconds. ^MContinuing in 109 seconds. ^MContinuing in 108 seconds. ^MContinuing in 107 seconds. ^MContinuing in 106 seconds.
Continuing in 105 seconds. ^MContinuing in 104 seconds. ^MContinuing in 103 seconds. ^MContinuing in 102 seconds. ^MContinuing in 101 seconds. ^MContinuing
 in 100 seconds. ^MContinuing in 99 seconds. ^MContinuing in 98 seconds. ^MContinuing in 97 seconds. ^MContinuing in 96 seconds. ^MContinuing in 95 seconds
. ^MContinuing in 94 seconds. ^MContinuing in 93 seconds. ^MContinuing in 92 seconds. ^MContinuing in 91 seconds. ^MContinuing in 90 seconds. ^MContinuing
in 89 seconds. ^MContinuing in 88 seconds. ^MContinuing in 87 seconds. ^MContinuing in 86 seconds. ^MContinuing in 85 seconds. ^MCo
Apr 23 18:06:33 node15 kernel: tinuing in 84 seconds. ^MContinuing in 83 seconds. ^MContinuing in 82 seconds. ^MContinuing in 81 seconds. ^MContinuing in 8
ng in 116 seconds. ^MContinuing in 115 seconds. ^MContinuing in 114 seconds. ^MContinuing in 113 seconds. ^MContinuing in 112 seconds. ^MContinuing in 111
seconds. ^MContinuing in 110 seconds. ^MContinuing in 109 seconds. ^MContinuing in 108 seconds. ^MContinuing in 107 seconds. ^MContinuing in 106 seconds.
Continuing in 105 seconds. ^MContinuing in 104 seconds. ^MContinuing in 103 seconds. ^MContinuing in 102 seconds. ^MContinuing in 101 seconds. ^MContinuing
 in 100 seconds. ^MContinuing in 99 seconds. ^MContinuing in 98 seconds. ^MContinuing in 97 seconds. ^MContinuing in 96 seconds. ^MContinuing in 95 seconds
. ^MContinuing in 94 seconds. ^MContinuing in 93 seconds. ^MContinuing in 92 seconds. ^MContinuing in 91 seconds. ^MContinuing in 90 seconds. ^MContinuing
in 89 seconds. ^MContinuing in 88 seconds. ^MContinuing in 87 seconds. ^MContinuing in 86 seconds. ^MContinuing in 85 seconds. ^MCo
Apr 23 18:06:33 node15 kernel: tinuing in 84 seconds. ^MContinuing in 83 seconds. ^MContinuing in 82 seconds. ^MContinuing in 81 seconds. ^MContinuing in 8
0 seconds. ^MContinuing in 79 seconds. ^MContinuing in 78 seconds. ^MContinuing in 77 seconds. ^MContinuing in 76 seconds. ^MContinuing in 75 seconds. ^MCo
ntinuing in 74 seconds. ^MContinuing in 73 seconds. ^MContinuing in 72 seconds. ^MContinuing in 71 seconds. ^MContinuing in 70 seconds. ^MContinuing in 69
seconds. ^MContinuing in 68 seconds. ^MContinuing in 67 seconds. ^MContinuing in 66 seconds. ^MContinuing in 65 seconds. ^MContinuing in 64 seconds. ^MCont
inuing in 63 seconds. ^MContinuing in 62 seconds. ^MContinuing in 61 seconds. ^MContinuing in 60 seconds. ^MContinuing in 59 seconds. ^MContinuing in 58 se
conds. ^MContinuing in 57 seconds. ^MContinuing in 56 seconds. ^MContinuing in 55 seconds. ^MContinuing in 54 seconds. ^MContinuing in 53 seconds. ^MContin
uing in 52 seconds. ^MContinuing in 51 seconds. ^MContinuing in 50 seconds. ^MContinuing in 49 seconds. ^MContinuing in 48 seconds.
Apr 23 18:06:33 node15 kernel: tinuing in 47 seconds. ^MContinuing in 46 seconds. ^MContinuing in 45 seconds. ^MContinuing in 44 seconds. ^MContinuing in 4
3 seconds. ^MContinuing in 42 seconds. ^MContinuing in 41 seconds. ^MContinuing in 40 seconds. ^MContinuing in 39 seconds. ^MContinuing in 38 seconds. ^MCo
ntinuing in 37 seconds. ^MContinuing in 36 seconds. ^MContinuing in 35 seconds. ^MContinuing in 34 seconds. ^MContinuing in 33 seconds. ^MContinuing in 32
seconds. ^MContinuing in 31 seconds. ^MContinuing in 30 seconds. ^MContinuing in 29 seconds. ^MContinuing in 28 seconds. ^MContinuing in 27 seconds. ^MCont
inuing in 26 seconds. ^MContinuing in 25 seconds. ^MContinuing in 24 seconds. ^MContinuing in 23 seconds. ^MContinuing in 22 seconds. ^MContinuing in 21 se
conds. ^MContinuing in 20 seconds. ^MContinuing in 19 seconds. ^MContinuing in 18 seconds. ^MContinuing in 17 seconds. ^MContinuing in 16 seconds. ^MContin
uing in 15 seconds. ^MContinuing in 14 seconds. ^MContinuing in 13 seconds. ^MContinuing in 12 seconds. ^MContinuing in 11 seconds.
Apr 23 18:06:33 node15 kernel: tinuing in 10 seconds. ^MContinuing in 9 seconds. ^MContinuing in 8 seconds. ^MContinuing in 7 seconds. ^MContinuing in 6 se
conds. ^MContinuing in 5 seconds. ^MContinuing in 4 seconds. ^MContinuing in 3 seconds. ^MContinuing in 2 seconds. ^MContinuing in 1 seconds.
Apr 23 18:06:33 node15 kernel:  <0>Fatal exception: panic in 5 seconds