[Linux-cluster] what does this mean?

[Adam Cassar]

(against latest gfs in cvs)

scenario:

- 3 machines in cluster
- one importing gnbd, two directly mounted to shared fc raid

* all 3 performing io
* reboot one machine, all of the machines hung on io

attempted to get rebooted machine to join the cluster, one machine spat
out the following:


kernel BUG at /usr/src/GFS/cluster/cman-kernel/src/membership.c:611!
invalid operand: 0000 [#1]
SMP 
Modules linked in: gnbd gfs lock_dlm dlm cman lock_harness 8250
serial_core dm_mod
CPU:    0
EIP:    0060:[<f88efc18>]    Not tainted
EFLAGS: 00010246   (2.6.8.1) 
EIP is at send_joinconf+0x10/0x75 [cman]
eax: 00000000   ebx: 00000003   ecx: 018a60be   edx: c180e08c
esi: f7ff6fc0   edi: 00000000   ebp: 00000001   esp: f324fe74
ds: 007b   es: 007b   ss: 0068
Process cman_memb (pid: 822, threadinfo=f324e000 task=f3242c70)
Stack: f8907380 00000000 00000003 f7ff6fc0 00000000 00000003 f88f15ec
f8907380 
       018a7446 c01193bf c0331140 c0401a8d f7ff6fc0 0000824d 0000824d
0000824d 
       00000000 f7ff6fc0 00000000 00000007 f88f087f 00000000 00000000
00000038 
Call Trace:
 [<f88f15ec>] do_process_startack+0x14d/0x38e [cman]
 [<c01193bf>] __call_console_drivers+0x55/0x57
 [<f88f087f>] start_transition+0x20f/0x2c1 [cman]
 [<f88ccee1>] cman_callback+0x35/0x38 [dlm]
 [<f88e952a>] notify_kernel_listeners+0x41/0x68 [cman]
 [<f88f0a94>] a_node_just_died+0x163/0x181 [cman]
 [<f88f2a5c>] do_process_leave+0x6b/0x7d [cman]
 [<f88f043c>] do_membership_packet+0x98/0x1f0 [cman]
 [<f88f2e9c>] dispatch_messages+0xe3/0x104 [cman]
 [<f88ef492>] membership_kthread+0x216/0x3e6 [cman]
 [<c0103d22>] ret_from_fork+0x6/0x14
 [<c01157c8>] default_wake_function+0x0/0x12
 [<f88ef27c>] membership_kthread+0x0/0x3e6 [cman]
 [<c0102161>] kernel_thread_helper+0x5/0xb
Code: 0f 0b 63 02 40 bd 8f f8 89 44 24 10 c7 05 f0 73 90 f8 02 00 
 
kernel BUG at /usr/src/GFS/cluster/cman-kernel/src/membership.c:611!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<f88efc18>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010246   (2.6.8.1) 
eax: 00000000   ebx: 00000003   ecx: 018a60be   edx: c180e08c
esi: f7ff6fc0   edi: 00000000   ebp: 00000001   esp: f324fe74
ds: 007b   es: 007b   ss: 0068
Stack: f8907380 00000000 00000003 f7ff6fc0 00000000 00000003 f88f15ec
f8907380 
       018a7446 c01193bf c0331140 c0401a8d f7ff6fc0 0000824d 0000824d
0000824d 
       00000000 f7ff6fc0 00000000 00000007 f88f087f 00000000 00000000
00000038 
 [<f88f15ec>] do_process_startack+0x14d/0x38e [cman]
 [<c01193bf>] __call_console_drivers+0x55/0x57
 [<f88f087f>] start_transition+0x20f/0x2c1 [cman]
 [<f88ccee1>] cman_callback+0x35/0x38 [dlm]
 [<f88e952a>] notify_kernel_listeners+0x41/0x68 [cman]
 [<f88f0a94>] a_node_just_died+0x163/0x181 [cman]
 [<f88f2a5c>] do_process_leave+0x6b/0x7d [cman]
 [<f88f043c>] do_membership_packet+0x98/0x1f0 [cman]
 [<f88f2e9c>] dispatch_messages+0xe3/0x104 [cman]
 [<f88ef492>] membership_kthread+0x216/0x3e6 [cman]
 [<c0103d22>] ret_from_fork+0x6/0x14
 [<c01157c8>] default_wake_function+0x0/0x12
 [<f88ef27c>] membership_kthread+0x0/0x3e6 [cman]
 [<c0102161>] kernel_thread_helper+0x5/0xb
Code: 0f 0b 63 02 40 bd 8f f8 89 44 24 10 c7 05 f0 73 90 f8 02 00 


>>EIP; f88efc18 <pg0+384c4c18/3fbd3000>   <=====

>>ecx; 018a60be Before first symbol
>>edx; c180e08c <pg0+13e308c/3fbd3000>
>>esi; f7ff6fc0 <pg0+37bcbfc0/3fbd3000>
>>esp; f324fe74 <pg0+32e24e74/3fbd3000>

Code;  f88efc18 <pg0+384c4c18/3fbd3000>
00000000 <_EIP>:
Code;  f88efc18 <pg0+384c4c18/3fbd3000>   <=====
   0:   0f 0b                     ud2a      <=====
Code;  f88efc1a <pg0+384c4c1a/3fbd3000>
   2:   63 02                     arpl   %ax,(%edx)
Code;  f88efc1c <pg0+384c4c1c/3fbd3000>
   4:   40                        inc    %eax
Code;  f88efc1d <pg0+384c4c1d/3fbd3000>
   5:   bd 8f f8 89 44            mov    $0x4489f88f,%ebp
Code;  f88efc22 <pg0+384c4c22/3fbd3000>
   a:   24 10                     and    $0x10,%al
Code;  f88efc24 <pg0+384c4c24/3fbd3000>
   c:   c7 05 f0 73 90 f8 02      movl   $0x2,0xf89073f0
Code;  f88efc2b <pg0+384c4c2b/3fbd3000>
  13:   00 00 00