[Crash-utility] crash-6.0.2 won't do module source lines on our kernel

[Bob Montgomery]

On Thu, 2012-01-26 at 15:24 -0500, Dave Anderson wrote:
> 
> ----- Original Message -----
> > 
> > 
> > ----- Original Message -----
> > > I've reverted back to crash-5.1.9 and applied my kmem patch to that for
> > > our use here.  We're using a 3.1-based kernel, and need the kmem patch
> > > so crash can deal with the change in CONFIG_SLAB, but we're building
> > > with gcc-4.4.5 and don't really need the new gdb in crash-6.0.2, and
> > > crash-6.0.2 is not giving module source line numbers for us with "dis -l".
> > > 
> > > This is just a heads up.  I don't know why 6.0.2 is failing this, and
> > > since I found the last module source line number problem, it's not my
> > > turn ;-)
> > > 
> > > Bob Montgomery
> >  
> > What kernel version?  I'll try to reproduce it.

Here's your test on my system:

crash-6.0.2> help | grep version
crash-6.0.2 version: 6.0.2    gdb version: 7.3.1

crash-6.0.2> help -k | grep proc_version
  proc_version: Linux version 3.1.4-clim-2-amd64 (buildd at hpdebuild) (gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011

crash-6.0.2> mod -s bnx2 /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
     MODULE       NAME                  SIZE  OBJECT FILE
ffffffffa01f85e0  bnx2                 65655  /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko 

/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6265
0xffffffffa01f32e1 <bnx2_open>: push   %rbp
0xffffffffa01f32e2 <bnx2_open+1>:       mov    %rsp,%rbp
0xffffffffa01f32e5 <bnx2_open+4>:       push   %r15
0xffffffffa01f32e7 <bnx2_open+6>:       push   %r14
0xffffffffa01f32e9 <bnx2_open+8>:       push   %r13
0xffffffffa01f32eb <bnx2_open+10>:      push   %r12
0xffffffffa01f32ed <bnx2_open+12>:      mov    %rdi,%r12
0xffffffffa01f32f0 <bnx2_open+15>:      push   %rbx
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6266
0xffffffffa01f32f1 <bnx2_open+16>:      lea    0x640(%rdi),%rbx
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6265
0xffffffffa01f32f8 <bnx2_open+23>:      sub    $0x8,%rsp
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c: 6269

Ummmm.  Ooops? Is it time to prepare my embarrassing explanation of my
prior claim???

=====================

First start over and do what I actually was doing when I saw the
problem:


$ crash-6.0.2 --no_kmem_cache dump.201201062015 kernel_link
...

crash-6.0.2> help | grep version
crash-6.0.2 version: 6.0.2    gdb version: 7.3.1

crash> help -k | grep proc_version
  proc_version: Linux version 3.1.4-clim-2-amd64 (buildd at hpdebuild) (gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011

crash-6.0.2> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
...

crash-6.0.2> mod | grep sunrpc
ffffffffa01cab80  auth_rpcgss
40572  /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 
ffffffffa03a7070  sunrpc
202679  /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/sunrpc.ko 

crash-6.0.2> dis -l rpc_sleep_on_priority
0xffffffffa038f71b <rpc_sleep_on_priority>:     push   %rbp
0xffffffffa038f71c <rpc_sleep_on_priority+1>:   mov    %rsp,%rbp
0xffffffffa038f71f <rpc_sleep_on_priority+4>:   push   %r12
0xffffffffa038f721 <rpc_sleep_on_priority+6>:   mov    %ecx,%r12d
0xffffffffa038f724 <rpc_sleep_on_priority+9>:   push   %rbx
0xffffffffa038f725 <rpc_sleep_on_priority+10>:  mov    %rdi,%rbx
0xffffffffa038f728 <rpc_sleep_on_priority+13>:  sub    $0x10,%rsp
0xffffffffa038f72c <rpc_sleep_on_priority+17>:  mov    0x70(%rsi),%rax
0xffffffffa038f730 <rpc_sleep_on_priority+21>:  test   $0x4,%al
0xffffffffa038f732 <rpc_sleep_on_priority+23>:  jne
0xffffffffa038f738
0xffffffffa038f734 <rpc_sleep_on_priority+25>:  ud2    

Well, that's a relief...

Oh, by the way, now that I've used mod -S to load all modules instead of
individually loading bnx2.ko:

crash-6.0.2> mod | grep bnx2
ffffffffa01f85e0  bnx2                 65655  /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko 


crash-6.0.2> dis -l bnx2_open
0xffffffffa01f32e1 <bnx2_open>: push   %rbp
0xffffffffa01f32e2 <bnx2_open+1>:       mov    %rsp,%rbp
0xffffffffa01f32e5 <bnx2_open+4>:       push   %r15
0xffffffffa01f32e7 <bnx2_open+6>:       push   %r14
0xffffffffa01f32e9 <bnx2_open+8>:       push   %r13
0xffffffffa01f32eb <bnx2_open+10>:      push   %r12
0xffffffffa01f32ed <bnx2_open+12>:      mov    %rdi,%r12
0xffffffffa01f32f0 <bnx2_open+15>:      push   %rbx
0xffffffffa01f32f1 <bnx2_open+16>:      lea    0x640(%rdi),%rbx
0xffffffffa01f32f8 <bnx2_open+23>:      sub    $0x8,%rsp
0xffffffffa01f32fc <bnx2_open+27>:      callq  0xffffffff8129477b <netif_carrier_off>

No source lines.

===================

If I do the same steps on my 5.1.9 version of crash, I get:

crash> help | grep version
crash version: 5.1.9    gdb version: 7.0

crash> help -k | grep proc_version
  proc_version: Linux version 3.1.4-clim-2-amd64 (buildd at hpdebuild) (gcc
version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011

crash> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
...

crash> dis -l rpc_sleep_on_priority
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c: 350
0xffffffffa038f71b <rpc_sleep_on_priority>:     push   %rbp
0xffffffffa038f71c <rpc_sleep_on_priority+1>:   mov    %rsp,%rbp
0xffffffffa038f71f <rpc_sleep_on_priority+4>:   push   %r12
0xffffffffa038f721 <rpc_sleep_on_priority+6>:   mov    %ecx,%r12d
0xffffffffa038f724 <rpc_sleep_on_priority+9>:   push   %rbx
0xffffffffa038f725 <rpc_sleep_on_priority+10>:  mov    %rdi,%rbx
0xffffffffa038f728 <rpc_sleep_on_priority+13>:  sub    $0x10,%rsp
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/arch/x86/include/asm/bitops.h: 312
0xffffffffa038f72c <rpc_sleep_on_priority+17>:  mov    0x70(%rsi),%rax
/build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c: 352
0xffffffffa038f730 <rpc_sleep_on_priority+21>:  test   $0x4,%al
0xffffffffa038f732 <rpc_sleep_on_priority+23>:  jne
0xffffffffa038f738 <rpc_sleep_on_priority+29>
0xffffffffa038f734 <rpc_sleep_on_priority+25>:  ud2a   

So 5.1.9 works, and 6.0.2 fails if I load all modules with mod -S.
That's interesting.

Bob Montgomery