[Crash-utility] crash-6.0.2 won't do module source lines on our kernel
Dave Anderson
anderson at redhat.com
Tue Jan 31 14:16:29 UTC 2012
Hi Bob,
OK, I'm currently provisioning a fresh Fedora 16 system and
I'll retry it.
I wonder if it has something to do with this crash-6.0.1 update:
- If the "--mod <directory-tree>" command line option, or the
setting of the CRASH_MODULE_PATH environment variable, or the
"mod -S <directory-tree>" point to a tree that contains only the
separate debuginfo "<module>.ko.debug" files, then those
debuginfo files will be used as the internal "add-symbol-file"
arguments to the embedded gdb module. Without the patch, it was
only acceptable to point to a directory tree that contained the
base "<module>.ko" files, and the separate debuginfo files
were found automatically based upon the directory path to the
base module file. This will allow an alternate module-debuginfo
directory tree to be set up like so:
# cd <directory>
# rpm2cpio kernel-debuginfo-<release>.rpm | cpio -idv
Having done that, the <directory> may be used with the "--mod",
command line argument, or as the CRASH_MODULE_PATH environment
variable, or as the "mod -S <directory> argument.
(anderson at redhat.com)
I'll check it myself, but I wonder what happens if you pass "mod -S"
the directory tree containing the <module>.ko.debug files instead of
the base <module>.ko files? It seems to be acting as if line between
the base <module>.ko and its <module>.ko.debug is not being recognized.
But then again, why would it work when individually loaded?
BTW, how's the kmem patch looking? I'm guessing it's more involved
than you first thought?
Thanks,
Dave
----- Original Message -----
> On Thu, 2012-01-26 at 15:24 -0500, Dave Anderson wrote:
> >
> > ----- Original Message -----
> > >
> > >
> > > ----- Original Message -----
> > > > I've reverted back to crash-5.1.9 and applied my kmem patch to
> > > > that for
> > > > our use here. We're using a 3.1-based kernel, and need the
> > > > kmem patch
> > > > so crash can deal with the change in CONFIG_SLAB, but we're
> > > > building
> > > > with gcc-4.4.5 and don't really need the new gdb in
> > > > crash-6.0.2, and
> > > > crash-6.0.2 is not giving module source line numbers for us
> > > > with "dis -l".
> > > >
> > > > This is just a heads up. I don't know why 6.0.2 is failing
> > > > this, and
> > > > since I found the last module source line number problem, it's
> > > > not my
> > > > turn ;-)
> > > >
> > > > Bob Montgomery
> > >
> > > What kernel version? I'll try to reproduce it.
>
> Here's your test on my system:
>
> crash-6.0.2> help | grep version
> crash-6.0.2 version: 6.0.2 gdb version: 7.3.1
>
> crash-6.0.2> help -k | grep proc_version
> proc_version: Linux version 3.1.4-clim-2-amd64 (buildd at hpdebuild)
> (gcc
> version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
>
> crash-6.0.2> mod -s bnx2
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
> MODULE NAME SIZE OBJECT FILE
> ffffffffa01f85e0 bnx2 65655
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
>
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6265
> 0xffffffffa01f32e1 <bnx2_open>: push %rbp
> 0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
> 0xffffffffa01f32e5 <bnx2_open+4>: push %r15
> 0xffffffffa01f32e7 <bnx2_open+6>: push %r14
> 0xffffffffa01f32e9 <bnx2_open+8>: push %r13
> 0xffffffffa01f32eb <bnx2_open+10>: push %r12
> 0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
> 0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6266
> 0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6265
> 0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/drivers/net/bnx2.c:
> 6269
>
> Ummmm. Ooops? Is it time to prepare my embarrassing explanation of
> my
> prior claim???
>
> =====================
>
> First start over and do what I actually was doing when I saw the
> problem:
>
>
> $ crash-6.0.2 --no_kmem_cache dump.201201062015 kernel_link
> ...
>
> crash-6.0.2> help | grep version
> crash-6.0.2 version: 6.0.2 gdb version: 7.3.1
>
> crash> help -k | grep proc_version
> proc_version: Linux version 3.1.4-clim-2-amd64 (buildd at hpdebuild)
> (gcc
> version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
>
> crash-6.0.2> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
> ...
>
> crash-6.0.2> mod | grep sunrpc
> ffffffffa01cab80 auth_rpcgss
> 40572
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko
> ffffffffa03a7070 sunrpc
> 202679
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/net/sunrpc/sunrpc.ko
>
> crash-6.0.2> dis -l rpc_sleep_on_priority
> 0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
> 0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
> 0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
> 0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
> 0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
> 0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
> 0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
> 0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov
> 0x70(%rsi),%rax
> 0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
> 0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
> 0xffffffffa038f738
> 0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2
>
> Well, that's a relief...
>
> Oh, by the way, now that I've used mod -S to load all modules instead
> of
> individually loading bnx2.ko:
>
> crash-6.0.2> mod | grep bnx2
> ffffffffa01f85e0 bnx2 65655
> /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64/kernel/drivers/net/bnx2.ko
>
>
> crash-6.0.2> dis -l bnx2_open
> 0xffffffffa01f32e1 <bnx2_open>: push %rbp
> 0xffffffffa01f32e2 <bnx2_open+1>: mov %rsp,%rbp
> 0xffffffffa01f32e5 <bnx2_open+4>: push %r15
> 0xffffffffa01f32e7 <bnx2_open+6>: push %r14
> 0xffffffffa01f32e9 <bnx2_open+8>: push %r13
> 0xffffffffa01f32eb <bnx2_open+10>: push %r12
> 0xffffffffa01f32ed <bnx2_open+12>: mov %rdi,%r12
> 0xffffffffa01f32f0 <bnx2_open+15>: push %rbx
> 0xffffffffa01f32f1 <bnx2_open+16>: lea 0x640(%rdi),%rbx
> 0xffffffffa01f32f8 <bnx2_open+23>: sub $0x8,%rsp
> 0xffffffffa01f32fc <bnx2_open+27>: callq 0xffffffff8129477b
> <netif_carrier_off>
>
> No source lines.
>
> ===================
>
> If I do the same steps on my 5.1.9 version of crash, I get:
>
> crash> help | grep version
> crash version: 5.1.9 gdb version: 7.0
>
> crash> help -k | grep proc_version
> proc_version: Linux version 3.1.4-clim-2-amd64 (buildd at hpdebuild)
> (gcc
> version 4.4.5 (Debian 4.4.5-8) ) #1 SMP Fri Dec 16 00:12:55 UTC 2011
>
> crash> mod -S /usr/lib/debug/lib/modules/3.1.4-clim-2-amd64
> ...
>
> crash> dis -l rpc_sleep_on_priority
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c:
> 350
> 0xffffffffa038f71b <rpc_sleep_on_priority>: push %rbp
> 0xffffffffa038f71c <rpc_sleep_on_priority+1>: mov %rsp,%rbp
> 0xffffffffa038f71f <rpc_sleep_on_priority+4>: push %r12
> 0xffffffffa038f721 <rpc_sleep_on_priority+6>: mov %ecx,%r12d
> 0xffffffffa038f724 <rpc_sleep_on_priority+9>: push %rbx
> 0xffffffffa038f725 <rpc_sleep_on_priority+10>: mov %rdi,%rbx
> 0xffffffffa038f728 <rpc_sleep_on_priority+13>: sub $0x10,%rsp
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/arch/x86/include/asm/bitops.h:
> 312
> 0xffffffffa038f72c <rpc_sleep_on_priority+17>: mov
> 0x70(%rsi),%rax
> /build/buildd/linux-3.1-clim-3.1.4-clim/debian/build/build_amd64_none_amd64/net/sunrpc/sched.c:
> 352
> 0xffffffffa038f730 <rpc_sleep_on_priority+21>: test $0x4,%al
> 0xffffffffa038f732 <rpc_sleep_on_priority+23>: jne
> 0xffffffffa038f738 <rpc_sleep_on_priority+29>
> 0xffffffffa038f734 <rpc_sleep_on_priority+25>: ud2a
>
> So 5.1.9 works, and 6.0.2 fails if I load all modules with mod -S.
> That's interesting.
>
> Bob Montgomery
>
>
>
>
>
More information about the Crash-utility
mailing list