[Crash-utility] Re: Crash-utility Digest, Vol 35, Issue 6

[Dave Anderson]

Durga Prasad wrote:
>      > Hi Dave,
>      > I attempted to try your suggested commands, but it tends to fail
>     on my
>      > machine:
>      >
>      >
>      > libl041:/var/log/dump # crash -s vmlinux.sles10sp2.smp.x86_64
>      > 2008-08-08-23\:27/vmcore
>      > crash> struct ext3_xattr_search
>      > struct: invalid data structure reference: ext3_xattr_search
>      > crash> mod -s ext3
>      >      MODULE       NAME                   SIZE  OBJECT FILE
>      > ffffffff8856a500  ext3                 153360
>      > /lib/modules/2.6.16.60-0.21-smp/kernel/fs/ext3/ext3.ko
>      > crash> struct ext3_xattr_search
>      > struct: invalid data structure reference: ext3_xattr_search
>      > crash> struct ext3_xattr_info
>      > struct: invalid data structure reference: ext3_xattr_info
>      >
>      >
>      > crash> sys
>      >       KERNEL: vmlinux.sles10sp2.smp.x86_64
>      >     DUMPFILE: 2008-08-08-23:27/vmcore
>      >         CPUS: 4
>      >         DATE: Fri Aug  8 23:20:09 2008
>      >       UPTIME: 01:31:48
>      > LOAD AVERAGE: 2.48, 1.32, 0.56
>      >        TASKS: 157
>      >     NODENAME: libl041
>      >      RELEASE: 2.6.16.60-0.21-smp
>      >      VERSION: #1 SMP Tue May 6 12:41:02 UTC 2008
>      >      MACHINE: x86_64  (1995 Mhz)
>      >       MEMORY: 3.9 GB
>      >        PANIC: "Oops: 0000 [1] SMP " (check log for details)
>      >
>      > crash version: 4.0-4.10   gdb version: 6.1
>      >
>      > I tried the same after compiling and installing the latest crash
>     version
>      > (crash version: 4.0-6.3   gdb version: 6.1) with no difference in
>     results.
>      >
>      > What would you suggest?
>      >
>      > Thanks
>      > Durga
> 
>     The first thing to verify is whether those two structures existed
>     in the 2.6.16-era timeframe?
> 
>     Dave
> 
> 
>  
> 
>     The 2.6.16.60-021 kernel DO have the said structures.

OK, can you confirm that the ext3.ko file has debuginfo data in it?

I'm not familiar with how SUSE handles kernel modules and their debuginfo data.
But, for example, RHEL kernels require that an associated kernel-debuginfo package
be installed, which supplies both the vmlinux file as well as the "partner" .ko.debug
files for each kernel module.

So for example:

   crash> mod -s ext3
    MODULE   NAME                  SIZE  OBJECT FILE
   e08cd080  ext3                123593  /lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko
   crash>

and the tools that use a /lib/modules/.../*.ko file know how to link it to its
associated *.ko.debug debuginfo piece:

   $ ls -l /lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko
   -rwxr--r-- 1 root root 145544 Apr  1 20:02 /lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko
   $ ls -l /usr/lib/debug/lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko.debug
   -rwxr--r-- 1 root root 1676292 Apr  1 20:02 /usr/lib/debug/lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko.debug*
   $

If the relevant debuginfo package was not installed, the "mod" command would
do and look the same as above, but there would be no debuginfo data available
for the ext3 module.  (It would just pull extra symbol values from the
ext3.ko file if available)

So what does gdb alone show?  With the kernel debuginfo package installed,
I see this:

   $ gdb /lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko
   GNU gdb Red Hat Linux (6.5-37.el5rh)
   Copyright (C) 2006 Free Software Foundation, Inc.
   GDB is free software, covered by the GNU General Public License, and you are
   welcome to change it and/or distribute copies of it under certain conditions.
   Type "show copying" to see the conditions.
   There is absolutely no warranty for GDB.  Type "show warranty" for details.
   This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1".

   (gdb) ptype struct ext3_xattr_search
   type = struct ext3_xattr_search {
       struct ext3_xattr_entry *first;
       void *base;
       void *end;
       struct ext3_xattr_entry *here;
       int not_found;
   }
   (gdb)

But if I move the associated ext3.ko.debug out of the way -- to simulate that the
debuginfo package is not installed -- gdb indicates "(no debugging symbols found)":

   $ mv /usr/lib/debug/lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko.debug /tmp
   $ gdb /lib/modules/2.6.18-88.el5/kernel/fs/ext3/ext3.ko
   GNU gdb Red Hat Linux (6.5-37.el5rh)
   Copyright (C) 2006 Free Software Foundation, Inc.
   GDB is free software, covered by the GNU General Public License, and you are
   welcome to change it and/or distribute copies of it under certain conditions.
   Type "show copying" to see the conditions.
   There is absolutely no warranty for GDB.  Type "show warranty" for details.
   This GDB was configured as "i386-redhat-linux-gnu"...(no debugging symbols found)
   Using host libthread_db library "/lib/libthread_db.so.1".

   (gdb) ptype struct ext3_xattr_search
   No struct type named ext3_xattr_search.
   (gdb)

Dave