[Crash-utility] crash: invalid structure member offset

Thu Aug 12 18:56:10 UTC 2010

Thanks,

Using crash 5.0.6 worked nicely.
However, I can't really look at a lot because of a bad EIP code.

[  726.601381] 802.1Q VLAN Support v1.8 Ben Greear <greearb at candelatech.com>
[  726.601384] All bugs added by David S. Miller <davem at redhat.com>
[  726.646757] BUG: unable to handle kernel NULL pointer dereference at 00000000
[  726.732410] IP: [<00000000>]
[  726.766933] *pdpt = 0000000000431001 *pde = 0000000000000000 
[  726.766937] Oops: 0010 [#1] SMP 
[  726.790844] Modules linked in: 8021q iptable_filter ip_tables x_tables ip_gre af_packet i2c_dev i2c_qs i2c_algo_bit i2c_core garp stp llc ixgbe inet_lro psmouse serio_raw intel_agp shpchp iTCO_wdt pci_hotplug iTCO_vendor_support agpgart ext3 jbd mbcache sd_mod crc_t10dif sg ata_piix ata_generic ahci libata scsi_mod ehci_hcd uhci_hcd usbcore [last unloaded: 8021q]
[  726.790844] 
[  726.790844] Pid: 4, comm: ksoftirqd/0 Tainted: P          (2.6.27)
[  726.790844] EIP: 0060:[<00000000>] EFLAGS: 00010202 CPU: 0
[  726.790844] EIP is at 0x0
[  726.790844] EAX: e7f4c498 EBX: 00000000 ECX: 77470000 EDX: e7f4c498
[  726.790844] ESI: 4bd1d300 EDI: 00000007 EBP: f784df88 ESP: f784df78
[  726.790844]  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
[  726.790844] Process ksoftirqd/0 (pid: 4, ti=f784c000 task=f783a5b0 task.ti=f784c000)
[  726.790844] Stack: 40168080 00000001 403daaa0 4042c500 f784df90 401681bf f784dfb0 4012fe92 
[  726.790844]        0000000a 00000000 40429340 00000246 00000000 40130120 f784dfbc 4012ff55 
[  726.790844]        4042c500 f784dfcc 40130182 fffffffc 00000000 f784dfe0 4013e707 4013e6c0 
[  726.790844] Call Trace:
[  726.790844]  [<40168080>] ? __rcu_process_callbacks+0x70/0x190
[  726.790844]  [<401681bf>] ? rcu_process_callbacks+0x1f/0x40
[  726.790844]  [<4012fe92>] ? __do_softirq+0x82/0x100
[  726.790844]  [<40130120>] ? ksoftirqd+0x0/0xe0
[  726.790844]  [<4012ff55>] ? do_softirq+0x45/0x50
[  726.790844]  [<40130182>] ? ksoftirqd+0x62/0xe0
[  726.790844]  [<4013e707>] ? kthread+0x47/0x80
[  726.790844]  [<4013e6c0>] ? kthread+0x0/0x80
[  726.790844]  [<4010494f>] ? kernel_thread_helper+0x7/0x10
[  726.790844]  =======================
[  726.790844] Code:  Bad EIP value.
[  726.790844] EIP: [<00000000>] 0x0 SS:ESP 0068:f784df78

So now I can't figure out the piece of code where this dereferencing occurred. :(

crash>  bt
PID: 4      TASK: f783a5b0  CPU: 0   COMMAND: "ksoftirqd/0"
 #0 [f784de88] crash_kexec at 401534a8
 #1 [f784df28] __slab_free at 4019677f
 #2 [f784df8c] rcu_process_callbacks at 401681ba
 #3 [f784df94] __do_softirq at 4012fe90
 #4 [f784dfb4] do_softirq at 4012ff50
 #5 [f784dfd0] kthread at 4013e705
 #6 [f784dfe4] kernel_thread_helper at 4010494d

Thanks,

Reinoud.

> -----Original Message-----
> From: crash-utility-bounces at redhat.com [mailto:crash-utility-
> bounces at redhat.com] On Behalf Of Dave Anderson
> Sent: Thursday, August 12, 2010 6:14 AM
> To: Discussion list for crash utility usage, maintenance and
> development
> Subject: Re: [Crash-utility] crash: invalid structure member offset
> 
> 
> ----- "Reinoud Koornstra" <koornstra at hp.com> wrote:
> 
> > Hi Everyone,
> >
> > I am trying to read a core file into crash, but I've got bad luck as
> you can see below.
> > Is core file corrupt? It is a vmcore file from a 32 bits kernel that
> > was compiled with PAE, could that have corrupted things?
> > Any hints here?
> > Thanks,
> >
> > Reinoud.
> >
> > $ crash System.map-2.6.27 ./vmlinux-2.6.27 ./vmcore
> >
> > crash 4.0-3.7
> 
> I don't know if the vmcore is corrupt, but PAE wouldn't be an issue.
> 
> However, you are running a version of crash that was released almost
> 4 years ago (13-Oct-2006) against a two-year-old kernel that was
> released 15-Oct-2008.  That's pretty much a guarantee of failure.
> 
> Try updating to version 5.0.6 and see what happens.
> 
> And BTW, if the vmlinux file is the exact same kernel as the
> one that generated the vmcore file, you don't need a System.map
> argument.
> 
> Dave
> 
> 
> 
> 15-Oct-2008
> 
> > Copyright   2002, 2003, 2004, 2005, 2006  Red Hat, Inc.
> > Copyright   2004, 2005, 2006  IBM Corporation
> > Copyright   1999-2006  Hewlett-Packard Co
> > Copyright   2005  Fujitsu Limited
> > Copyright   2005  NEC Corporation
> > Copyright   1999, 2002  Silicon Graphics, Inc.
> > Copyright   1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public
> License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions.  Enter "help copying" to see the conditions.
> > This program has absolutely no warranty.  Enter "help warranty" for
> > details.
> >
> > GNU gdb 6.1
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and
> you are
> > welcome to change it and/or distribute copies of it under certain
> conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> > This GDB was configured as "i686-pc-linux-gnu"...
> >
> > please wait... (gathering kmem slab cache data)
> >
> > crash: invalid structure member offset: kmem_cache_s_c_num
> >        FILE: memory.c  LINE: 6891  FUNCTION: kmem_cache_init()
> >
> > [/usr/bin/crash] error trace: 80827a9 => 8095398 => 80aa7ef =>
> > 8131e88
> > /usr/bin/nm: /usr/bin/crash: no symbols
> > /usr/bin/nm: /usr/bin/crash: no symbols
> > /usr/bin/nm: /usr/bin/crash: no symbols
> > /usr/bin/nm: /usr/bin/crash: no symbols
> >
> > WARNING: Because this kernel was compiled with gcc version 4.1.2,
> certain
> >          commands or command options may fail unless crash is invoked
> with
> >          the  "--readnow" command line option.
> 
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility