[Crash-utility] Crash on Linux 3.6 rc1

[Dave Anderson]

----- Original Message -----
> 
> 
> ----- Original Message -----
> > On 08/13/12 13:56, Dave Anderson wrote:
> > >
> > >
> > > ----- Original Message -----
> > >> I trip over this bug on Linux 3.6 rc1. Crash runs fine on Linux
> > >> 3.5.
> > 
> > ...
> > 
> > >>
> > >> crash: invalid structure member offset: kmem_cache_s_next
> > >>          FILE: memory.c  LINE: 7945  FUNCTION: kmem_cache_init()
> > >>
> > >> [/usr/bin/crash] error trace: 468317 =>  49dbb2 =>  487f28 =>
> > >>  5083da
> > >>
> > >>     5083da: OFFSET_verify+202
> > >>     487f28: kmem_cache_init+312
> > >>     49dbb2: vm_init+5794
> > >>     468317: main_loop+215
> > >>
> > 
> > ...
> > 
> > >
> > > It's this part of commit 3b0efdfa1e719303536c04d9abca43abeb40f80a
> > > that changed the kmem_cache.next member from "next" to "list":
> > >
> > > diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h
> > > index fbd1117..1d93f27 100644
> > > --- a/include/linux/slab_def.h
> > > +++ b/include/linux/slab_def.h
> > > @@ -27,7 +27,7 @@ struct kmem_cache {
> > >          unsigned int limit;
> > >          unsigned int shared;
> > >
> > > -       unsigned int buffer_size;
> > > +       unsigned int size;
> > >          u32 reciprocal_buffer_size;
> > >   /* 2) touched by every alloc&  free from the backend */
> > >
> > > @@ -52,7 +52,10 @@ struct kmem_cache {
> > >
> > >   /* 4) cache creation/removal */
> > >          const char *name;
> > > -       struct list_head next;
> > > +       struct list_head list;
> > > +       int refcount;
> > > +       int object_size;
> > > +       int align;
> > >
> > >   /* 5) statistics */
> > >   #ifdef CONFIG_DEBUG_SLAB
> > >
> > > I'm curious whether if you change line 505 in memory.c:
> > >
> > >      502                 } else {
> > >      503                         STRUCT_SIZE_INIT(kmem_cache_s,
> > >      "kmem_cache");
> > >      504
> > >                              MEMBER_OFFSET_INIT(kmem_cache_s_num,
> > >      "kmem_cache", "num");
> > >      505        =====>
> > >                 MEMBER_OFFSET_INIT(kmem_cache_s_next,
> > >      "kmem_cache", "next");
> > >      506
> > >                              MEMBER_OFFSET_INIT(kmem_cache_s_name,
> > >      "kmem_cache", "name");
> > >      507
> > >                              MEMBER_OFFSET_INIT(kmem_cache_s_colour_off,
> > >      "kmem_cache",
> > >      508                                 "colour_off");
> > >
> > > to this:
> > >
> > >              MEMBER_OFFSET_INIT(kmem_cache_s_next, "kmem_cache",
> > >              "list");
> > >
> > > that it might work?
> > >
> > > Worse case, you can work around it by using the "--no_kmem_cache"
> > > option
> > > on the crash command line.
> > >
> > > Thanks,
> > >    Dave
> > >
> > > --
> > > Crash-utility mailing list
> > > Crash-utility at redhat.com
> > > https://www.redhat.com/mailman/listinfo/crash-utility
> > 
> > The error changed to:
> > 
> > ~/xfs # crash System.map vmlinux
> > 
> > crash 6.0.8
> > Copyright (C) 2002-2012  Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> > Copyright (C) 1999-2006  Hewlett-Packard Co
> > Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> > Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> > Copyright (C) 2005, 2011  NEC Corporation
> > Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it  under
> > certain conditions.  Enter "help copying" to see the conditions.
> > This program has absolutely no warranty.  Enter "help warranty" for
> > details.
> > 
> > GNU gdb (GDB) 7.3.1
> > Copyright (C) 2011 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> > 
> > 
> > crash: cannot resolve: "cache_chain"
> > 
> > --Mark.
> 
> Yeah, that would have been too convenient...
> 
> So anyway, there's more to it than just the structure member name change.
> For this error, it's because of this global data name change, from:
> 
>   static struct list_head cache_chain;
> 
> to:
> 
>   LIST_HEAD(slab_caches);
> 
> Cscope shows this as the result of a string search for "cache_chain":
> 
> 0 memory.c 7744 int has_cache_chain;
> 1 memory.c 7745 ulong cache_chain;
> 2 memory.c 7758 if (symbol_exists("cache_chain")) {
> 3 memory.c 7759 has_cache_chain = TRUE;
> 4 memory.c 7760 cache_chain = symbol_value("cache_chain");
> 5 memory.c 7762 has_cache_chain = FALSE;
> 6 memory.c 7763 cache_chain = 0;
> 7 memory.c 7802 if (has_cache_chain && (cache == cache_chain))
> 8 memory.c 7804 "cache_chain", FAULT_ON_ERROR);
> 9 memory.c 7973 *  slab chain list.  2.6 uses the "cache_chain" list_head.
> a memory.c 7976 get_symbol_data("cache_chain", sizeof(ulong), &cache);
> b memory.c 7978 cache_end = symbol_value("cache_chain");
> c memory.c 8908 get_symbol_data("cache_chain", sizeof(ulong), &si->cache);
> d memory.c 8910 cache_end = symbol_value("cache_chain");
> 
> So it would be a matter of recognizing tha "cache_chain" does not exist
> as a symbol and "slab_caches" does, and baiting-and-switching as appropriate.
> 
> I'm not sure when I'm going to get the chance to look at this.
> If anybody else on the list has some cycles to spare, fixing this
> probably won't be a major undertaking.
>  
> Dave

I've got a fix queued for this for crash-6.0.9.  It was a little
bit more involved that the bait-and-switch above, and there was
also a related "kmem -s" bug in 3.6 kernels that are configured
with CONFIG_SLUB (Fedora 18).

Dave