[Crash-utility] failed to do crash analysis after insert custom module in the first kernel.

[Dave Anderson]

----- Original Message -----
> Hi Experts:
> 
> Here is crash tool failed to work issue:
> 
> 0, it occur on pwoerpc p2041(e500mc) board, p2020(e500v2) will not have this issue.
> kernel version: 2.6.34
> crash tool version: 6.1.4

That's a 3-year-old version of crash, you really should upgrade to the most recent version.

> 
> 1, before trigger kdump, if insert a custom kernel module, the created vmcore
> will failed to be analysis.
> 
> 2, before trigger kdump, if no custom kernel module was inserted, the crash
> can worked well on created vmcore file.

I'm guessing that when crash does work OK in #2, there were *no* kernel modules installed
on the kdump'd kernel, correct?
 
> Do you have some comments on this scene?
> 
> The error log when run crash tool is here listed for reference:
> /*********************************************************************/
> /coredump> ./crash.p4 vmlinux.host.20160108nokgdbserial vmcore-1970-01-01
> 
> crash.p4 6.1.4
> Copyright (C) 2002-2013 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
> 
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "powerpc-wrs-linux"...
> 
> please wait... (gathering module symbol data)
> crash.p4: invalid structure member offset: module_core_size
> FILE: kernel.c LINE: 2976 FUNCTION: module_init()
> 
> [./crash.p4] error trace: 1006b48c => 100deff8 => 1011ac7c => 10067740
> 
> 10067740: OFFSET_verify.part.27+76
> 1011ac7c: OFFSET_verify+76
> 100deff8: module_init+1576
> 1006b48c: main_loop+236
> 
> root at RCU-1:/mnt/userdir/coredump>
> /*********************************************************************/

Crash has a hardwired presumption that the kernel's "module" structure has
a "core_size" member.  That member has existed since the beginning of Linux
kernel version 2.6 if I'm not mistaken.  (It used to be "module.size" back in
the 2.4 era)  The module.core_size member is not inside an #ifdef of any kind,
so I can't explain why it is not being found by the embedded gdb module.  

If you do this, is the core_size member not shown?:

  $ gdb vmlinux.host.20160108nokgdbserial
  ...
  (gdb) ptype struct module
  type = struct module {
  ...

I only have one 2.6.34-era kernel on-hand, and when I run the command above,
it looks like this:

  $ gdb vmlinux-2.6.34-11.fc14
  GNU gdb (GDB) Fedora (7.4.50.20120120-54.fc17)
  Copyright (C) 2012 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-redhat-linux-gnu".
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>...
  Reading symbols from /tmp/vmlinux-2.6.34-11.fc14...done.
  (gdb) ptype struct module
  type = struct module {
      enum module_state state;
      struct list_head list;
      char name[56];
      struct module_kobject mkobj;
      struct module_attribute *modinfo_attrs;
      const char *version;
      const char *srcversion;
      struct kobject *holders_dir;
      const struct kernel_symbol *syms;
      const long unsigned int *crcs;
      unsigned int num_syms;
      struct kernel_param *kp;
      unsigned int num_kp;
      unsigned int num_gpl_syms;
      const struct kernel_symbol *gpl_syms;
      const long unsigned int *gpl_crcs;
      const struct kernel_symbol *unused_syms;
      const long unsigned int *unused_crcs;
      unsigned int num_unused_syms;
      unsigned int num_unused_gpl_syms;
      const struct kernel_symbol *unused_gpl_syms;
      const long unsigned int *unused_gpl_crcs;
      const struct kernel_symbol *gpl_future_syms;
      const long unsigned int *gpl_future_crcs;
      unsigned int num_gpl_future_syms;
      unsigned int num_exentries;
      struct exception_table_entry *extable;
      int (*init)(void);
      void *module_init;
      void *module_core;
      unsigned int init_size;
      unsigned int core_size;                  <== NOTE core_size member here
      unsigned int init_text_size;
      unsigned int core_text_size;
      struct mod_arch_specific arch;
      unsigned int taints;
      unsigned int num_bugs;
      struct list_head bug_list;
      struct bug_entry *bug_table;
      Elf64_Sym *symtab;
      Elf64_Sym *core_symtab;
      unsigned int num_symtab;
      unsigned int core_num_syms;
      char *strtab;
      char *core_strtab;
      struct module_sect_attrs *sect_attrs;
      struct module_notes_attrs *notes_attrs;
      void *percpu;
      unsigned int percpu_size;
      char *args;
      struct tracepoint *tracepoints;
      unsigned int num_tracepoints;
      const char **trace_bprintk_fmt_start;
      unsigned int num_trace_bprintk_fmt;
      struct ftrace_event_call *trace_events;
      unsigned int num_trace_events;
      long unsigned int *ftrace_callsites;
      unsigned int num_ftrace_callsites;
      struct list_head modules_which_use_me;
      struct task_struct *waiter;
      void (*exit)(void);
      struct module_ref *refptr;
      ctor_fn_t *ctors;
      unsigned int num_ctors;
  }
  (gdb) 

By the way, you should be able to at least come up into a crash session by entering:

  $ crash --no_modules vmlinux.host.20160108nokgdbserial vmcore-1970-01-01

Doing that will skip kernel module initialization and therefore the function that's
killing your crash session.  However, I'm guessing that you're trying to debug a 
kernel module, and so it may not be all that helpful.

Dave