[Crash-utility] [RFC] gcore subcommand: a process coredump feature

Wed Aug 11 15:06:49 UTC 2010

----- "HATAYAMA Daisuke" <d.hatayama at jp.fujitsu.com> wrote:

> From: Adrien Kunysz <adk at redhat.com>

> > Admittedly, that thread is just about extracting an userland
> > backtrace. Maybe your patch can be adapted to add a "btu" command (or
> > maybe "bt -u") to extract an userland backtrace directly?
> 
> Oh, thanks. It helps a lot.
> 
> If I've understood correctly, crash doesn't prepare any means of
> handling symbolic information for user-space processes. In this sense,
> I think the ``direct backtrace'' impossible.
> 
> On the other hand, if extracting user-space process image, one can use
> GDB to see the process's backtrace using GDB's bt sub-command.

Right -- I agree with Daisuke that the whole purpose of this gcore
command is to extract a core file for subsequent use with gdb.

But FWIW, you can load the debuginfo data of a user-space program into a 
crash session with the "add-symbol-file" gdb command.  The user-space
symbol values are then maintained only in the embedded gdb module, i.e.,
they are not available with the crash "sym" command, for example.  But
they can be accessed with commands that in turn use embedded gdb commands.

For example, I've done it in the past for debugging a live crash session
from itself.  For example:

  # ./crash

  crash 5.0.6
  Copyright (C) 2002-2010  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.

  GNU gdb (GDB) 7.0
  Copyright (C) 2009 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-unknown-linux-gnu"...

        KERNEL: /usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux
      DUMPFILE: /dev/crash
          CPUS: 8
          DATE: Wed Aug 11 10:43:03 2010
        UPTIME: 2 days, 02:20:05
  LOAD AVERAGE: 0.13, 0.08, 0.06
         TASKS: 252
      NODENAME: crash.usersys.redhat.com
       RELEASE: 2.6.18-128.el5
       VERSION: #1 SMP Wed Dec 17 11:41:38 EST 2008
       MACHINE: x86_64  (1995 Mhz)
        MEMORY: 1 GB
           PID: 24111
       COMMAND: "crash"
          TASK: ffff81001d88e7e0  [THREAD_INFO: ffff81002912e000]
           CPU: 5
         STATE: TASK_RUNNING (ACTIVE)

  crash> 

You have to set/leave the crash context to that of the program you're
interested in, because when accessing user virtual addresses, the current
context's memory space is read.

First, get the starting text address:

  crash> vm
  PID: 24111  TASK: ffff81001d88e7e0  CPU: 5   COMMAND: "crash"
         MM               PGD          RSS    TOTAL_VM
  ffff81003e075480  ffff8100032cd000  112640k  191264k 
        VMA           START       END     FLAGS FILE
  ffff810021a99768     400000     95d000   1875 /var/CVS/crash/crash
  ffff81001e06c298     b5d000     b7d000 101873 /var/CVS/crash/crash
  ffff81001e06c558     b7d000     ce5000 100073 
  ffff81001e06ce48   15976000   19062000 100073 
  ffff81001e06c768 3fad600000 3fad61a000    875 /lib64/ld-2.5.so
  ffff81001e06c348 3fad81a000 3fad81b000 100871 /lib64/ld-2.5.so
  ...

And then load the debuginfo data into the crash session at that
user text address:

  crash> add-symbol-file ./crash 0x400000
  add symbol table from file "./crash" at
          .text_addr = 0x400000
  Reading symbols from /var/CVS/crash/crash...done.
  crash>

And then, for example, the embedded gdb knows about the global 
"pc" pointer used by the crash utility to store general context
data:

  crash> whatis pc
  struct program_context *
  crash>

And its pointer value can be determined from "p -u":

  crash> p -u pc
  $2 = (struct program_context *) 0xb7d420
  crash>

And then either displayed with "struct -u":

  crash> struct -u program_context 0xb7d420
  struct program_context {
    program_name = 0x7fffcd91dc24 "crash", 
    program_path = 0x7fffcd91dc22 "./crash", 
    program_version = 0x7bc720 "5.0.6", 
    gdb_version = 0x828e00 "7.0", 
    prompt = 0x15989a20 "crash> ", 
    flags = 879609304517639, 
    namelist = 0x1597a500 "/usr/lib/debug/lib/modules/2.6.18-128.el5/vmlinux", 
    dumpfile = 0x0, 
    live_memsrc = 0x76f467 "/dev/crash", 
    system_map = 0x0, 
    namelist_debug = 0x0, 
    debuginfo_file = 0x0, 
    memory_module = 0x77c0f7 "crash", 
    memory_device = 0x76f467 "/dev/crash", 
    machine_type = 0x76f460 "X86_64", 
    editing_mode = 0x76ee85 "vi", 
    server = 0x0, 
    server_memsrc = 0x0, 
    server_namelist = 0x0, 
    nfd = -1, 
    mfd = 4, 
    kfd = -1, 
    dfd = -1, 
    confd = -2, 
    sockfd = 0, 
    port = 0, 
    rmfd = 0, 
    rkfd = 0, 
    program_pid = 24111, 
    ...

Or the same thing could be done like so:

   crash> p -u *pc
   $3 = {
     program_name = 0x7fffcd91dc24 "crash", 
     program_path = 0x7fffcd91dc22 "./crash", 
     program_version = 0x7bc720 "5.0.6", 
     ...

It's an ugly process, but it can be accomplished if necessary.

Dave