[Crash-utility] crashdc: problem when running crash in kexec environment for SLES11

[Dave Anderson]

----- "Louis Bouchard" <louis.bouchard at hp.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hello,
> 
> As I indicated previously, I'm having a problem with running crash from
> within the kexec environment of SLES11.
> 
> Actually, crash itself runs fine, but its gdb portion seems to be having
> problems. Here is a capture of what happens :
> 
> > Running /usr/bin/run-crashdc-sles11.sh
> > crashexe : /usr/bin/crash
> > crashoutput : /root/var/crash/2009-09-29-17:45/crash-data-200909291746.txt
> > namelist : /root/boot/vmlinux-2.6.27.23-0.1-default
> > vmcorefile : /root/var/crash/2009-09-29-17:45/vmcore
> > debuginfofile : /root/usr/lib/debug/boot/vmlinux-2.6.27.23-0.1-default.debug
> > 
> This portion above is debug info from crashdc.
> 
> > crash 4.0-7.6
> > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008  Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006  IBM Corporation
> > Copyright (C) 1999-2006  Hewlett-Packard Co
> > Copyright (C) 2005, 2006  Fujitsu Limited
> > Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> > Copyright (C) 2005  NEC Corporation
> > Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions.  Enter "help copying" to see the conditions.
> > This program has absolutely no warranty.  Enter "help warranty" for details.
> >  
> > NOTE: stdin: not a tty
> > 
> > cannot determine relocation value: not a live system
> > gdb /root/usr/lib/debug/boot/vmlinux-2.6.27.23-0.1-default.debug 
> > 
> > 
> > dlopen failed on 'libthread_db.so.1' - libthread_db.so.1: cannot open shared object file: No such file or directory
> > GDB will not be able to debug pthreads.
> > 
> > GNU gdb 6.1
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you are
> > welcome to change it and/or distribute copies of it under certain conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for details.
> > This GDB was configured as "i686-pc-linux-gnu".../usr/bin/crashdc:
> line 160:   682 Killed                  $crashexe -d $crashdebug $namelist $debuginfofile $vmcorefile < $crashcmd
> > File /root/var/crash/2009-09-29-17:45/crash-data-200909291746.txt has been generated
> > generated a crash-data file to /root/var/crash/2009-09-29-17:45 
> > ..done
> > Restarting system.
> 
> My feeling is that the "dlopen failed on 'libthread_db.so.1'" might be
> causing this.

I note that if you move the library away entirely, the embedded gdb in the
crash utility complains as above, but crash/gdb still continues to run.  

> 
> Bernhard Walle might have an idea. Here is what I have in
> /etc/sysconfig/kdump :
> 
> KDUMP_REQUIRED_PROGRAMS="/bin/basename /usr/bin/crash /usr/bin/crashdc /usr/bin/run-crashdc-sles11.sh /etc/rc.status /bin/gzip /bin/logger /usr/bin/gdb /lib/libpthread.so.0 /lib/libthread_db.so.1"

Anyway, this is just a wild guess, and I'm presuming that you're running
with x86_64 above, but on RHEL it uses the 64-bit version of that library:

  crash> vm
  PID: 29013  TASK: ffff81003c2eb0c0  CPU: 4   COMMAND: "crash"
         MM               PGD          RSS    TOTAL_VM
  ffff81003b128040  ffff81001e9cf000  116016k  190620k 
        VMA           START       END     FLAGS FILE
  ffff81000fd416b8     400000     830000   1875 /usr/bin/crash
  ffff810008b2b558     a2f000     a51000 101873 /usr/bin/crash
  ffff8100133a0558     a51000     bb9000 100073 
  ffff810009fd2b88    5ce8000    9782000 100073 
  ffff81003a5e8ef8 3fad600000 3fad61a000    875 /lib64/ld-2.5.so
  ffff810025d75298 3fad81a000 3fad81b000 100871 /lib64/ld-2.5.so
  ffff8100137be298 3fad81b000 3fad81c000 100873 /lib64/ld-2.5.so
  ffff810032b43a28 3fada00000 3fada06000     75 /lib64/libthread_db-1.0.so
  ffff810032b431e8 3fada06000 3fadc05000     70 /lib64/libthread_db-1.0.so
  ffff810032b43088 3fadc05000 3fadc06000 100071 /lib64/libthread_db-1.0.so
  ffff810032b43ce8 3fadc06000 3fadc07000 100073 /lib64/libthread_db-1.0.so
  ...

Does the KDUMP_REQUIRED_PROGRAMS list need to specify the /lib64 version,
or does SLES have a different 32/64-bit library setup?

Dave