[Crash-utility] Crash Setup

[Dave Anderson]

----- "ADITYA KRISHNAN" <akrishn7 at ncsu.edu> wrote:

> Hi,
> 
> 
> I am new to crash utility. I want to run crash ( crash 4.0-2.18.1) on
> dual-core AMD Opteron (x86_64) cluster. The cluster nodes are running
> xen 3.0 the proc/version gives the following output : 2.6.16.13-xen
> i am running xen 3.0 which has been installed on fc5.
> 

If the crash version 4.0-2.18.1 you are using is the Red Hat
version 4.0-2.18.1, then it is well over 2 years old. I don't
think that xen kernels were even supported back then?

(see http://people.redhat.com/anderson/crash.changelog.html)

In any case, please try the latest version by downloading 
crash-4.0-7.4.tar.gz from http://people.redhat.com/anderson,
and build it like this:

 # tar xvzmf crash-4.0-7.4.tar.gz
 ...
 # cd crash-4.0-7.4
 # make
 ...
 #

If there is a bug here, then I need to know if it exists in
the latest version.  I don't debug old versions.

> 
> I guess the kernel was not built with debugging info on

I don't think so.  If that were true, it would have failed much
earlier with a message like:

 vmlinux-syms-2.6.16.13-xen: no debugging data available

And given that the name of the file is "vmlinux-syms-2.6.16.13-xen"
the "syms" presumably implies it's got debugging data in it.
But I don't know how it works on whatever distribution you're
running.  It seems strange that a "vmlinux-syms-2.6.16.13-xen"
with full debuginfo data would be located in /boot?

> 
> 
> Running crash gives the following output:
> 
> 
> 
> crash 4.0-2.18.1
> Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005 Fujitsu Limited
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
> 
> 
> WARNING: cannot read linux_banner string
> crash: /boot/vmlinux-syms-2.6.16.13-xen and /dev/mem do not match!
> 
> 
> I read up on the previous threads that this is because /dev/mem
> returns all zereos.
> I think I need something like a patch for installing the /dev/crash
> module. I am new to this so can anyone help me out with a detailed
> procedure of what needs to be done.
> 

I wouldn't think the /dev/crash driver is necessary on a kernel of
that vintage.  Only Red Hat kernels from RHEL4 onward and upstream
kernels with the CONFIG_STRICT_DEVMEM configuration turned on would
require an alternative like the Red Hat /dev/crash driver.

But that being said, the "cannot read linux_banner string" means
that either one of these two things happened:

 1. the read() from /dev/mem failed, or
 2. the read() from /dev/mem worked, but returned zeroes.

I can't tell which happened, but I'm guessing the latter, because
there are a handful of other read()'s done from /dev/mem before that
particular one is done, which is trying to access the linux_banner string.
It may be due to an incorrect access to the linux_banner string, which has
changed over the years.  In some kernels, the "linux_banner" symbol
is the address of the actual string, whereas on others it contains
a pointer to the actual string.  The crash utility (*now*) has
this code segment in its verify_version() function:

        if (!(sp = symbol_search("linux_banner")))
                error(FATAL, "linux_banner symbol does not exist?\n");
        else if ((sp->type == 'R') || (sp->type == 'r'))
                linux_banner = symbol_value("linux_banner");
        else
                get_symbol_data("linux_banner", sizeof(ulong), &linux_banner);

If you do an "nm -Bn vmlinux-syms-2.6.16.13-xen | grep linux_banner",
what do you see?  If it's type R or r, then crash reads directly from
that symbol value address.  Otherwise it reads the contents of the
that symbol value address, and reads the string from there.
You can also try "gdb vmlinux-syms-2.6.16.13-xen", from which
you can determine whether there's a level of indirection there.

But I'm blanking out w/respect to your "previous threads that this is 
because /dev/mem returns all zeroes" statement.  I'm not aware (at 
least any more) of any case where /dev/mem would return zeroes instead
of failing the read() call itself?  Can you point me to those
threads? 

But again -- I'm not interested in issues with older versions
of crash.  Please upgrade to the latest version, and if that
doesn't work, run it as "crash -d3 /boot/vmlinux-syms-2.6.16.13-xen",
and there will be a bunch of debug output displayed that may help nail
it down.

Dave