[Crash-utility] crash tool not working
Dave Anderson
anderson at redhat.com
Mon Feb 6 15:00:28 UTC 2012
----- Original Message -----
> Hi List,
>
> I am new to crash / kdump tool and failing some problem as mentioned
> below.
>
> I am referring the Linux Kernel Crash Book (http://www.dedoimedo.com/)
> and URL http://www.dedoimedo.com/computers/crash.html.
Well, that's your first mistake...
The section in that book that has caused your confusion is this part:
| You can use the old way. Here's an example on CentOS 5.4:
|
| crash /boot/System.map-2.6.18-164.10.1.el5 /boot/vmlinuz-2.6.18-164.10.1.el5 vmcore
|
| Notice the use of vmlinuz kernel image, as opposed to vmlinux previously required.
Given CentOS 5.4 is a copy of RHEL5.4, I cannot even imagine where the
author got the impression that the /boot/vmlinuz-<release> file
could *ever* have been used by the crash utility. The vmlinuz file is
a stripped and compressed output from the vmlinux file, and is completely
unsuitable for use by the crash utility. His statement above is just
complete fanatasy. The crash utility needs an unstripped vmlinux file
containing the debuginfo sections for the base kernel. He does subsequently
state that "alternatively" you can use vmlinux file:
| Alternatively, you can use only the debug information under /usr/lib/debug. The
| information is extracted during the installation of kernel-debuginfo packages
| matching the kernel that was running at the time of the kernel crash.
But in reality -- that is the only file that can be used with the
crash utility, at least from a RHEL5/CentOS perspective. I can't
speak for any other distribution.
> I am building modified kernel source (2.6.32 based) and added my
> modules (for study purpose). Building, installing and booting kernel
> is successful. I have enabled the options for kdump as mentioned in
> the book:
>
> Enable Kexec system call:
> CONFIG_KEXEC=y
>
> Enable kernel crash dumps:
> CONFIG_CRASH_DUMP=y
>
> Optional: Disable Symmetric Multi-Processing (SMP) support
> CONFIG_SMP=y
>
> Enable sysfs file system support:
> CONFIG_SYSFS=y
>
> Enable /proc/vmcore support:
> CONFIG_PROC_VMCORE=y
>
> Configure the kernel with debug info:
> CONFIG_DEBUG_INFO=y
>
> Configure the start section for reserved RAM for the crash kernel:
> CONFIG_PHYSICAL_START=0x200000 (2MB)
>
> Configure kdump kernel so it can be identified:
> CONFIG_LOCALVERSION="-crash"
>
>
> Kdump configuration /etc/sysconfig/kdump:
> KDUMP_KERNELVER=""
>
> KDUMP_COMMANDLINE=""
>
> KDUMP_COMMANDLINE_APPEND="maxcpus=1 "
>
> KEXEC_OPTIONS=""
>
> KDUMP_IMMEDIATE_REBOOT="yes"
>
> KDUMP_TRANSFER=""
>
> KDUMP_SAVEDIR="file:///var/crash"
>
> KDUMP_KEEP_OLD_DUMPS="5"
>
> KDUMP_FREE_DISK_SIZE="64"
>
> KDUMP_VERBOSE="3"
>
> KDUMP_DUMPLEVEL="0"
>
> KDUMP_DUMPFORMAT="compressed"
>
> There is no option KDUMP_DUMPDEV option
> There is no option KDUMP_RUNLEVEL
>
> I booted successfully with this kernel and tried to crash it by
> module. After rebooting, I found that vmcore is generated under
> /var/crash/ But I am not able to analyze it with crash command.
>
> linux:/home/adil # cat /proc/cmdline
> root=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part1
> resume=/dev/disk/by-id/ata-WDC_WD800BD-22LRA1_WD-WMAM9ZS19445-part2
> splash=silent crashkernel=256M-:128M vga=0x31a
> linux:/home/adil #
>
> linux:/home/adil # crash /boot/System.map-2.6.32.12-crash-crash
> /boot/vmlinuz-2.6.32.12-crash-crash
>
> crash 5.0.1
> Copyright (C) 2002-2010 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> crash: /boot/vmlinuz-2.6.32.12-crash-crash: not a supported file format
>
> Usage:
> crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist]
> [dumpfile]
>
> Enter "crash -h" for details.
> linux:/home/adil #
Like the error message indicates, the vmlinuz-2.6.32.12-crash-crash
file is not a supported file format. Never has been, never will be...
>
> The URL http://www.dedoimedo.com/computers/crash.html mentioned that
> "The newer versions of Kdump can work with compressed kernel images.
> Furthermore, they copy the System map file and the kernel image into
> the crash directory, making the use of crash utility somewhat
> simpler."
What has been supported since crash-5.1.3 is the usage of
vmlinux files that have been subsequently compressed with either
gzip or bzip2. But again, the /boot/vmlinuz-<release> file is
a completely different animal.
> linux:/home/adil # ls -al /var/crash/2012-01-30-18\:08/
> total 1335536
> drwxr-xr-x 2 root root 4096 2012-01-30 18:13 .
> drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
> -rw-r--r-- 1 root root 187 2012-01-30 18:13 README.txt
> -rw-r--r-- 1 root root 1716605 2012-01-30 18:13 System.map-2.6.32.12-0.7-default
> -rw------- 1 root root 1360732590 2012-01-30 18:13 vmcore
> -rw-r--r-- 1 root root 3774506 2012-01-30 18:13 vmlinux-2.6.32.12-0.7-default.gz
> linux:/home/adil # ls -al /var/crash/2012-01-31-12\:17/
> total 1343860
> drwxr-xr-x 2 root root 4096 2012-01-31 12:24 .
> drwxr-xr-x 8 root root 4096 2012-01-31 12:17 ..
> -rw-r--r-- 1 root root 187 2012-01-31 12:24 README.txt
> -rw------- 1 root root 1374748735 2012-01-31 12:24 vmcore
> linux:/home/adil #
>
> linux:/home/adil # ls /boot/
> backup_mbr
> boot
> boot.readme
> config-2.6.32.12-0.7-default
> config-2.6.32.12-0.7-xen
> grub
> initrd
> initrd-2.6.32.12-0.7-default
> initrd-2.6.32.12-0.7-default-kdump
> initrd-2.6.32.12-0.7-xen
> initrd-2.6.32.12-crash-crash
> initrd-2.6.32.12-crash-crash-kdump
> initrd-xen
> message
> symsets-2.6.32.12-0.7-default.tar.gz
> symtypes-2.6.32.12-0.7-default.gz
> symvers-2.6.32.12-0.7-default.gz
> symvers-2.6.32.12-0.7-xen.gz
> System.map-2.6.32.12-0.7-default
> System.map-2.6.32.12-0.7-xen
> System.map-2.6.32.12-crash-crash
> vmlinux-2.6.32.12-0.7-xen.gz
> vmlinuz
> vmlinuz-2.6.32.12-0.7-default
> vmlinuz-2.6.32.12-0.7-xen
> vmlinuz-2.6.32.12-crash-crash
> vmlinuz-xen
> vmlinux-2.6.32.12-0.7-default.gz
> linux:/home/adil #
>
>
> Another observation is boot.kdump seems to on but manually start
> giving me error:
>
> linux:/home/adil # chkconfig boot.kdump
> boot.kdump on
> linux:/home/adil #
>
> linux:/home/adil # /etc/init.d/boot.kdump start
> Loading kdump
> Regenerating kdump initrd ...
> Can't find kernel text map area from kcore
> Cannot load /boot/vmlinuz-2.6.32.12-crash-crash
>
> failed
> linux:/home/adil #
>
> Other Query: Following is not clear mentioned in the book under
> "section 11.2 Crash (capture) kernel":
> --------------
> This means that while your production kernels will most likely be
> named vmlinuz, the Kdump crash kernels need to be uncompressed,
> hence named vmlinux, or rather vmlinux-kdump.
> ---------------
>
> Please help how to correctly setup and use crash on my machine.
With respect to the crash utility, if your kernel build procedure
has created a /usr/lib/debug/lib/modules/<kernel-version>/vmlinux,
then you don't need a System.map file, but rather:
$ crash /usr/lib/debug/lib/modules/<kernel-version>/vmlinux vmcore
If you don't have the file above, and if whatever non-RHEL system you're
running actually puts the vmlinux-<version>.gz file in the /var/crash
subdirectory with the vmcore, then use that file:
$ crash vmlinux-<kernel-version>.gz vmcore
You may also need the System.map file found there as well. I believe
Suse post-builds their debuginfo kernels instead of saving the vmlinux
file from when it was originally built. Could be that their version
of kexec-tools does the copy of the extra files into /var/crash?
I really don't know -- in a RHEL system, only the vmcore files are
copied there.
Anyway, if you have neither of the above, then check your modified kernel
build tree. In the top-level directory, there should be a "vmlinux" file,
which is the file that the crash utility requires. This also presumes that
you built the kernel with -g, which would get turned on when the kernel
is configured with CONFIG_DEBUG_INFO. However, it should be noted
that if your kernel build procedure built more than one kernel "flavor",
then the vmlinux file found there will be that of the last kernel that
was built -- which may not be the one you need.
For example, RHEL5 x86_64 builds three x86_64 kernel rpms:
kernel-2.6.18-232.el5.x86_64.rpm
kernel-debug-2.6.18-232.el5.x86_64.rpm
kernel-xen-2.6.18-232.el5.x86_64.rpm
The rpm files above contain the /boot/vmlinuz-<release> files.
Then, each kernel above has a debuginfo package associated
with it, which contains its original unstripped vmlinux file,
which gets put in /usr/lib/debug/lib/modules/<kernel-version>/vmlinux:
kernel-debuginfo-2.6.18-232.el5.x86_64.rpm
kernel-debug-debuginfo-2.6.18-232.el5.x86_64.rpm
kernel-xen-debuginfo-2.6.18-232.el5.x86_64.rpm
And there is a "common" debuginfo file that is typically installed
with the kernel-specific file above:
kernel-debuginfo-common-2.6.18-232.el5.x86_64.rpm
Again, I can't speak for the procedures used by any other distribution.
Also, do yourself a favor, and upgrade your crash utility version
from crash-5.0.1, which is two years old.
And lastly, although you may get some responses, this is not the list for
kdump-related questions. That is found here:
http://lists.infradead.org/mailman/listinfo/kexec
Dave Anderson
More information about the Crash-utility
mailing list