[Crash-utility] crash failing with CentOS 5 under VMware

[Dave Anderson]

----- Original Message -----
> I'm trying to explore crash dumps under these conditions:
> 
> - VMware workstation 7.1.5 build-491717
> - CentOS 5.7 +updates, as of today
> - kernel 2.6.18-274.17.1.el5 x86
> - crash 6.0.2
> 
> I've sucessfully enabled kdump to generate crash dumps, but the
> 'crash' utility can neither find the vmlinux image for the live
> system, nor match the crash dump with the vmlinux image I direct
> it to.
> 
> Misc RPMs installed:
> 
>   [root at 172-20-1-25 modules]# uname -r; rpm -qa | grep kernel
>   2.6.18-274.17.1.el5
>   kernel-2.6.18-274.17.1.el5
>   kernel-debug-2.6.18-274.17.1.el5
>   kernel-debuginfo-2.6.18-274.17.1.el5.centos.plus
>   kernel-debuginfo-common-2.6.18-274.17.1.el5.centos.plus
>   kernel-devel-2.6.18-274.17.1.el5
> 
> Here are various invocations I've attempted; can anyone suggest what's
> causing these failures, and how to work around them?
> 
> Please let me know if there are any details I can provide to assist.
> 
> Additional notes:
> 
> - I've tries both crashkernel=64M at 16M and crashkernel=128M at 16M as
>   kernel arguments, with the same results.
> 
> - I've attempted a symlink to match the debug modules directly to
>   the name of the kernel, but that didn't change the results:
>   2.6.18-274.17.1.el5.centos.plus -> 2.6.18-274.17.1.el5
> 
> -------------------------------
> Against the live system:
> 
>   [root at 172-20-1-25 modules]# crash
> 
>   crash 6.0.2
>   [...]
>    
>    crash: cannot find booted kernel -- please enter namelist argument
> 
> -------------------------------
> Against the vmlinux file supplied by the kernel-debuginfo RPM:
> 
>   [root at 172-20-1-25 modules]# crash /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
> 
>   crash 6.0.2
>   [...]
> 
>   WARNING: /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
>            and /proc/version do not match!
> 
>   WARNING: /proc/version indicates kernel version: 2.6.18-274.17.1.el5
> 
>   crash: please use the vmlinux file for that kernel version, or try using
>          the System.map for that kernel version as an additional
>          argument.

There is an inconsistency between the 2.6.18-274.17.1.el5.centos.plus kernel
and the running kernel 2.6.18-274.17.1.el5.  I don't know what the "centos.plus"
vmlinux kernel is, but its symbols apparently do no match the 2.6.18-274.17.1.el5
kernel that is running.  Is the system running a bona fide RHEL5 2.6.18-274.17.1.el5
kernel, and you're using a centos-generated kernel-debuginfo package?  In other words,
in a real RHEL5 environment there should be "kernel-debuginfo-2.6.18-274.17.1.el5" and
associated "kernel-debuginfo-common-2.6.18-274.17.1.el5" packages that would have
the vmlinux file for the running 2.6.18-274.17.1.el5 kernel.  So again, I don't have a
clue about what those "centos.plus" packages are all about...

Anyway, you could try what the error message suggests.  There is probably a 
"/boot/System.map-2.6.18-274.17.1.el5" on that system, and presuming so,
try this on the live system:

 $ crash usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux /boot/System.map-2.6.18-274.17.1.el5

> 
> -------------------------------
> Against my crash dump:
> 
>   [root at 172-20-1-25 modules]# crash /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux /home/crash/127.0.0.1-2012-01-26-19\:19\:19/vmcore
> 
>   crash 6.0.2
>   [...]
> 
>   crash: cannot determine base kernel version
>   WARNING: cannot read linux_banner string
>   crash: /usr/lib/debug/lib/modules/2.6.18-274.17.1.el5.centos.plus/vmlinux
>   and /home/crash/127.0.0.1-2012-01-26-19:19:19/vmcore do not match!

Same thing is happening with the vmcore.  In the case of a vmcore, it reads
the "linux_banner" string (what /proc/version shows on a live system) at the
address where the "centos.plus" vmlinux file says it is, but after doing so,
it doesn't match up.  (so it's reading the wrong location)

If you do this:

  $ strings /home/crash/127.0.0.1-2012-01-26-19:19:19/vmcore | grep "Linux version"

I'm guessing you won't see the "centos.plus" string?

As with the live system, you could try throwing the /boot/System.map-<version> file 
on the command line.

In any case, if you add a debug count on the command line (the larger the number
the more debug data gets displayed), it may illuminate the failure, i.e., by 
entering:

  $ crash -d7 /path-to/vmlinux /path-to/vmcore

Dave