[Crash-utility] Error when running crash utility - crash: read error: kernel virtual address: c1399f10 type: "xtime"

[Dave Anderson]

----- "Mani" <mani2885 at gmail.com> wrote:

> Hello,
> 
> I am trying to analyse the dump created by the kdump utility. This is what I did
> I took 2.6.25.14 vanilla version of kernel and made two copies of the
> kernel named vmlinuz-2.6.25.14-main (Production kernel) and
> vmlinuz-2.6.25.14-kdump (Crash kernel) by following the procedure
> mentioned in the following website
> http://www.dedoimedo.com/computers/kdump.html
> 
> When I run crash utility using the following command
> crash /usr/src/linux-2.6.25.14-main/vmlinux /var/crash/2009-08-08-23:32/vmcore
> 
> I get the following error
> 
> crash /usr/src/linux-2.6.25.14-main/vmlinux /var/crash/2009-08-08-23:32/vmcore
> 
> crash 4.0-6.0.5
> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
> 
> GNU gdb 6.1
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for details.
> This GDB was configured as "i686-pc-linux-gnu"...
> 
> crash: read error: kernel virtual address: c1399f10 type: "xtime"
> 
> 
> I checked for the error and found this link which mentions few
> work-around/fixes for the issue with reading of /dev/mem
> http://www.mail-archive.com/crash-utility@redhat.com/msg01553.html

Alright -- before even talking about the "xtime" read error above --
you've gone off into the weeds...

The CONFIG_STRICT_DEVMEM issue only applies to analyzing the *live* system
with the crash utility, so it's completely irrelevant to your attempt
to analyze a kdump vmcore.

And with respect to CONFIG_STRICT_DEVMEM, you want to *avoid* building
your kernel with it if you want to look at the live system with crash.
If you *add* CONFIG_STRICT_DEVMEM, then you have to work around its
disabling of the usage of /dev/mem.  One of the ways to do that is
to try the kretprobe hack.  Other than that, you'd have to port the
Red Hat "/dev/crash" driver to your kernel for live system analysis.
 
So -- don't do this:
 
> I tried adding CONFIG_STRICT_DEVMEM to the .config file (This
> parameter was not already present in the config file)
> I got the following error while compiling the kernel
> 
> scripts/kconfig/mconf arch/x86/Kconfig
> .config:3869:warning: trying to assign nonexistent symbol STRICT_DEVMEM
> 
> I tried to create the kretprobe module, it works fine for do_fork()
> but when I give devmem_is_allowed() as the functioin name I get the
> following error

and if you don't build your kernel with CONFIG_STRICT_DEVMEM to begin with,
then there's no need to do this:
  
> [root at buzzsaw kprobes]# insmod kretprobe_devmem.ko
> insmod: error inserting 'kretprobe_devmem.ko': -1 Operation not
> permitted

In any case, I don't know why the insmod is failing unless you're trying to
do it as a non-root user.

> I am new to the area of kernel debugging. Please let me know how I can
> solve this issue. Thanks

Getting back to the original "xtime" read error issue, I don't
know what's causing it off-hand.  Here's what I can suggest:

(1) Update your version of the crash utility.  4.0-8.11 is available
    on the upstream site at http://people.redhat.com/anderson.  The
    easiest way to go is to take the tar.gz file there, un-tar it, go
    into the resultant directory, and type "make".  The crash binary
    will be built into that top-level directory.
(2) Indicate what your kernel's CONFIG_PHYSICAL_START and CONFIG_PHYSICAL_ALIGN
    config options are equal to.  It's possible that you may be running into
    http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
(3) If that version does not work, then post the output of 
    "crash -d7 your-vmlinux your-vmcore", and we can go from there.

Thanks,
  Dave