Hi Dave, Thanks for the quick reply! I followed approach 1 of disabling CONFIG_STRICT_DEVMEM and the problem disappeared. Now a new problem :-) On invoking crash, I get the following warning: WARNING: cannot access vmalloc'd module memory Because of this I am not able to run commands like "mod": crash> mod mod: cannot access vmalloc'd module memory crash> Should the approach 2 or 3 that you mentioned below solve this problem? I have upgraded crash to the latest version (4.0-7.7). Thanks and regards, Adhiraj. <div class="gmail_quote">On Wed, Feb 25, 2009 at 7:45 PM, Dave Anderson <<a href="mailto:anderson@redhat.com">anderson@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div><div></div><div class="Wj3C7c"> ----- "Adhiraj Joshi" <<a href="mailto:adhiraj@linsyssoft.com">adhiraj@linsyssoft.com</a>> wrote: > Hi All, > > On my x86 machine with fedora 10, I get an error which says: "crash: > read error: kernel virtual address:...". The running kernel is not the > Fedora 10 kernel, I have installed a latest vanilla kernel (2.6.27.12) > on my machine. > > On googling, I found that someone has reported a similar problem but > which was on Fedora kernel and x86_64 arch. Here is the link to that bug: > <a href="https://bugzilla.redhat.com/show_bug.cgi?id=237383" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=237383</a> > > In my case, this is the output from the crash command: > > [root@maveric ~]# crash /lib/modules/`uname -r`/build/vmlinux > crash 4.0-7 > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > GNU gdb 6.1 > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i686-pc-linux-gnu"... > > crash: read error: kernel virtual address: c08b0ae8 type: "cpu_possible_map" > WARNING: cannot read cpu_possible_map > crash: read error: kernel virtual address: c0801a94 type: "cpu_present_map" > WARNING: cannot read cpu_present_map > crash: read error: kernel virtual address: c080155c type: "cpu_online_map" > WARNING: cannot read cpu_online_map > crash: read error: kernel virtual address: c08dd700 type: "xtime" > [root@maveric ~]# > > This is the "uname -a" output: > "Linux maveric 2.6.27.12-adhi #1 SMP Fri Feb 20 22:13:54 IST 2009 i686 > i686 i386 GNU/Linux" > > Does anyone has any idea on this? </div></div>Your kernel is configured with CONFIG_STRICT_DEVMEM, which disallows the reading of /dev/mem above 1MB physical. That makes it useless for the crash utility: <a href="https://www.redhat.com/archives/crash-utility/2009-February/msg00003.html" target="_blank">https://www.redhat.com/archives/crash-utility/2009-February/msg00003.html</a> Fedora and RHEL kernels contain the /dev/crash driver, which gets automatically installed upon crash invocation on the live system. That driver, found in the kernel file "drivers/char/crash.c", is a replacement for the (restricted) /dev/mem driver. For your kernel, you've got 3 options: (1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction. (2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel. (3) Write a kretprobe module that tinkers with the return value of the kernel's devmem_is_allowed() function such that it always returns 1. And w/respect to each option: (1) Obviously this is the course of least resistance. (2) Porting the RHEL/Fedora /dev/crash driver is possible, but since page_is_ram() is not EXPORT_GPL()'d in 2.6.27 upstream kernels, then its usage by the driver would have to be removed. And if you were to make it EXPORT_GPL(), then it makes far more sense to just remove the CONFIG_STRICT_DEVMEM instead. (3) I'll append some information re: writing a kretprobe module, but like porting the /dev/crash driver, it may require rebuilding your kernel anyway. And if that's the case, just remove the CONFIG_STRICT_DEVMEM restriction. Also, you should upgrade from 4.0-7. There have been several fixes for 2.6.27 kernels since that version. Dave ---------------------------------------------------------------------------- Writing a kretprobe for devmem_is_allowed(): There are are currently three types of probes: kprobes, jprobes, and kretprobes (also called return probes), but for this purpose, a kretprobe is required because it allows the return value of the devmem_is_allowed() function to be modified such that it will always return 1. The kernel documentation contains both directions for building a kretprobe module and example module files for each kprobe type. Pre-2.6.25 kernel trees put sample cut-and-pastable module files contained within the "./Documentation/kprobes.txt" file itself. 2.6.25 and later kernel trees locate them in the "./samples/kprobes" directory. Note that pre-2.6.25 kernels must have been configured with both CONFIG_KPROBES, CONFIG_KALLSYMS and CONFIG_MODULES turned on. Additionally, 2.6.25 and later kernels also need CONFIG_KRETPROBES, In any case, if the target kernel configs preclude the installation of the module, then it probably makes more sense to rebuild the kernel with CONFIG_STRICT_DEVMEM turned off, and avoid this kprobe approach entirely. Take the "kretprobe-example.c" cut-and-pastable file from the the "./Documentation/kprobes.txt" file, or the standalone "./samples/kprobes/kretprobe_example.c" file, whichever is appropriate, and modify the kretprobe_init() and ret_handler() functions like so: static int __init kretprobe_init(void) { int ret; - my_kretprobe.kp.symbol_name = func_name; + my_kretprobe.kp.symbol_name = "devmem_is_allowed"; ret = register_kretprobe(&my_kretprobe); if (ret < 0) { printk(KERN_INFO "register_kretprobe failed, returned %d\n", ret); return -1; } printk(KERN_INFO "Planted return probe at %s: %p\n", my_kretprobe.kp.symbol_name, my_kretprobe.kp.addr); return 0; } and make the ret_handler() function to simply do this: static int ret_handler(struct kretprobe_instance *ri, struct pt_regs *regs) { regs->ax = 1; return 0; } Earlier kernel versions may have different pt_regs structure member names for the return register, i.e., "regs->eax" or "regs->rax" for the x86 and x86_64 architectures respectively. Build the kretprobe module as directed in ./Documentation/kprobes.txt" and insmod it, after which crash will run just fine with /dev/mem. -- Crash-utility mailing list <a href="mailto:Crash-utility@redhat.com">Crash-utility@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/crash-utility" target="_blank">https://www.redhat.com/mailman/listinfo/crash-utility</a> </blockquote></div>