Thanks Dave. The kretprobe worked for me.<div><div> </div><div>Dheeraj <div class="gmail_quote">On Thu, Dec 4, 2008 at 8:14 PM, Dave Anderson <<a href="mailto:anderson@redhat.com">anderson@redhat.com</a>> wrote: <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div></div><div class="Wj3C7c"> ----- "Dheeraj Sangamkar" <<a href="mailto:dheerajrs@gmail.com">dheerajrs@gmail.com</a>> wrote: > On the same machine, I am unable to live-debug the running kernel. > Here is the error I get. Please let me know if there is something I > can do to get crash working. > > /dheeraj/crash/crash-4.0-7.4 # crash -d 255 > /boot/System.map-2.6.27.4-2-default > /root/dheeraj/linux-2.6.27.4-2.1/vmlinux > > crash 4.0-7.4 > Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008 Red Hat, Inc. > Copyright (C) 2004, 2005, 2006 IBM Corporation > Copyright (C) 1999-2006 Hewlett-Packard Co > Copyright (C) 2005, 2006 Fujitsu Limited > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. > Copyright (C) 2005 NEC Corporation > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. > This program is free software, covered by the GNU General Public > License, > and you are welcome to change it and/or distribute copies of it under > certain conditions. Enter "help copying" to see the conditions. > This program has absolutely no warranty. Enter "help warranty" for > details. > > crash: diskdump / compressed kdump: dump does not have panic dump > header > get_live_memory_source: /dev/mem > crash: pv_init_ops exists: ARCH_PVOPS > _text: ffffffff80200000 Kernel code: 200000 -> phys_base: 0 > > gdb /root/dheeraj/linux-2.6.27.4-2.1/vmlinux > GNU gdb 6.1 > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "x86_64-unknown-linux-gnu"... > GETBUF(248 -> 0) > GETBUF(1500 -> 1) > > FREEBUF(1) > FREEBUF(0) > <readmem: ffffffff804be660, KVADDR, "kernel_config_data", 32768, > (ROE), 1ae5040> > /dev/mem: Operation not permitted > crash: read(/dev/mem, 4be660, 2464): 4294967295 (ffffffff) > crash: read error: kernel virtual address: ffffffff804be660 type: > "kernel_config_data" > WARNING: cannot read kernel_config_data > GETBUF(248 -> 0) > FREEBUF(0) > GETBUF(16 -> 0) > <readmem: ffffffff80a14800, KVADDR, "cpu_possible_map", 16, (ROE), > a49160> > /dev/mem: Operation not permitted > crash: read(/dev/mem, a14800, 16): 4294967295 (ffffffff) > crash: read error: kernel virtual address: ffffffff80a14800 type: > "cpu_possible_map" > WARNING: cannot read cpu_possible_map > <readmem: ffffffff80872710, KVADDR, "cpu_present_map", 16, (ROE), > a49160> > /dev/mem: Operation not permitted > crash: read(/dev/mem, 872710, 16): 4294967295 (ffffffff) > crash: read error: kernel virtual address: ffffffff80872710 type: > "cpu_present_map" > WARNING: cannot read cpu_present_map > <readmem: ffffffff80846c90, KVADDR, "cpu_online_map", 16, (ROE), > a49160> > /dev/mem: Operation not permitted > crash: read(/dev/mem, 846c90, 16): 4294967295 (ffffffff) > crash: read error: kernel virtual address: ffffffff80846c90 type: > "cpu_online_map" > WARNING: cannot read cpu_online_map > FREEBUF(0) > <readmem: ffffffff80a74400, KVADDR, "xtime", 16, (FOE), a0d2f0> > /dev/mem: Operation not permitted > crash: read(/dev/mem, a74400, 16): 4294967295 (ffffffff) > crash: read error: kernel virtual address: ffffffff80a74400 type: > "xtime" </div></div>Live system analysis is impossible if /dev/mem won't allow you to read physical memory. Each read attempt above results in an "Operation not permitted" (EPERM) error. They fail because of a pain-in-the-ass CONFIG_STRICT_DEVMEM configuration option that was recently added to upstream kernels. When the crash utility makes a read() from /dev/mem, it ends up in the kernel's drivers/char/mem.c driver, and proceeds like so: read_mem() range_is_allowed() devmem_is_allowed() Here is the call in read_mem() that returns the EPERM: if (!range_is_allowed(p >> PAGE_SHIFT, count)) return -EPERM; Here are the two versions of range_is_allowed() in drivers/char/mem.c, which depend upon CONFIG_STRICT_DEVMEM: #ifdef CONFIG_STRICT_DEVMEM static inline int range_is_allowed(unsigned long pfn, unsigned long size) { u64 from = ((u64)pfn) << PAGE_SHIFT; u64 to = from + size; u64 cursor = from; while (cursor < to) { if (!devmem_is_allowed(pfn)) { printk(KERN_INFO "Program %s tried to access /dev/mem between %Lx->%Lx.\n", current->comm, from, to); return 0; } cursor += PAGE_SIZE; pfn++; } return 1; } #else static inline int range_is_allowed(unsigned long pfn, unsigned long size) { return 1; } #endif And the call to devmem_is_allowed() above ends up here, where only the first 256 pages (1MB) of physical memory is accessible: int devmem_is_allowed(unsigned long pagenr) { if (pagenr <= 256) return 1; if (!page_is_ram(pagenr)) return 1; return 0; } So that makes /dev/mem useless for the crash utility... Bernhard Walle made a valiant attempt to make this a run-time configurable rather than a compile-time config option, <a href="http://lkml.org/lkml/2008/11/16/117" target="_blank">http://lkml.org/lkml/2008/11/16/117</a> But the LKML thread went off into the weeds arguing whether the CONFIG_STRICT_DEVMEM imposition itself should be ripped out, rather than putting lipstick on a pig. And so it got nowhere... This same 256-page restriction has been in place in RHEL4 and RHEL5 kernels as well, because the same guy who managed to push it upstream formerly worked at Red Hat, and convinced the powers-that-be here that /dev/mem was a security hole. As a result, I had to write a whole new physical memory-access device driver (/dev/crash) that is a read-only "misc" driver that the crash utility automatically loads when run on a live system. But it's only available in the Red Hat RHEL4 and RHEL5 distribution, although it can be ported easily enough. So you've got 3 options: 1. Rebuild your kernel with CONFIG_STRICT_DEVMEM turned off. 2. Port the RHEL /dev/crash driver to your kernel. I did suggest that in another crash-utility thread here: <a href="https://www.redhat.com/archives/crash-utility/2008-October/msg00057.html" target="_blank">https://www.redhat.com/archives/crash-utility/2008-October/msg00057.html</a> 3. Write a kprobe module that forces devmem_is_allowed() to return 1 always: <a href="http://www.redhat.com/archives/crash-utility/2008-March/msg00036.html" target="_blank">http://www.redhat.com/archives/crash-utility/2008-March/msg00036.html</a> If you've got the power, obviously #1 is the best option. But if you cannot change kernels, probably the kprobe option #3 is easier than porting the RHEL /dev/crash driver. Check Documentation/kprobes.txt file and the samples/kprobes directory in your kernel build tree, and then use the RHEL kprobe example I put in the thread referenced in #3 above. If that won't work (and I don't know why it wouldn't), then you can always port the RHEL /dev/crash driver that's attached to the thread reference in #2 above. <div><div></div><div class="Wj3C7c"> Dave -- Crash-utility mailing list <a href="mailto:Crash-utility@redhat.com">Crash-utility@redhat.com</a> <a href="https://www.redhat.com/mailman/listinfo/crash-utility" target="_blank">https://www.redhat.com/mailman/listinfo/crash-utility</a> </div></div></blockquote></div> -- Simplicity of character is the natural result of profound thought. </div></div>