[Crash-utility] Problems debugging live system with 4.0-2.24 crash
Dave Anderson
anderson at redhat.com
Mon May 8 14:35:56 UTC 2006
"Graham, Simon" wrote:
> I've been experimenting with the latest crash version on a system running Xen (3.0.2-2 with kernel 2.6.16) and am
> experiencing problems when attempting to debug the system live; debugging of saved guest files and guest crash dumps
> works well, it's just live debugging of dom0 that is problematical.
>
> The symptom is that I get the error:
>
> WARNING: cannot read linux_banner string
>
> When I debug crash, I can see that it is finding the correct unstripped kernel from which it finds the address of the
> linux_banner variable _but_ when it attempts to read the string from /dev/mem it gets all zeroes back. I have
> confirmed that the address is correct by using gdb against /proc/kcore - gdb reads the string quite nicely from the
> virtual address (happens to be 0xc0450000, I can see that crash translates this to physical address 450000 and gets
> all zeroes when it reads this location from /dev/mem).
Hi Graham,
The xen /dev/mem driver read_mem() function does this:
if ((v = ioremap(p, sz)) == NULL) {
/*
* Some programs (e.g., dmidecode) groove off into weird RAM
* areas where no tables can possibly exist (because Xen will
* have stomped on them!). These programs get rather upset if
* we let them know that Xen failed their access, so we fake
* out a read of all zeroes. :-)
*/
if (clear_user(buf, count))
return -EFAULT;
read += count;
break;
}
when ioremap() returns NULL. Here's the NULL-returner in __ioremap():
/*
* Don't allow anybody to remap normal RAM that we're using..
*/
if (is_local_lowmem(phys_addr)) {
char *t_addr, *t_end;
struct page *page;
t_addr = bus_to_virt(phys_addr);
t_end = t_addr + (size - 1);
for(page = virt_to_page(t_addr); page <= virt_to_page(t_end); page++)
if(!PageReserved(page))
return NULL;
domid = DOMID_SELF;
}
So /dev/mem cannot be used. So to get crash to work on a live
x86 xen0 system (FC5), I've had to use the RHEL4 crash.c module
(/dev/crash) to make an end-run around /dev/mem. RHEL4's x86 and
x86_64 /dev/mem driver only allows reads of the first 256 pages of
physical memory. And now Xen has the restriction above.
FC5 has a CONFIG_CRASH configurable:
$ diff kernel-2.6.15-i686-guest.config.orig kernel-2.6.15-i686-guest.config
2526c2526
< # CONFIG_CRASH is not set
---
> CONFIG_CRASH=m
2965d2964
< # CONFIG_CRASH is not set
$
If you're building from an upstream source, you could build the
crash.c module from FC5/RHEL4 into that kernel, or perhaps
patch the xen /dev/mem driver above to remove the local_lowmem()
restriction, or something like that...
Dave
>
>
> I'm pretty sure that the various files in /boot for dom0 all match the running system. Output from `crash -d2'
> attached in case its useful...
>
> So - the question is - what's the obvious thing I am missing!
>
> Thanks,
>
> Simon
>
> crash 4.0-2.24
>
> Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
>
> Copyright (C) 2004, 2005, 2006 IBM Corporation
>
> Copyright (C) 1999-2006 Hewlett-Packard Co
>
> Copyright (C) 2005 Fujitsu Limited
>
> Copyright (C) 2005 NEC Corporation
>
> Copyright (C) 1999, 2002 Silicon Graphics, Inc.
>
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>
> This program is free software, covered by the GNU General Public License,
>
> and you are welcome to change it and/or distribute copies of it under
>
> certain conditions. Enter "help copying" to see the conditions.
>
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> find_booted_kernel: search for [Linux version 2.6.16-xen0 (sgraham at anna.sn.stratus.com) (gcc version 3.4.4 20050721
> (Red Hat 3.4.4-2)) #3 Sat May 6 19:04:09 EDT 2006
>
> ]
>
> mount_points[0]: / (83a1ac0)
>
> mount_points[1]: /proc (83a1ad0)
>
> mount_points[2]: /dev (83a1ae0)
>
> mount_points[3]: / (83a1af0)
>
> mount_points[4]: /dev (83a1b00)
>
> mount_points[5]: /proc (83a1b10)
>
> mount_points[6]: /sys (83a1b20)
>
> mount_points[7]: /dev/pts (83a1b30)
>
> mount_points[8]: /boot (83a1b48)
>
> mount_points[9]: /dev/shm (83a1b58)
>
> searchdirs[9]: /usr/lib/debug/lib/modules/2.6.16-xen0/
>
> searchdirs[0]: /usr/src/linux/
>
> searchdirs[1]: /boot/
>
> searchdirs[2]: /boot/efi/redhat
>
> searchdirs[3]: /boot/efi/EFI/redhat
>
> searchdirs[4]: /
>
> searchdirs[5]: /usr/src/kernels/
>
> searchdirs[6]: /usr/src/debug/
>
> searchdirs[7]: /usr/src/redhat/BUILD/kernel-2.6.16/linux/
>
> searchdirs[8]: /usr/src/redhat/BUILD/kernel-2.6.16/linux-2.6.16/
>
> find_booted_kernel: check: /boot/xen-syms-3.0.2-2
>
> find_booted_kernel: check: /boot/vmlinux-syms-2.6.16-xenU
>
> find_booted_kernel: check: /boot/vmlinux-syms-2.6.16-xen0
>
> find_booted_kernel: found: /boot/vmlinux-syms-2.6.16-xen0
>
> get_live_memory_source: /dev/mem
>
> /proc/version:
>
> Linux version 2.6.16-xen0 (sgraham at anna.sn.stratus.com) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3 Sat May 6
> 19:04:09 EDT 2006
>
> /boot/vmlinux-syms-2.6.16-xen0:
>
> Linux version 2.6.16-xen0 (sgraham at anna.sn.stratus.com) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3 Sat May 6
> 19:04:09 EDT 2006
>
> gdb /boot/vmlinux-syms-2.6.16-xen0
>
> GNU gdb 6.1
>
> Copyright 2004 Free Software Foundation, Inc.
>
> GDB is free software, covered by the GNU General Public License, and you are
>
> welcome to change it and/or distribute copies of it under certain conditions.
>
> Type "show copying" to see the conditions.
>
> There is absolutely no warranty for GDB. Type "show warranty" for details.
>
> This GDB was configured as "i686-pc-linux-gnu"...
>
> WARNING: cannot read linux_banner string
>
> /proc/version:
>
> Linux version 2.6.16-xen0 (sgraham at anna.sn.stratus.com) (gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3 Sat May 6
> 19:04:09 EDT 2006
>
> linux_banner:
>
> crash: /boot/vmlinux-syms-2.6.16-xen0 and /dev/mem do not match!
>
> Usage:
>
> crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile] [namelist] [dumpfile]
>
> Enter "crash -h" for details.
>
> -----------------------------------------------------------------------------------------------------------
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20060508/8a802c3e/attachment.htm>
More information about the Crash-utility
mailing list