[Crash-utility] Problems debugging live system with 4.0-2.24 crash

Mon May 8 14:59:37 UTC 2006

Thanks for the info Dave - maybe this belongs in the appropriate Xen
list, but this code seems fundamentally broken to me for a couple of
reasons:

1.	I'm pretty sure that the address passed into read_mem is a
physical address NOT a machine-address and yet the code in
is_local_lowmem seems to treat this as an mfn (at least, I would assume
that mfn_to_local_pfn would be doing so)
2.	That aside, it just seems wrong to me to break the operation of
/dev/mem so that the one thing it _cant_ return is low memory for the
guest!

Time for a Xen bug I think!

Simon

________________________________

From: crash-utility-bounces at redhat.com
[mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
Sent: Monday, May 08, 2006 10:36 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] Problems debugging live system with
4.0-2.24 crash

"Graham, Simon" wrote: 

I've been experimenting with the latest crash version on a system
running Xen (3.0.2-2 with kernel 2.6.16) and am experiencing problems
when attempting to debug the system live; debugging of saved guest files
and guest crash dumps works well, it's just live debugging of dom0 that
is problematical. 

The symptom is that I get the error: 

WARNING: cannot read linux_banner string 

When I debug crash, I can see that it is finding the correct unstripped
kernel from which it finds the address of the linux_banner variable
_but_ when it attempts to read the string from /dev/mem it gets all
zeroes back. I have confirmed that the address is correct by using gdb
against /proc/kcore - gdb reads the string quite nicely from the virtual
address (happens to be 0xc0450000, I can see that crash translates this
to physical address 450000 and gets all zeroes when it reads this
location from /dev/mem).

Hi Graham, 

The xen /dev/mem driver read_mem() function does this: 

                if ((v = ioremap(p, sz)) == NULL) { 
                        /* 
                         * Some programs (e.g., dmidecode) groove off
into weird RAM 
                         * areas where no tables can possibly exist
(because Xen will 
                         * have stomped on them!). These programs get
rather upset if 
                         * we let them know that Xen failed their
access, so we fake 
                         * out a read of all zeroes. :-) 
                         */ 
                        if (clear_user(buf, count)) 
                                return -EFAULT; 
                        read += count; 
                        break; 
                } 

when ioremap() returns NULL.  Here's the NULL-returner in __ioremap(): 

        /* 
         * Don't allow anybody to remap normal RAM that we're using.. 
         */ 
        if (is_local_lowmem(phys_addr)) { 
                char *t_addr, *t_end; 
                struct page *page; 

                t_addr = bus_to_virt(phys_addr); 
                t_end = t_addr + (size - 1); 

                for(page = virt_to_page(t_addr); page <=
virt_to_page(t_end); page++) 
                        if(!PageReserved(page)) 
                                return NULL; 

                domid = DOMID_SELF; 
        } 

So /dev/mem cannot be used.  So to get crash to work on a live 
x86 xen0 system (FC5), I've had to use the RHEL4 crash.c module 
(/dev/crash) to make an end-run around /dev/mem.  RHEL4's x86 and 
x86_64 /dev/mem driver only allows reads of the first 256 pages of 
physical memory.  And now Xen has the restriction above. 

FC5 has a CONFIG_CRASH configurable: 

$ diff kernel-2.6.15-i686-guest.config.orig
kernel-2.6.15-i686-guest.config 
2526c2526 
< # CONFIG_CRASH is not set 
--- 
> CONFIG_CRASH=m 
2965d2964 
< # CONFIG_CRASH is not set 
$ 

If you're building from an upstream source, you could build the 
crash.c module from FC5/RHEL4 into that kernel, or perhaps 
patch the xen /dev/mem driver above to remove the local_lowmem() 
restriction, or something like that... 

Dave 

	I'm pretty sure that the various files in /boot for dom0 all
match the running system. Output from `crash -d2' attached in case its
useful... 

	So - the question is - what's the obvious thing I am missing! 

	Thanks, 

	Simon 

	crash 4.0-2.24 

	Copyright (C) 2002, 2003, 2004, 2005, 2006  Red Hat, Inc. 

	Copyright (C) 2004, 2005, 2006  IBM Corporation 

	Copyright (C) 1999-2006  Hewlett-Packard Co 

	Copyright (C) 2005  Fujitsu Limited 

	Copyright (C) 2005  NEC Corporation 

	Copyright (C) 1999, 2002  Silicon Graphics, Inc. 

	Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux,
Inc. 

	This program is free software, covered by the GNU General Public
License, 

	and you are welcome to change it and/or distribute copies of it
under 

	certain conditions.  Enter "help copying" to see the conditions.

	This program has absolutely no warranty.  Enter "help warranty"
for details. 

	find_booted_kernel: search for [Linux version 2.6.16-xen0
(sgraham at anna.sn.stratus.com) (gcc version 3.4.4 20050721 (Red Hat
3.4.4-2)) #3 Sat May 6 19:04:09 EDT 2006 

	] 

	mount_points[0]: / (83a1ac0) 

	mount_points[1]: /proc (83a1ad0) 

	mount_points[2]: /dev (83a1ae0) 

	mount_points[3]: / (83a1af0) 

	mount_points[4]: /dev (83a1b00) 

	mount_points[5]: /proc (83a1b10) 

	mount_points[6]: /sys (83a1b20) 

	mount_points[7]: /dev/pts (83a1b30) 

	mount_points[8]: /boot (83a1b48) 

	mount_points[9]: /dev/shm (83a1b58) 

	searchdirs[9]: /usr/lib/debug/lib/modules/2.6.16-xen0/ 

	searchdirs[0]: /usr/src/linux/ 

	searchdirs[1]: /boot/ 

	searchdirs[2]: /boot/efi/redhat 

	searchdirs[3]: /boot/efi/EFI/redhat 

	searchdirs[4]: / 

	searchdirs[5]: /usr/src/kernels/ 

	searchdirs[6]: /usr/src/debug/ 

	searchdirs[7]: /usr/src/redhat/BUILD/kernel-2.6.16/linux/ 

	searchdirs[8]: /usr/src/redhat/BUILD/kernel-2.6.16/linux-2.6.16/

	find_booted_kernel: check: /boot/xen-syms-3.0.2-2 

	find_booted_kernel: check: /boot/vmlinux-syms-2.6.16-xenU 

	find_booted_kernel: check: /boot/vmlinux-syms-2.6.16-xen0 

	find_booted_kernel: found: /boot/vmlinux-syms-2.6.16-xen0 

	get_live_memory_source: /dev/mem 

	/proc/version: 

	Linux version 2.6.16-xen0 (sgraham at anna.sn.stratus.com) (gcc
version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3 Sat May 6 19:04:09 EDT 2006

	/boot/vmlinux-syms-2.6.16-xen0: 

	Linux version 2.6.16-xen0 (sgraham at anna.sn.stratus.com) (gcc
version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3 Sat May 6 19:04:09 EDT 2006

	gdb /boot/vmlinux-syms-2.6.16-xen0  

	GNU gdb 6.1 

	Copyright 2004 Free Software Foundation, Inc. 

	GDB is free software, covered by the GNU General Public License,
and you are 

	welcome to change it and/or distribute copies of it under
certain conditions. 

	Type "show copying" to see the conditions. 

	There is absolutely no warranty for GDB.  Type "show warranty"
for details. 

	This GDB was configured as "i686-pc-linux-gnu"... 

	WARNING: cannot read linux_banner string 

	/proc/version: 

	Linux version 2.6.16-xen0 (sgraham at anna.sn.stratus.com) (gcc
version 3.4.4 20050721 (Red Hat 3.4.4-2)) #3 Sat May 6 19:04:09 EDT 2006

	linux_banner: 

	crash: /boot/vmlinux-syms-2.6.16-xen0 and /dev/mem do not match!

	Usage: 

	  crash [-h [opt]][-v][-s][-i file][-d num] [-S] [mapfile]
[namelist] [dumpfile] 

	Enter "crash -h" for details. 

________________________________

	--
	Crash-utility mailing list
	Crash-utility at redhat.com
	https://www.redhat.com/mailman/listinfo/crash-utility

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20060508/9880365f/attachment.htm>