[Fwd: [Crash-utility] crash warnings]
David Wilder
dwilder at us.ibm.com
Thu May 4 18:47:03 UTC 2006
Dave Anderson wrote:
> David Wilder wrote:
>
>> Dave Anderson wrote:
>>
>> >Badari Pulavarty wrote:
>> >
>> >
>> >
>> >>...
>> >>
>> >>
>> >>
>> >>>>original non-g built vmlinux, things would work.
>> >>>>
>> >>>>In your case, it's defaulting to the /usr/src/linux/System.map,
>> which is
>> >>>>associated with the vmlinux that you built with -g.
>> >>>>
>> >>>>Presuming you have the /boot/System.map associated with the original
>> >>>>distro's vmlinux file (the kernel that's running), what happens
>> if you do
>> >>>>this:
>> >>>>
>> >>>>$ crash vmlinux /boot/System.map [vmcore]
>> >>>>
>> >>>>
>> >>Much better with /boot/System.map.
>> >>
>> >>
>> >>
>> >
>> >Yay!
>> >
>> >So, we seem to have resolved the funky text disassembly issue...
>> >
>> >But I still would like to continue working with the vmcore -- I'm
>> sitting here
>> >with Vivek, and he will grab a copy of the System.map.
>> >
>> >Thanks, (and phew...)
>> >
>> >Dave
>> >
>> >
>> >
>> >
>> >>elm3a242:/usr/src/linux # /root/cr*23/crash vmlinux
>> /boot/System.map-2.6.16-20-smp
>> >>
>> >>crash 4.0-2.23
>> >>Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
>> >>Copyright (C) 2004, 2005, 2006 IBM Corporation
>> >>Copyright (C) 1999-2006 Hewlett-Packard Co
>> >>Copyright (C) 2005 Fujitsu Limited
>> >>Copyright (C) 2005 NEC Corporation
>> >>Copyright (C) 1999, 2002 Silicon Graphics, Inc.
>> >>Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> >>This program is free software, covered by the GNU General Public
>> >>License,
>> >>and you are welcome to change it and/or distribute copies of it under
>> >>certain conditions. Enter "help copying" to see the conditions.
>> >>This program has absolutely no warranty. Enter "help warranty" for
>> >>details.
>> >>
>> >>GNU gdb 6.1
>> >>Copyright 2004 Free Software Foundation, Inc.
>> >>GDB is free software, covered by the GNU General Public License,
>> and you
>> >>are
>> >>welcome to change it and/or distribute copies of it under certain
>> >>conditions.
>> >>Type "show copying" to see the conditions.
>> >>There is absolutely no warranty for GDB. Type "show warranty" for
>> >>details.
>> >>This GDB was configured as "x86_64-unknown-linux-gnu"...
>> >>
>> >> SYSTEM MAP: /boot/System.map-2.6.16-20-smp
>> >>DEBUG KERNEL: vmlinux (2.6.16-20-smp)
>> >> DUMPFILE: /dev/mem
>> >> CPUS: 2
>> >> DATE: Thu Apr 27 03:18:57 2006
>> >> UPTIME: 16:47:17
>> >>LOAD AVERAGE: 0.00, 0.00, 0.00
>> >> TASKS: 64
>> >> NODENAME: elm3a242
>> >> RELEASE: 2.6.16-20-smp
>> >> VERSION: #1 SMP Mon Apr 10 04:51:13 UTC 2006
>> >> MACHINE: x86_64 (3000 Mhz)
>> >> MEMORY: 4.6 GB
>> >> PID: 12963
>> >> COMMAND: "crash"
>> >> TASK: ffff810121ef5790 [THREAD_INFO: ffff81011d3c4000]
>> >> CPU: 1
>> >> STATE: TASK_RUNNING (ACTIVE)
>> >>
>> >>crash> dis sys_read
>> >>0xffffffff8017b751 <sys_read>: push %r13
>> >>0xffffffff8017b753 <sys_read+2>: mov %rsi,%r13
>> >>0xffffffff8017b756 <sys_read+5>: push %r12
>> >>0xffffffff8017b758 <sys_read+7>: mov
>> $0xfffffffffffffff7,%r12
>> >>0xffffffff8017b75f <sys_read+14>: push %rbp
>> >>0xffffffff8017b760 <sys_read+15>: mov %rdx,%rbp
>> >>0xffffffff8017b763 <sys_read+18>: push %rbx
>> >>0xffffffff8017b764 <sys_read+19>: sub $0x18,%rsp
>> >>0xffffffff8017b768 <sys_read+23>: lea 0x14(%rsp),%rsi
>> >>0xffffffff8017b76d <sys_read+28>: callq 0xffffffff8017bce3
>> >><fget_light>
>> >>0xffffffff8017b772 <sys_read+33>: test %rax,%rax
>> >>0xffffffff8017b775 <sys_read+36>: mov %rax,%rbx
>> >>0xffffffff8017b778 <sys_read+39>:
>> >> je 0xffffffff8017b7b1 <sys_read+96>
>> >>0xffffffff8017b77a <sys_read+41>: mov 0x38(%rax),%rax
>> >>0xffffffff8017b77e <sys_read+45>: lea 0x8(%rsp),%rcx
>> >>0xffffffff8017b783 <sys_read+50>: mov %rbp,%rdx
>> >>0xffffffff8017b786 <sys_read+53>: mov %r13,%rsi
>> >>0xffffffff8017b789 <sys_read+56>: mov %rbx,%rdi
>> >>0xffffffff8017b78c <sys_read+59>: mov %rax,0x8(%rsp)
>> >>0xffffffff8017b791 <sys_read+64>: callq 0xffffffff8017b2ef
>> >><vfs_read>
>> >>0xffffffff8017b796 <sys_read+69>: mov %rax,%r12
>> >>0xffffffff8017b799 <sys_read+72>: mov 0x8(%rsp),%rax
>> >>
>> >>--
>> >>Crash-utility mailing list
>> >>Crash-utility at redhat.com
>> >>https://www.redhat.com/mailman/listinfo/crash-utility
>> >>
>> >>
>> >
>> >--
>> >Crash-utility mailing list
>> >Crash-utility at redhat.com
>> >https://www.redhat.com/mailman/listinfo/crash-utility
>> >
>> >
>> >
>> Dave I am seeing a similar problem to Badari, however in my case I am
>> running the kernel built with -g.
>> I can run crash live fine. But running on a vmcore I get the following
>> error.
>>
>> deimos:/var/log/dump/2006-05-02-13:10 # ~/crash-4.0-2.24/crash
>> /usr/src/linux/vmlinux ./vmcore
>>
>> crash 4.0-2.24
>> Copyright (C) 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
>> Copyright (C) 2004, 2005, 2006 IBM Corporation
>> Copyright (C) 1999-2006 Hewlett-Packard Co
>> Copyright (C) 2005 Fujitsu Limited
>> Copyright (C) 2005 NEC Corporation
>> Copyright (C) 1999, 2002 Silicon Graphics, Inc.
>> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> This program is free software, covered by the GNU General Public
>> License,
>> and you are welcome to change it and/or distribute copies of it under
>> certain conditions. Enter "help copying" to see the conditions.
>> This program has absolutely no warranty. Enter "help warranty" for
>> details.
>>
>> GNU gdb 6.1
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and
>> you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB. Type "show warranty" for
>> details.
>> This GDB was configured as "powerpc64-unknown-linux-gnu"...
>>
>> Wilder vmlist = 0xc00000000fa26940
>> crash: read error: kernel virtual address: c00000000fa26940 type: "first
>> vmlist
>> addr"
>>
>> Errors like the one above typically occur when the kernel and memory
>> source
>> do not match. These are the files being used:
>>
>> KERNEL: /usr/src/linux/vmlinux
>> DUMPFILE: ./vmcore
>>
>> --
>> When I check System.map for vmlist or do a "nm vmlinux | grep vmlist" I
>> get a different number than shown above.
>> I tried specifying the System.map file on the command line but no
>> difference.
>>
>>
> Hi Dave,
>
> Like Badari's issue, which actually was a non-issue, it's
> difficult to debug from here.
>
> A couple things...
>
> When you use an additional System.map argument on the command
> line, you should see the message stating that several thousand
> symbol values are being changed from what is in the vmlinux file
> to what is in the System.map file:
>
> please wait... (patching ##### gdb minimal_symbol values)
>
> This gets kicked off in the gdb_interface.c gdb_session_init() function
> here:
>
> /*
> * Patch gdb's symbol values with the correct values from either
> * the System.map or non-debug vmlinux, whichever is in effect.
> */
> if ((pc->flags & SYSMAP) ||
> (pc->namelist_debug && !pc->debuginfo_file)) {
> req->command = GNU_PATCH_SYMBOL_VALUES;
> req->flags = GNU_RETURN_ON_ERROR;
> gdb_interface(req);
> if (req->flags & GNU_COMMAND_FAILED)
> error(FATAL, "patching of gdb symbol values
> failed\n");
> }
>
> So, presuming that worked OK, what's strange is the read error
> you get on the vmcore:
>
>>crash: read error: kernel virtual address: c00000000fa26940 type: "first vmlist addr"
>>
>
> The first_vmalloc_address() function is simply reading
> reading the contents of the vmlist pointer, which points
> to the first vm_struct in the list, and which is returning
> c00000000fa26940. But then it fails to read what's in the
> "addr" offset of the vm_struct at that address:
>
> get_symbol_data("vmlist", sizeof(void *), &vmlist);
>
> if (!readmem(vmlist+OFFSET(vm_struct_addr), KVADDR, &addr,
> sizeof(void *), "first vmlist addr", RETURN_ON_ERROR))
> non_matching_kernel();
>
> Since the "addr" offset is 0, it just reads address
> c00000000fa26940, which is a just a kmalloc'd unity-mapped
> address.
>
> So the question is, why is that address not capable of being
> read from the vmcore? If you do a "help -n" to dump the
> header contents of the vmcore, does physical address fa26940
> not fit into any of the PT_LOAD segments?
>
> (Again -- this all presumes the the "vmlist" symbol address is
> correct...)
>
> Dave
>
>
>
>------------------------------------------------------------------------
>
>--
>Crash-utility mailing list
>Crash-utility at redhat.com
>https://www.redhat.com/mailman/listinfo/crash-utility
>
>
How do I do a help -n if crash wont start?
--
David Wilder
IBM Linux Technology Center
Beaverton, Oregon, USA
dwilder at us.ibm.com
(503)578-3789
More information about the Crash-utility
mailing list