[Crash-utility] [PATCH v3 2/2] arm/arm64: read elf notes for qemu generated cores
Andrew Jones
drjones at redhat.com
Fri Nov 20 21:43:58 UTC 2015
On Fri, Nov 20, 2015 at 04:25:38PM -0500, Dave Anderson wrote:
>
>
> ----- Original Message -----
> > On Fri, Nov 20, 2015 at 03:18:55PM -0500, Dave Anderson wrote:
> > >
> > >
> > > ----- Original Message -----
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > QEMU can generate both non-makedumpfile (just elf) and makedumpfile
> > > > > formatted kdumps. In neither case will crash_notes have prstatus, as
> > > > > crash_kexec doesn't run in the kernel, however the elf notes will
> > > > > contain the prstatus, and we can dig them out of there.
> > > >
> > > > I don't have a lot of ARM and ARM64 dumpfiles, but just doing a
> > > > quick sanity test of your patch, I came across this ARM dumpfile,
> > > > which I believe may be a QEMU-generated ELF vmcore. I'm not sure,
> > > > but it only has 1 NT_PRSTATUS note for the 1 online cpu (of 5 cpus).
> >
> > If it's more than a day old then it won't be from qemu. I just posted
> > the patches for that yesterday morning :-)
>
> Even for 32-bit ARM? I honestly can't remember where/who it came from,
Yup. The qemu patches I posted are for both arm and aarch64. Before those
patches it wasn't possible to generate a dump with qemu for either.
> although it's in my notes somewhere. Because the non-panicking cpus
> are offline, it's possible that it's a kdump-generated dumpfile, and
> the other cpu's note were not captured. I don't know, but regardless,
> it turns out to be a damn good test case!
>
> >
> > > >
> > > > But anyway, note that as expected, it cannot find the registers in the
> > > > kernel's uninitialized crash_notes -- here without your patch:
> >
> > So it looks like there may be other dump types (besides qemu generated)
> > that can result in missing crash_notes. I don't know what those are, other
> > than just corruption? In any case, I think this dump is still a good test
> > case for the reason you found below.
> >
> > > >
> > > > $ crash vmcore.pae vmlinux.pae.gz
> > > >
> > > > crash 7.1.4rc15
> > > > Copyright (C) 2002-2014 Red Hat, Inc.
> > > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > > > Copyright (C) 1999-2006 Hewlett-Packard Co
> > > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > > > Copyright (C) 2005, 2011 NEC Corporation
> > > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > > > This program is free software, covered by the GNU General Public
> > > > License,
> > > > and you are welcome to change it and/or distribute copies of it under
> > > > certain conditions. Enter "help copying" to see the conditions.
> > > > This program has absolutely no warranty. Enter "help warranty" for
> > > > details.
> > > >
> > > > GNU gdb (GDB) 7.6
> > > > Copyright (C) 2013 Free Software Foundation, Inc.
> > > > License GPLv3+: GNU GPL version 3 or later
> > > > <http://gnu.org/licenses/gpl.html>
> > > > This is free software: you are free to change and redistribute it.
> > > > There is NO WARRANTY, to the extent permitted by law. Type "show
> > > > copying"
> > > > and "show warranty" for details.
> > > > This GDB was configured as "--host=x86_64-unknown-linux-gnu
> > > > --target=arm-elf-linux"...
> > > >
> > > > WARNING: invalid note (n_type != NT_PRSTATUS)
> > > > WARNING: cannot retrieve registers for active tasks
> > > >
> > > > KERNEL: vmlinux.pae.gz
> > > > DUMPFILE: vmcore.pae
> > > > CPUS: 5 [OFFLINE: 4]
> > > > DATE: Sun Jun 8 18:27:39 2014
> > > > UPTIME: 00:03:22
> > > > LOAD AVERAGE: 0.16, 0.16, 0.07
> > > > TASKS: 51
> > > > NODENAME: buildroot
> > > > RELEASE: 3.13.5
> > > > VERSION: #3 SMP Mon Jun 9 05:58:39 CST 2014
> > > > MACHINE: armv7l (unknown Mhz)
> > > > MEMORY: 256 MB
> > > > PANIC: "SysRq : Trigger a crash"
> > > > PID: 732
> > > > COMMAND: "sh"
> > > > TASK: 8bcead00 [THREAD_INFO: 8ad32000]
> > > > CPU: 0
> > > > STATE: TASK_RUNNING (SYSRQ)
> > > >
> > > > crash> bt -a
> > > > PID: 732 TASK: 8bcead00 CPU: 0 COMMAND: "sh"
> > > > bt: WARNING: cannot determine starting stack frame for task 8bcead00
> > > >
> > > > PID: 0 TASK: 8bc561c0 CPU: 1 COMMAND: "swapper/1"
> > > > bt: WARNING: cannot determine starting stack frame for task 8bc561c0
> > > >
> > > > PID: 0 TASK: 8bc56580 CPU: 2 COMMAND: "swapper/2"
> > > > bt: WARNING: cannot determine starting stack frame for task 8bc56580
> > > >
> > > > PID: 0 TASK: 8bc56940 CPU: 3 COMMAND: "swapper/3"
> > > > bt: WARNING: cannot determine starting stack frame for task 8bc56940
> > > >
> > > > PID: 0 TASK: 8bc56d00 CPU: 4 COMMAND: "swapper/4"
> > > > bt: WARNING: cannot determine starting stack frame for task 8bc56d00
> > > > crash>
> > > >
> > > >
> > > > With your patch applied, it generates a SIGSEGV in arm_get_crash_notes():
> > > >
> > > > $ ./crash /usr/dumps/ARM/vmcore.pae /usr/dumps/ARM/vmlinux.pae.gz
> > > >
> > > > crash 7.1.4rc15
> > > > Copyright (C) 2002-2014 Red Hat, Inc.
> > > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > > > Copyright (C) 1999-2006 Hewlett-Packard Co
> > > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > > > Copyright (C) 2005, 2011 NEC Corporation
> > > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > > > This program is free software, covered by the GNU General Public
> > > > License,
> > > > and you are welcome to change it and/or distribute copies of it under
> > > > certain conditions. Enter "help copying" to see the conditions.
> > > > This program has absolutely no warranty. Enter "help warranty" for
> > > > details.
> > > >
> > > > GNU gdb (GDB) 7.6
> > > > Copyright (C) 2013 Free Software Foundation, Inc.
> > > > License GPLv3+: GNU GPL version 3 or later
> > > > <http://gnu.org/licenses/gpl.html>
> > > > This is free software: you are free to change and redistribute it.
> > > > There is NO WARRANTY, to the extent permitted by law. Type "show
> > > > copying"
> > > > and "show warranty" for details.
> > > > This GDB was configured as "--host=x86_64-unknown-linux-gnu
> > > > --target=arm-elf-linux"...
> > > >
> > > > Segmentation fault (core dumped)
> > > > $
> > > >
> > > > I haven't debugged it other than determining that the "note" it looks to
> > > > have found the single note OK, but then upon continuation the next time
> > > > through the loop, the "note" pointer is valid at line 597, but your
> > > > function sets it back to NULL, and therefore it craps out at line 622:
> > > >
> > > > 597 note = (Elf32_Nhdr *)buf;
> > > > 598 p = buf + sizeof(Elf32_Nhdr);
> > > > 599
> > > > 600 /*
> > > > 601 * dumpfiles created with qemu won't have
> > > > crash_notes, but there will
> > > > 602 * be elf notes.
> > > > 603 */
> > > > 604 if (note->n_namesz == 0 && (DISKDUMP_DUMPFILE()
> > > > ||
> > > > KDUMP_DUMPFILE())) {
> > > > 605 if (DISKDUMP_DUMPFILE())
> > > > 606 note =
> > > > diskdump_get_prstatus_percpu(i);
> > > > 607 else if (KDUMP_DUMPFILE())
> > > > 608 note =
> > > > netdump_get_prstatus_percpu(i);
> > > > 609 if (note) {
> > > > 610 /*
> > > > 611 * SIZE(note_buf) accounts for a
> > > > "final note", which is a
> > > > 612 * trailing empty elf note
> > > > header.
> > > > 613 */
> > > > 614 long notesz = SIZE(note_buf) -
> > > > sizeof(Elf32_Nhdr);
> > > > 615
> > > > 616 if (sizeof(Elf32_Nhdr) +
> > > > roundup(note->n_namesz, 4) +
> > > > 617 note->n_descsz == notesz)
> > > > 618 BCOPY((char *)note, buf,
> > > > notesz);
> > > > 619 }
> > > > 620 }
> > > > 621
> > > > 622 if (note->n_type != NT_PRSTATUS) {
> > > > 623 error(WARNING, "invalid note (n_type !=
> > > > NT_PRSTATUS)\n");
> > > > 624 goto fail;
> > > > 625 }
> > > >
> > > > Not sure how you want to handle that, probably just bail out the same way
> > > > if note becomes NULL?
> > >
> > > If I add this to arm_get_crash_notes(), just after your new function:
> > >
> > > if (!note) {
> > > error(WARNING, "cannot find NT_PRSTATUS note for cpu: %d\n",
> > > i);
> > > continue;
> > > }
> > >
> > > I get this:
> > >
> > > $ ./crash /usr/dumps/ARM/vmcore.pae* /usr/dumps/ARM/vmlinux.pae.gz
> > >
> > > crash 7.1.4rc15
> > > Copyright (C) 2002-2014 Red Hat, Inc.
> > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > > Copyright (C) 1999-2006 Hewlett-Packard Co
> > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > > Copyright (C) 2005, 2011 NEC Corporation
> > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > > This program is free software, covered by the GNU General Public License,
> > > and you are welcome to change it and/or distribute copies of it under
> > > certain conditions. Enter "help copying" to see the conditions.
> > > This program has absolutely no warranty. Enter "help warranty" for
> > > details.
> > >
> > > GNU gdb (GDB) 7.6
> > > Copyright (C) 2013 Free Software Foundation, Inc.
> > > License GPLv3+: GNU GPL version 3 or later
> > > <http://gnu.org/licenses/gpl.html>
> > > This is free software: you are free to change and redistribute it.
> > > There is NO WARRANTY, to the extent permitted by law. Type "show
> > > copying"
> > > and "show warranty" for details.
> > > This GDB was configured as "--host=x86_64-unknown-linux-gnu
> > > --target=arm-elf-linux"...
> > >
> > > WARNING: cannot find NT_PRSTATUS note for cpu: 1
> > > WARNING: cannot find NT_PRSTATUS note for cpu: 2
> > > WARNING: cannot find NT_PRSTATUS note for cpu: 3
> > > WARNING: cannot find NT_PRSTATUS note for cpu: 4
> > > KERNEL: /usr/dumps/ARM/vmlinux.pae.gz
> > > DUMPFILE: /usr/dumps/ARM/vmcore.pae
> > > CPUS: 5 [OFFLINE: 4]
> > > DATE: Sun Jun 8 18:27:39 2014
> > > UPTIME: 00:03:22
> > > LOAD AVERAGE: 0.16, 0.16, 0.07
> > > TASKS: 51
> > > NODENAME: buildroot
> > > RELEASE: 3.13.5
> > > VERSION: #3 SMP Mon Jun 9 05:58:39 CST 2014
> > > MACHINE: armv7l (unknown Mhz)
> > > MEMORY: 256 MB
> > > PANIC: "SysRq : Trigger a crash"
> > > PID: 732
> > > COMMAND: "sh"
> > > TASK: 8bcead00 [THREAD_INFO: 8ad32000]
> > > CPU: 0
> > > STATE: TASK_RUNNING (SYSRQ)
> > >
> > > crash> bt -a
> > > PID: 732 TASK: 8bcead00 CPU: 0 COMMAND: "sh"
> > > #0 [<80265064>] (sysrq_handle_crash) from [<80265810>]
> > > #1 [<80265810>] (__handle_sysrq) from [<80265928>]
> > > #2 [<80265928>] (write_sysrq_trigger) from [<80112120>]
> > > #3 [<80112120>] (proc_reg_write) from [<800c9840>]
> > > #4 [<800c9840>] (vfs_write) from [<800c9be4>]
> > > #5 [<800c9be4>] (sys_write) from [<8000e3e0>]
> > > pc : [<76e9cfdc>] lr : [<0000f998>] psr: 600d0010
> > > sp : 7eab862c ip : 00000000 fp : 000a82a4
> > > r10: 00000020 r9 : 000a8294 r8 : 00000001
> > > r7 : 00000004 r6 : 000a9bf0 r5 : 00000001 r4 : 000a7d88
> > > r3 : 00000000 r2 : 00000002 r1 : 000a9bf0 r0 : 00000001
> > > Flags: nZCv IRQs on FIQs on Mode USER_32 ISA ARM
> > >
> > > PID: 0 TASK: 8bc561c0 CPU: 1 COMMAND: "swapper/1"
> > > bt: WARNING: cannot determine starting stack frame for task 8bc561c0
> > >
> > > PID: 0 TASK: 8bc56580 CPU: 2 COMMAND: "swapper/2"
> > > bt: WARNING: cannot determine starting stack frame for task 8bc56580
> > >
> > > PID: 0 TASK: 8bc56940 CPU: 3 COMMAND: "swapper/3"
> > > bt: WARNING: cannot determine starting stack frame for task 8bc56940
> > >
> > > PID: 0 TASK: 8bc56d00 CPU: 4 COMMAND: "swapper/4"
> > > bt: WARNING: cannot determine starting stack frame for task 8bc56d00
> > > crash>
> > >
> > > Note that if I did "goto fail" instead of "continue", I lose the good
> > > cpu 0 backtrace from the NT_PRSTATUS that your patch found, so doing it
> > > this way is the best of both worlds.
> >
> > I agree that the 'if (!note) continue' that you added is a good idea to
> > try and salvage this type of dump. It shouldn't happen with qemu generated
> > dumps, but anything's possible when a kernel panics...
> >
> > Would you like me to spin a v4 with this condition added? Or, since it
> > actually seems to be addressing a non-qemu-generated dump issue, then
> > maybe you just want to submit it as a new patch on top of the qemu
> > patch?
>
> No, I'm cleaning it up now -- there are a couple other gotcha's w/respect
> to backtracing, since we're now going to keep the ms->panic_task_regs[cpus]
> array intact, even if there's no note in some of them. I'm going to make
> sure it's zero-filled in the not-found case, and I have to make sure that
> it's recognized on both architectures. I'll post the the updated additions
> to arm.c and arm64.c so that you can do a quick test on a legitimate
> QEMU dump.
>
> Might be Monday, I'm not sure...
Sounds good to me :-)
Thanks,
drew
More information about the Crash-utility
mailing list