[Crash-utility] [PATCH] ppc64: fix 'bt' command for vmcore captured with fadump.
Dave Anderson
anderson at redhat.com
Tue Jan 24 18:23:47 UTC 2017
----- Original Message -----
>
>
> On Monday 23 January 2017 11:43 PM, Dave Anderson wrote:
> >
> > ----- Original Message -----
> >>
> >> On Saturday 21 January 2017 02:00 AM, Dave Anderson wrote:
> >>> ----- Original Message -----
> >>>
> >>> ... [cut] ...
> >>>
> >>>>> Also, the exception frame doesn't even show the [bracketed] type of
> >>>>> exception
> >>>>> that occurred -- it's just a register dump followed by the remainder of
> >>>>> the
> >>>>> backtrace. Upon a quick glance, it's not obvious that they are even
> >>>>> active
> >>>>> tasks. And traditionally, all of the other architectures have always
> >>>>> dumped
> >>>>> a full trace.
> >>>>>
> >>>>> I'm not sure what the mechanism is for shutting down the non-active
> >>>>> FADUMP tasks, so that's why I asked if you could restrict this change
> >>>>> to just those types of dumps. (For that matter, is it even possible to
> >>>>> differentiate a real kdump from an FADUMP dumpfile -- aside from a
> >>>> Hi Dave,
> >>>>
> >>>> Differentiating a kdump and fadump dumpfile is not possible except that
> >>>> the
> >>>> stack search would invariably fail and ptregs are guaranteed to be saved
> >>>> by
> >>>> firmware in case of fadump. Posted v2 that doesn't change bt output for
> >>>> anything
> >>>> but active tasks in case of fadump..
> >>> Ok, so let me get this straight. The only difference I see with the v2
> >>> patch
> >>> is that fadump non-panicking active tasks change from something like
> >>> this:
> >>>
> >>> PID: 0 TASK: c000000000e7f6d0 CPU: 0 COMMAND: "swapper"
> >>> #0 [c000000000f2ba30] (null) at 3aae291c67 (unreliable)
> >>> #1 [c000000000f2bae0] .tick_dev_program_event at c0000000000d16fc
> >>> #2 [c000000000f2bb90] .__hrtimer_start_range_ns at c0000000000c4bcc
> >>> #3 [c000000000f2bcb0] .tick_nohz_stop_sched_tick at c0000000000d2d30
> >>> #4 [c000000000f2bdc0] .cpu_idle at c000000000015bf0
> >>> #5 [c000000000f2be70] .rest_init at c000000000009de4
> >>> #6 [c000000000f2bef0] .start_kernel at c000000000850eb4
> >>> #7 [c000000000f2bf90] .start_here_common at c0000000000083d8
> >>>
> >>> to this:
> >>>
> >>> PID: 0 TASK: c000000000e7f6d0 CPU: 0 COMMAND: "swapper"
> >>> #0 [c000000000f2bd50] (null) at 0 (unreliable)
> >>> #1 [c000000000f2bdc0] .cpu_idle at c000000000015bf0
> >>> #2 [c000000000f2be70] .rest_init at c000000000009de4
> >>> #3 [c000000000f2bef0] .start_kernel at c000000000850eb4
> >>> #4 [c000000000f2bf90] .start_here_common at c0000000000083d8
> >>>
> >>> But with your v1 patch, you also dumped the exception frame:
> >>>
> >>> PID: 0 TASK: c000000000e7f6d0 CPU: 0 COMMAND: "swapper"
> >>> R0: 0000000000000000 R1: c000000000f2bd50 R2:
> >>> c000000000f27628
> >>> R3: 0000000000000000 R4: 0000000000000000 R5:
> >>> 8000000002144400
> >>> R6: 800000001314c4f8 R7: 0000000000000000 R8:
> >>> 0000000000000000
> >>> R9: ffffffffffffffff R10: 0000000000000000 R11:
> >>> 80003fbff901700c
> >>> R12: 0000000000000000 R13: c000000000ff2500 R14:
> >>> 0000000001a3fa58
> >>> R15: 00000000002230a8 R16: 0000000000223150 R17:
> >>> 0000000000223144
> >>> R18: 0000000000c8a098 R19: 0000000002b13a58 R20:
> >>> 0000000000000000
> >>> R21: 0000000002b135d8 R22: 0000000002b13530 R23:
> >>> 0000000002280000
> >>> R24: 0000000002b135f0 R25: c000000000fd5c48 R26:
> >>> c0000000010942f0
> >>> R27: c0000000010942f0 R28: c0000000005fd168 R29:
> >>> 0000000000000008
> >>> R30: c000000000eb1d68 R31: c000000000f28080
> >>> NIP: c000000000055730 MSR: 8000000000009032 OR3:
> >>> 0000000000000000
> >>> CTR: 0000000000000000 LR: c000000000057350 XER:
> >>> 0000000000000000
> >>> CCR: 0000000024000048 MQ: 0000000000000000 DAR:
> >>> 000001000ad763b0
> >>> DSISR: 0000000000000000 Syscall Result: 0000000000000000
> >>> NIP [c000000000055730] .plpar_hcall_norets
> >>> LR [c000000000057350] .pseries_shared_idle_sleep
> >>> #0 [c000000000f2bd50] (null) at 0 (unreliable)
> >>> #1 [c000000000f2bdc0] .cpu_idle at c000000000015bf0
> >>> #2 [c000000000f2be70] .rest_init at c000000000009de4
> >>> #3 [c000000000f2bef0] .start_kernel at c000000000850eb4
> >>> #4 [c000000000f2bf90] .start_here_common at c0000000000083d8
> >>>
> >>> Again, I don't understand how the non-panicking active tasks are stopped
> >>> by the fadump facility, but is it because you cannot differentiate kdumps
> >>> from fadumps that you don't show the exception frame with the v2 patch?
> >> Hi Dave,
> >>
> >> The crashing cpu makes rtas call ibm,os-term to the firmware which
> >> saves the regs info of all online cpus. AFAIK, there is no exception frame
> >> marker (which we are using to detect one) set for this stack frames
> >> by the kernel. With v1, I was printing the registers without looking for
> >> exception frame marker, if the registers are saved..
> >>
> >>> Would it be possible to also show the exception frame type in brackets
> >>> and
> >>> the register dump for those fadump non-panicking active tasks?
> >>>
> >> Hmmm.. Let me have a hard look at this.
> >> Will try and improve this..
> > Hari,
> >
> > I was tinkering around with ppc64_get_dumpfile_stack_frame() from your v2
> > patch,
> > and this seems to work:
> >
> > else {
> > *ksp = pt_regs->gpr[1];
> > if (IS_KVADDR(*ksp)) {
> > readmem(*ksp+16, KVADDR, nip, sizeof(ulong),
> > "Regs NIP value", FAULT_ON_ERROR);
> > + ppc64_print_regs(pt_regs);
> > return TRUE;
> > } else {
> > if (IN_TASK_VMA(bt_in->task, *ksp))
> > fprintf(fp, "%0lx: Task is running in user
> > space\n",
> > bt_in->task);
> > else
> > fprintf(fp, "%0lx: Invalid Stack Pointer
> > %0lx\n",
> > bt_in->task, *ksp);
> > *nip = pt_regs->nip;
> > ppc64_print_regs(pt_regs);
> > return FALSE;
> > }
> > }
> >
> > And if the task were to have been running in userspace, it already dumps
> > the
> > registers in the "else" section above.
> >
> > I see that the regs->trap is 0, so I understand now that there's nothing to
> > translate w/respect to the exception frame type, but a follow-up
> > translation
> > of the NIP and LR would at least show that there was some kind of hypercall
> > involved. (Whether it can be firmly determined whether FADUMP was
> > responsible
> > is another question)
> >
> >
>
> Hi Dave,
>
> I did think of it but I was wary considering two register prints like below,
> if there is an exception frame..
>
> PID: 2121 TASK: c0000001af90c600 CPU: 2 COMMAND: "sshd"
> R0: c0000000003e5280 R1: c0000001ae047a30 R2:
> c000000000fd5a00
> R3: 0000000000000001 R4: 000000000000019e R5:
> 000000000000000f
> R6: 0000000000000004 R7: c0000001ae047bb8 R8:
> 00000000000b3d9f
> R9: 00000000000000f0 R10: 0000000000000678 R11:
> c0000000008e0f38
> R12: c0000000003e6310 R13: c00000000b781200 R14:
> 0000000000000000
> R15: 0000000000000000 R16: 000001000b7dad70 R17:
> 000000005dfd3c08
> R18: 000000005dfd2838 R19: 00003ffff81eb620 R20:
> 000000005df74128
> R21: 000001000b7d89a0 R22: 000000000000de4c R23:
> 000000005df73b30
> R24: 000000005dfd3c88 R25: 00003ffff81eb428 R26:
> c0000001ae047bb8
> R27: c0000001b17f4d80 R28: c000000000c60580 R29:
> 000000000000019e
> R30: 000000000000000f R31: 000000000000090b
> NIP: 00003fffb6ac8400 MSR: 800000000000d033 OR3:
> 0000000000000000
> CTR: c0000000003e6310 LR: c0000000003e493c XER:
> 0000000020000000
> CCR: 0000000024004824 MQ: 0000000000000000 DAR:
> 000001000b7e1640
> DSISR: 0000000002000000 Syscall Result: 0000000000000000
> #0 [c0000001ae047a30] (null) at c0000000fd783c00 (unreliable)
> #1 [c0000001ae047a70] avc_has_perm at c0000000003e5280
> #2 [c0000001ae047b60] sock_has_perm at c0000000003e6238
> #3 [c0000001ae047be0] security_socket_sendmsg at c0000000003e28fc
> #4 [c0000001ae047c30] sock_sendmsg at c00000000072d53c
> #5 [c0000001ae047c60] sock_write_iter at c00000000072d644
> #6 [c0000001ae047d00] __vfs_write at c0000000002ed97c
> #7 [c0000001ae047d90] vfs_write at c0000000002ef328
> #8 [c0000001ae047de0] sys_write at c0000000002f0f00
> #9 [c0000001ae047e30] system_call at c00000000000b184
> System Call [c00] exception frame:
> R0: 0000000000000004 R1: 00003ffff81eb220 R2:
> 00003fffb6b99800
> R3: 0000000000000003 R4: 000001000b80e3c0 R5:
> 0000000000000034
> R6: 00003ffff81eb2e4 R7: 000000000000021e R8:
> 0000000000000000
> R9: 0000000000000000 R10: 0000000000000000 R11:
> 0000000000000000
> R12: 0000000000000000 R13: 00003fffb6497730 R14:
> 0000000000000000
> R15: 0000000000000000 R16: 000001000b7dad70 R17:
> 000000005dfd3c08
> R18: 000000005dfd2838 R19: 00003ffff81eb620 R20:
> 000000005df74128
> R21: 000001000b7d89a0 R22: 000000000000de4c R23:
> 000000005df73b30
> R24: 000000005dfd3c88 R25: 00003ffff81eb428 R26:
> 00003ffff81eb430
> R27: 00003ffff81eb420 R28: 00003ffff81eb424 R29:
> 00003ffff81eb2e4
> R30: 000001000b80e3c0 R31: 0000000000000034
> NIP: 00003fffb6ac8400 MSR: 800000000000d033 OR3:
> 0000000000000003
> CTR: 0000000000000000 LR: 000000005df1c3e4 XER:
> 0000000000000000
> CCR: 0000000044004824 MQ: 0000000000000001 DAR:
> 00003fffb729c590
> DSISR: 000000000a000000 Syscall Result: 0000000000000000
>
>
> instead of this..
>
> PID: 2121 TASK: c0000001af90c600 CPU: 2 COMMAND: "sshd"
> #0 [c0000001ae047a30] (null) at c0000000fd783c00 (unreliable)
> #1 [c0000001ae047a70] avc_has_perm at c0000000003e5280
> #2 [c0000001ae047b60] sock_has_perm at c0000000003e6238
> #3 [c0000001ae047be0] security_socket_sendmsg at c0000000003e28fc
> #4 [c0000001ae047c30] sock_sendmsg at c00000000072d53c
> #5 [c0000001ae047c60] sock_write_iter at c00000000072d644
> #6 [c0000001ae047d00] __vfs_write at c0000000002ed97c
> #7 [c0000001ae047d90] vfs_write at c0000000002ef328
> #8 [c0000001ae047de0] sys_write at c0000000002f0f00
> #9 [c0000001ae047e30] system_call at c00000000000b184
> System Call [c00] exception frame:
> R0: 0000000000000004 R1: 00003ffff81eb220 R2:
> 00003fffb6b99800
> R3: 0000000000000003 R4: 000001000b80e3c0 R5:
> 0000000000000034
> R6: 00003ffff81eb2e4 R7: 000000000000021e R8:
> 0000000000000000
> R9: 0000000000000000 R10: 0000000000000000 R11:
> 0000000000000000
> R12: 0000000000000000 R13: 00003fffb6497730 R14:
> 0000000000000000
> R15: 0000000000000000 R16: 000001000b7dad70 R17:
> 000000005dfd3c08
> R18: 000000005dfd2838 R19: 00003ffff81eb620 R20:
> 000000005df74128
> R21: 000001000b7d89a0 R22: 000000000000de4c R23:
> 000000005df73b30
> R24: 000000005dfd3c88 R25: 00003ffff81eb428 R26:
> 00003ffff81eb430
> R27: 00003ffff81eb420 R28: 00003ffff81eb424 R29:
> 00003ffff81eb2e4
> R30: 000001000b80e3c0 R31: 0000000000000034
> NIP: 00003fffb6ac8400 MSR: 800000000000d033 OR3:
> 0000000000000003
> CTR: 0000000000000000 LR: 000000005df1c3e4 XER:
> 0000000000000000
> CCR: 0000000044004824 MQ: 0000000000000001 DAR:
> 00003fffb729c590
> DSISR: 000000000a000000 Syscall Result: 0000000000000000
>
>
> On second thought, that may not be bad after all??
> So, I am ok with the change you propose.
Hmmm, except that in the "sshd" sample showing the firmware-generated eframe,
and which the task was presumably running in kernel space when firmware took
over (?), it has a userspace NIP of 00003fffb6ac8400. What's happening there?
Dave
More information about the Crash-utility
mailing list