[Crash-utility] crash does not get proper backtrace?

[KAMEZAWA Hiroyuki]

On Tue, 7 Sep 2010 10:24:29 -0400 (EDT)
Dave Anderson <anderson at redhat.com> wrote:

> 
> ----- "KAMEZAWA Hiroyuki" <kamezawa.hiroyu at jp.fujitsu.com> wrote:
> 
> > On Thu, 2 Sep 2010 08:44:12 -0400 (EDT)
> > Dave Anderson <anderson at redhat.com> wrote:
> > 
> > > 
> > > ----- hutao at cn.fujitsu.com wrote:
> > > 
> > > > Hi,
> > > > 
> > > > I got a problem where it seemed crash got a bad backtrace.
> > > > The problem occurred under the following conditions:
> > > > On a qemu guest system loading a module that stuck at
> > > > the init function(say, call a function that did deadlooping),
> > > > then dumped the guest by `virsh dump vm dumpfile', and run
> > > > crash on the dumpfile.
> > > > 
> > > > The module is:
> > > > 
> > > > ---
> > > > #include <linux/module.h>
> > > > 
> > > > int endless_loop(void)
> > > > {
> > > > 	printk("endless loop\n");
> > > > 	while (1);
> > > > 
> > > > 	return 0;
> > > > }
> > > > 
> > > > int __init endless_init(void)
> > > > {
> > > > 	endless_loop();
> > > > 
> > > > 	return 0;
> > > > }
> > > > module_init(endless_init);
> > > > 
> > > > MODULE_LICENSE("GPL");
> > > > ---
> > > > 
> > > > crash bt command got:
> > > > 
> > > > crash> bt -a
> > > > PID: 0      TASK: ffffffff81648020  CPU: 0   COMMAND: "swapper"
> > > >  #0 [ffffffff81601e08] schedule at ffffffff813e8a49
> > > >  #1 [ffffffff81601e18] apic_timer_interrupt at ffffffff8100344e
> > > >  #2 [ffffffff81601ea0] need_resched at ffffffff8100970c
> > > >  #3 [ffffffff81601eb0] default_idle at ffffffff81009f6b
> > > >  #4 [ffffffff81601ec0] cpu_idle at ffffffff81001bf5
> > > > 
> > > > PID: 1088   TASK: ffff88001dda2d60  CPU: 1   COMMAND: "insmod"
> > > >  #0 [ffff88001e751dc8] schedule at ffffffff813e8a49
> > > >  #1 [ffff88001e751dd0] schedule at ffffffff813e8aec
> > > >  #2 [ffff88001e751e80] preempt_schedule_irq at ffffffff813e8c90
> > > >  #3 [ffff88001e751e90] retint_kernel at ffffffff813eab86
> > > >  #4 [ffff88001e751f20] do_one_initcall at ffffffff81000210
> > > >  #5 [ffff88001e751f50] sys_init_module at ffffffff8106b7ca
> > > >  #6 [ffff88001e751f80] system_call_fastpath at ffffffff81002a82
> > > >     RIP: 00007f761bb58b7a  RSP: 00007fff67a43120  RFLAGS: 00010206
> > > >     RAX: 00000000000000af  RBX: ffffffff81002a82  RCX: 0000000000020010
> > > >     RDX: 0000000000b96010  RSI: 00000000000163da  RDI: 0000000000b96030
> > > >     RBP: 0000000000b96010   R8: 0000000000010011   R9: 0000000000080000
> > > >     R10: 00007f761bb4b140  R11: 0000000000000202  R12: 00000000000163da
> > > >     R13: 00007fff67a44985  R14: 00000000000163da  R15: 0000000000b96010
> > > >     ORIG_RAX: 00000000000000af  CS: 0033  SS: 002b
> > > > 
> > > > Does it lose some function calls between do_one_initcall and retint_kernel?
> > > > (endless_loop <- endless_init)
> > > 
> > > Your best bet is to use "bt -t" in a case such as that.
> > > 
> > > If there are no "starting hooks" for the backtrace code to use, then
> > > it simply defaults to the RSP value left in the task->thread_struct->rsp,
> > > and the RIP of the instruction following "__switch_to".  Those will be
> > > stale, because they represent the last time that the task blocked in
> > > kernel space.  In the case of your endless loop inside the kernel, there
> > > is nothing for the crash utility to grab onto as the starting points because
> > > the task is essentially "active".  It's somewhat similar in nature to 
> > > using "bt -a" on a live system -- the tasks are running either in
> > > kernel or user space, but do not have any "starting points" for the
> > > backtrace code to latch onto, so it's not even allowed as a command.
> > > 
> > 
> > Hmm. but, IIUC, vmcore on the real host (not on virtual machine) taken by kdump
> > can show endless_loop(). Is it because kdump() reads paniced-host-image ? And
> > we should take vmcore generated by "virsh dump" as 
> >  - "it's just a live dump image and there are no guarantee of synchronous register
> >     inforamtion. If you take care, please freeze kernel by some switch". 
> > 
> > Can SIGSTOP or somethig to qemu will help us to take synchronous snapshot of registers ?
> 
> Kdump works because the shutdown sends an NMI to each cpu, leaving an obvious
> shutdown trail that can be tracked from the NMI stack back to the process stack.
> 
> You could also try using alt-sysrq-c on the guest prior to taking the virsh dump
> from the host.  
> 

Thank you. I'm now considering sending NMI via virsh to guest before starting dump
rather than suspending. (if not live dump.) And see what happens.

Thanks,
-Kame