[Crash-utility] Debugging Xen Hypervisor with 'crash' question...

[Dave Anderson]

Roger Cruz wrote:
> Sorry if this is an obvious question but I’m new to the ‘crash’ utility. 
>  I read Anderson’s white paper on crash and didn’t find any references 
> to how to use ‘crash’ to debug the hypervisor.  I have crash running and 
> accessing Domain 0’s kernel tasks and other variables, so I am 
> comfortable thinking that I have the right setup.  I start crash with:
> 
>  
> 
> #crash xen-syms /dom0/proc/vmcore
> 
>  
> 
> And get the following output
> 
>  
> 
> #crash xen-syms /dom0/proc/vmcore
> 
>  
> 
> crash 4.0-4.7
> 
> Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007  Red Hat, Inc.
> 
> Copyright (C) 2004, 2005, 2006  IBM Corporation
> 
> Copyright (C) 1999-2006  Hewlett-Packard Co
> 
> Copyright (C) 2005, 2006  Fujitsu Limited
> 
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> 
> Copyright (C) 2005  NEC Corporation
> 
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> 
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> 
> This program is free software, covered by the GNU General Public License,
> 
> and you are welcome to change it and/or distribute copies of it under
> 
> certain conditions.  Enter "help copying" to see the conditions.
> 
> This program has absolutely no warranty.  Enter "help warranty" for details.
> 
>  
> 
> GNU gdb 6.1
> 
> Copyright 2004 Free Software Foundation, Inc.
> 
> GDB is free software, covered by the GNU General Public License, and you are
> 
> welcome to change it and/or distribute copies of it under certain conditions.
> 
> Type "show copying" to see the conditions.
> 
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> 
> This GDB was configured as "i686-pc-linux-gnu"... 
> 
>  
> 
>    KERNEL: xen-syms
> 
>  DUMPFILE: /dom0/proc/vmcore
> 
>      CPUS: 4
> 
>   DOMAINS: 4
> 
>    UPTIME: 00:01:30
> 
>   MACHINE: Intel(R) Xeon(R) CPU            5140  @ 2.33GHz  (2327 Mhz)
> 
>    MEMORY: 4 GB
> 
>   PCPU-ID: 2
> 
>      PCPU: ff1bbfb4
> 
>   VCPU-ID: 0
> 
>      VCPU: ffbe6080  (VCPU_RUNNING)
> 
> DOMAIN-ID: 0
> 
>    DOMAIN: ff238080  (DOMAIN_RUNNING)
> 
>     STATE: CRASH
> 
>  
> 
> I would like to know what commands there are to examine the memory 
> management system or any other internal data structures.  Also, how do I 
> look at a stack trace in the hypervisor for a crash.  I tried the ‘gdb 
> where’ command and it said no stack.
> 

Enter "help" -- it shows the commands when running against
a xen-syms hypervisor:

   crash> help

   *              dumpinfo       list           sched          vcpu
   alias          eval           log            search         vcpus
   ascii          exit           p              set            whatis
   bt             extend         pcpus          struct         wr
   dis            foreach        pte            sym            q
   domain         gdb            rd             sys
   doms           help           repeat         union

   crash version: 4.0-4.7   gdb version: 6.1
   For help on any command above, enter "help <command>".
   For help on input options, enter "help input".
   For help on output options, enter "help output".

   crash>

Then for any particular command, enter "help <command>",
so for backtrace options, enter "help bt".  I do note
that some of the common commands between running crash
on a vmlinux and a xen-syms show the help data for the
command as if it were running against a vmlinux, and
as such, some advertised options may not work on a
xen-syms session.

A limited set of gdb commands are runnable, although the
embedded gdb module has no clue of the vmcore file; it's
invoked internally as "gdb xen-syms".

I'm presuming that the crash occurred within the hypervisor
as opposed to the (vmlinux) kernel?  If it happened within
kernel code, substitute the xen-syms argument with the
vmlinux of the dom0 kernel, and you will be presented
with a different set of commands.

Dave

>  
> 
> Thanks in advance.
> 
>  
> 
>  
> 
> Roger Cruz
> 
> Principal SW Engineer
> 
> Marathon Technologies Corp.
> 
> 978-489-1153
>