[Crash-utility] PATCH 00/10] teach crash to work with "live" ramdump

[Dave Anderson]

----- Original Message -----
> Hi Dave,
> 
> Recently I used crash-tool for the first time and I was pleasantly surprised,
> it really looks like a very useful and handy debugging tool ;)
> 
> And I was surprised again when I figured out that it can be used to debug the
> live system on the same machine. Cool!
> 
> Now I am wondering if we can teach it to debug the live guests runnning under
> qemu/kvm. This looks certainly possible, qemu supports gdb remote protocol.
> 
> But. this obviously needs more work. And. Afaics crash-tool needs some fixes
> anyway. See 01/10-07/10, but probably it needs more changes, so far I only
> tried to audit task.c and kernel.c. I tried to document every change, but I
> am very new to this code so I can be easily wrong.
> 
> So can't we teach it to work with live/raw RAM dumpfiles for the start? See
> 09/10 and 10/10.
> 
> With these changes I can run qemu-kvm with the
> 
> 	-object memory-backend-file,id=MEM,size=128m,mem-path=/tmp/MEM,share=on
> 	-numa node,memdev=MEM
> 
> options and then do
> 
> 	$ crash path-to-guests-vmlinux live:/tmp/MEM at 0
> 
> to debug the live guest, No need to dump-guest-memory + restart
> /usr/bin/crash
> which can be slow.
> 
> What do you think?
> 
> Oleg.
> 
>  defs.h    |  1 +
>  filesys.c |  8 ++++----
>  kernel.c  |  7 +++----
>  main.c    | 11 +++++++++++
>  ramdump.c | 49 ++++++++++++++++++++++++++++++++++---------------
>  task.c    | 13 ++++++-------
>  tools.c   |  2 +-
>  7 files changed, 60 insertions(+), 31 deletions(-)
 

Hi Oleg,

This is going to take some time for me to digest.  But without my studying it
at all, there are a couple things that immediately come to mind.

First, a couple of nits about the files being modified.

I see that your overloading the ramdump.c file, but the ramdump facility was
proposed and added by companies that use a firmware-based facility for dumping
raw ARM/ARM64/MIPS RAM into one or more ramdump files, typically because they 
do/did not have kdump capability with their embedded hardware.  I really don't
feel comfortable conflating that facility with a remote/live QEMU/KVM memory 
source.  Those companies support it, and so any changes that you've added to
ramdump.c should be accomplished elsewhere.

Also, we don't want to get this confused in any way with the REMOTE_xxx
stuff.  The remote.c file and remote access facility was deprecated many
years ago, but recently resurrected in crash-7.0.4 by Verizon specifically
for use like so:

       - Resurrection of the remote analysis capability for use with the
         "xen-crashd" daemon running on a Xen Dom0 host, which communicates
         with a paused or shutdown DomU guest kernel.  The daemon can be
         accessed like so:

           $ crash localhost:5001,/dev/xenmem vmlinux

         (dslutz at verizon.com

I don't think you are modifying their "resurrected" usage, but just an FYI since
you refer to it in your patches.

>From a crash utility perspective, this QEMU/KVM live memory source is a completely
new live memory source, and as such should have its own source file and/or set of
functions, as is the case with the other actual dumpfile types, or live memory
sources /proc/kcore, /dev/mem and /dev/crash.  

Now, more to the point...

Anyway, this is a great idea, and one I've pondered in the past, but is there a
concrete reason that it could not be a simple matter of plugging in a new function
into pc->readmem()?  Currently there are 20+ pluggable readmem functions that exist
for supporting all of the supported dumpfile types and live memory sources.  
Regardless of how it's accomplished, the plug-in basically just needs to be able to
take a physical address and a count (all guaranteed to be contained within a page),
and access/return it.  How it accomplishes that task is hidden from the mainstream
crash code.  

Thanks,
  Dave