[Crash-utility] [RFC] gcore subcommand: a process coredump feature

[anderson at prospeed.net]

>
> Hello,
>
> For some weeks I've developed gcore subcommand for crash utility which
> provides process coredump feature for crash kernel dump, strongly
> demanded by users who want to investigate user-space applications
> contained in kernel crash dump.
>
> I've now finished making a prototype version of gcore and found out
> what are the issues to be addressed intensely. Could you give me any
> comments and suggestions on this work?

Hello Daisuke,

As I mentioned in my previous email re: cpu numbering, I am currently
on vacation, and cannot spend much time looking at this issue until
I get back on August 9th.

However, I think that this could be a useful feature, and I did
take a quick look at how it could be done several months ago when
it was brought up on this mailing list.  However, as you discovered,
I also noted that the user-space core dump code in the kernel has
undergone significant changes over time, and so the implemetation
by the crash utility would have to adapt to the kernel data structures
used by the various kernel versions.  And because of that, I don't
want to put it into the base crash binary, but rather it should be
maintained as one or more extension modules, which can be located
in the "extensions" subdirectory in the crash source package, as well
as stored in the "extensions" web page link from the crash "people"
web site.

It is quite simple to re-adapt your patch as an extension module.
Check the "snap.c" and "snap.mk" files in the extensions subdirectory
as templates for your "gcore" command.

As to the other questions below, I will get back to you after
August 9th.

Thanks,
  Dave


> Motivation
> ==========
>
> It's a relatively familiar technique that in a cluster system a
> currently running node triggers crash kernel dump mechanism when
> detecting a kind of a critical error in order for the running, error
> detecting server to cease as soon as possible. Concequently, the
> residual crash kernel dump contains a process image for the erroneous
> user application. At the case, developpers are interested in user
> space, rather than kernel space.
>
> There's also a merit of gcore that it allows us to use several
> userland debugging tools, such as GDB and binutils, in order to
> analyze user space memory.
>
>
> Current Status
> ==============
>
> I confirm the prototype version runs on the following configuration:
>
>   Linux Kernel Version: 2.6.34
>   Supporting Architecture: x86_64
>   Crash Version: 5.0.5
>   Dump Format: ELF
>
> I'm planning to widen a range of support as follows:
>
>   Linux Kernel Version: Any
>   Supporting Architecture: i386, x86_64 and IA64
>   Dump Format: Any
>
>
> Issues
> ======
>
> Currently, I have issues below.
>
> 1) Retrieval of appropriate register values
>
> The prototype version retrieves register values from a _wrong_
> location: a top of the kernel stack, into which register values are
> saved at any preemption context switch. On the other hand, the
> register values that should be included here are the ones saved at
> user-to-kernel context switch on any interrupt event.
>
> I've yet to implement this. Specifically, I need to do the following
> task from now.
>
>   (1) list all entries from user-space to kernel-space execution path.
>
>   (2) divide the entries according to where and how the register
>   values from user-space context are saved.
>
>   (3) compose a program that retrieves the saved register values from
>   appropriate locations that is traced by means of (1) and (2).
>
> Ideally, I think it's best if crash library provides any means of
> retrieving this kind of register values, that is, ones saved on
> various stack frames. Is there such a plan to do?
>
>
> 2) Getting a signal number for a task which was during core dump
> process at kernel crash
>
> If a target task is halfway of core dump process, it's better to know
> a signal number in order to know why the task was about to be core
> dumped.
>
> Unfortunately, I have no choice but backtrace the kernel stack to
> retrieve a signal number saved there as an argument of, for example,
> do_coredump().
>
>
> 3) Kernel version compatibility
>
> crash's policy is to support all kernel versions by the latest crash
> package. On the other hand, the prototype is based on kernel 2.6.34.
> This means more kernel versions need to be supported.
>
> Well, the question is: to what versions do I need to really test in
> addition to the latest upstream kernel? I think it's practically
> enough to support RHEL4, RHEL5 and RHEL6.
>
>
> Build Instruction
> =================
>
>   $ tar xf crash-5.0.5.tar.gz
>   $ cd crash-5.0.5/
>   $ patch -p 1 < gcore.patch
>   $ make
>
>
> Usage
> =====
>
> Use help subcommand of crash utility as ``help gcore''.
>
>
> Attached File
> =============
>
>   * gcore.patch
>
>     A patch implementing gcore subcommand for crash-5.0.5.
>
>     The diffstat output is as follows.
>
> $ diffstat gcore.patch
>  Makefile      |   10 +-
>  defs.h        |   15 +
>  gcore.c       | 1858
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  gcore.h       |  639 ++++++++++++++++++++
>  global_data.c |    3 +
>  help.c        |   28 +
>  netdump.c     |   27 +
>  tools.c       |   37 ++
>  8 files changed, 2615 insertions(+), 2 deletions(-)
>
> --
> HATAYAMA Daisuke
> d.hatayama at jp.fujitsu.com
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: gcore.patch
> Type: text/x-patch
> Size: 78046 bytes
> Desc: not available
> URL:
> <https://www.redhat.com/archives/crash-utility/attachments/20100802/710541de/attachment.bin>
>
> ------------------------------