[Crash-utility] [PATCH] Speed up usage of a flat makedumpfile vmcore.

[Dave Anderson]

----- Original Message -----
> On 04/28/15 17:05, Dave Anderson wrote:
> > 
> > 
> > ----- Original Message -----
> >> Using a bubble sort is slow, switch to an insertion sort.
> >>
> >> bubble sort:     real    3m45.168s
> >>
> >> insertion sort:  real    0m3.164s
> >>
> >> Signed-off-by: Don Slutz <dslutz at verizon.com>
> >> ---
> >> I do have a big (32G sized file, that gzipped is 357M).
> >> let me know if you want it.
> > 
> > Hi Don,
> > 
> > I'm running some sanity tests with a couple 94GB flat vmcores by comparing
> > the data output of some of the more verbose commands.  But upon invocation,
> > I do see some significant speed increases, for example:
> > 
> >   vmcore.1  2m45.545s  ->  1m31.428s
> >   vmcore.2  2m42.806s  ->  0m46.624s
> > 
> > The numbers change from invocation to invocation, but always in the
> > same direction.  Probably file page caching is affecting the times.
> > 
> 
> Yes, file cache size is important here.
> 
> > Anyway, I've never taken the time to actually understand how the flat
> > format
> > is laid out, and I would normally ask the author to sign off on any major
> > changes to it.  But I see that Ken'ichi Ohmichi is no longer a member
> > of this list, so for all practical purposes, it's currently "unmaintained".
> > I'll continue testing further tomorrow, but if by chance you can whip up a
> > simplified explanation of the flat dumpfile layout, I'd appreciate it.
> > 
> 
> Here is my understanding.
> 
> The file starts with a header:
> 
> 
> hyper-0-21-52:~/cores/15.04.27-13:26:11>hexdump -C vmcore.flat | more
> 00000000  6d 61 6b 65 64 75 6d 70  66 69 6c 65 00 7f 00 00
> |makedumpfile....|
> 00000010  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 01
> |................|
> 00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> |................|
> *
> 
> (from makedumpfile.h):
> 
> struct makedumpfile_header {
>         char    signature[SIG_LEN_MDF]; /* = "makedumpfile" */
>         int64_t type;
>         int64_t version;
> };
> 
> 
> The 1st data block starts at 4096:
> 
> 
> 00001000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 40
> |...............@|
> 
> 
> And is 2 64bit integers (from makedumpfile.h):
> 
> struct makedumpfile_data_header {
>         int64_t offset;
>         int64_t buf_size;
> };
> 
> 
> The offset is the "real file offset" where this data block is.
> 
> I.E. this is the 1st 0x40 bytes in the normal file.
> 
> The data header just keeps repeating:
> 
> 
> 00001050  00 00 00 00 00 00 09 a8  00 00 00 00 00 00 1d 00
> |................|
> 
> 
> 00002d60  00 00 00 00 00 00 26 a8  00 00 00 00 00 01 00 00
> |......&.........|
> 
> 
> 00012d70  00 00 00 00 00 01 26 a8  00 00 00 00 00 01 00 00
> |......&.........|
> 
> 
> Until the offset is -1.
> 
> Note: The data header is stored in big endian format.
> 
> 
> While this works for any data type, I have only seen "normal" elf data:
> 
> 00001010  7f 45 4c 46 02 01 01 00  00 00 00 00 00 00 00 00
> |.ELF............|
> 00001020  04 00 3e 00 01 00 00 00  00 00 00 00 00 00 00 00
> |..>.............|
> 00001030  40 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> |@...............|
> 00001040  00 00 00 00 40 00 38 00  2b 00 00 00 00 00 00 00
> |.... at .8.+.......|
> 
> 
> 00001060  05 00 00 00 50 01 00 00  01 00 00 00 43 4f 52 45
> |....P.......CORE|
> 00001070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00
> |................|
> *
> 000010e0  00 00 00 00 03 00 00 00  00 00 00 00 e8 0d 33 80
> |..............3.|
> ...
> 
> So basically what this is a stream way of providing a file so that you
> can fill out the (for an ELF file) the set of PT_LOAD sections at the
> end and not need 2 passes (or a local file) to store the result.
> 
> 
> Hope this helps.
>    -Don Slutz

Thanks Don, it helps very much.  I really appreciate your taking the time.

And in the case of compressed kdumps, I believe it goes back and writes the
pfn bitmask multiple times during the stream.  

Anyway, the patch is a nice enhancement to what is a painful process
when analyzing really large flattened dumpfiles.

Queued for crash-7.1.1:

  https://github.com/crash-utility/crash/commit/d304aab38b804207c21f0f77c25e737a6b81d490

Thanks again,
  Dave