[Crash-utility] About the use of 'gcore'

Mon Jan 6 21:08:03 UTC 2014

----- Original Message -----
> Hello all,
> 
> First of all, I wish a Happy New Year (with less crash, but still enhanced
> tools...)
> 
> Thanks for the links, they were very useful.
> I dig further in the way of analyzing the User Space, but it seems that I'm
> linked to a dead-end way.
> Below is a snapshot of kernel / userland stack dump.
> 
> What I've done :
> - Crash is triggered by a page fault inside a kernel module (write 0 in
> 0xFFFFFFFF, classic).
> - Using gcore to create the 'core.<pid>.bash (which is the user task running
> at time of crash).

I'm curious as to how the bash task was related to the module crash?
Did the bash task write to a procfs interface that the module created
to then generate the "write 0 to 0xFFFFFFFF"?  Does the crash utility
indicate that the bash task is the panic task?  And if so, what does
its "bt" show?  (i.e., the kernel-mode backtrace)

> - Evaluating an EBP (between { }) chaining value (hypothesis), EIP value
> (between [ ]) is then just pushed beside
> 
> The purpose of this study is to find a method to analyze futur crashes from
> kernel space down to user space applications.
> 
> Do you have an idea about the cause of this non-dumping of the memory in
> user-space ?
> Should I use other extension as 'gcore' ?
> 
> Thank in advance.
> Best regards,
> Patrick Agrain
> 
> 
> -------
> ===============================================================================
> --------------------- Go down into User Space Territory
> -----------------------
> 
> Last pt_regs of kernel stack is:
> | pt_regs
> 00000001 094a5408 00000003 ..~......TJ..... | bx cx dx
> c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di bp ax
> c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es fs gs
> c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... | orig_eax ip
> cs flags
> c2699ff0: bfd1b6d8 0000007b | sp ss
> v cccccccc cccccccc ....{........... | padding
> | 
> |----------------------------------------------------------------|
> | 
> (gdb) x/32xw 0xbfd1b680 |
> 0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
> 0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
> 0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
> 0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
> 0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
> 0xbfd1b6d0: 0x094b4588 0x080cc080 |
> 0xb7698b43 <--|
> 0xb7757ff4
> 0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
> 0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
> 
> |------------------------------------------ Hypothesis : this is an EBP
> |value...
> v
> 0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
> 0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
> 0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
> 0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
> 0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
> 0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
> 0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5] >-|
> 0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
> 0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
> | 
> Disassemble Try: EIP at 0xb76312b5
> <---------------------------------------------|
> (gdb) disassemble 0xb7631200, 0xb7631300
> Dump of assembler code from 0xb7631200 to 0xb7631300:
> 0xb7631200: Cannot access memory at address 0xb7631200
> (gdb)
> ----------

Anyway, I'm guessing that the 0xb76312b5 IP address is in some
library, probably libc?  If you do a "vm" on the active bash task
from within the crash utility, you will see where it comes from.
Try reading the user-space address from the crash utility to see
if it was available to copy to the core.<pid>.bash file, i.e.,
try this command:

 crash> rd -u 0xb76312b5

The command above presumes that you are in the context of the
"bash" task while running crash.  (i.e., if you enter "set" alone,
it shows that particular task)

Dave

> 
> Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
> 
> 
> 
> Hi Patrick,
> 
> The following links may also be helpful to understand gdb and
> it's usage for application core analysis.
> 
> http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
> https://sourceware.org/gdb/onlinedocs/gdb/
> 
> -- BKS
> 
> 
> On 17 December 2013 21:36, Patrick Agrain < patrick.agrain at alcatel-lucent.com
> > wrote:
> 
> 
> 
> Hello all,
> 
> Now that we have dumped the kernel stack, I'm intesresting in the user
> process from which we came just before the 'panic'.
> Googling around, I found mention of the 'gcore' extension.
> 
> I compiled version 1.22 and installed it.
> Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process 'bash'
> (in which I trigger a kernel panic) :
> 
> 
> 
> crash> gcore -v 1 845
> gcore: Opening file core.845.bash ...
> gcore: done.
> gcore: Writing ELF header ...
> gcore: done.
> gcore: Retrieving and writing note information ...
> gcore: done.
> gcore: Writing PT_NOTE program header ...
> gcore: done.
> gcore: Writing PT_LOAD program headers ...
> gcore: done.
> gcore: Writing PT_LOAD segment ...
> gcore: PT_LOAD[0]: 8048000 - 8048000
> gcore: PT_LOAD[1]: 80e2000 - 80e9000
> gcore: PT_LOAD[2]: 80e9000 - 80ed000
> gcore: PT_LOAD[3]: 94a2000 - 94d1000
> gcore: PT_LOAD[4]: b7374000 - b7374000
> gcore: PT_LOAD[5]: b7375000 - b7376000
> gcore: PT_LOAD[6]: b7376000 - b7377000
> gcore: PT_LOAD[7]: b7377000 - b7377000
> gcore: PT_LOAD[8]: b737e000 - b737e000
> gcore: PT_LOAD[9]: b737f000 - b737f000
> gcore: PT_LOAD[10]: b73bb000 - b73bb000
> gcore: PT_LOAD[11]: b75bb000 - b75bb000
> gcore: PT_LOAD[12]: b75c7000 - b75c8000
> gcore: PT_LOAD[13]: b75c8000 - b75c9000
> gcore: PT_LOAD[14]: b75c9000 - b75ca000
> gcore: PT_LOAD[15]: b75ca000 - b75ca000
> gcore: PT_LOAD[16]: b7756000 - b7758000
> gcore: PT_LOAD[17]: b7758000 - b7759000
> gcore: PT_LOAD[18]: b7759000 - b775c000
> gcore: PT_LOAD[19]: b775c000 - b775c000
> gcore: PT_LOAD[20]: b775f000 - b7760000
> gcore: PT_LOAD[21]: b7760000 - b7761000
> gcore: PT_LOAD[22]: b7761000 - b7761000
> gcore: PT_LOAD[23]: b7764000 - b7765000
> gcore: PT_LOAD[24]: b7769000 - b776a000
> gcore: PT_LOAD[25]: b776a000 - b776b000
> gcore: PT_LOAD[26]: b776b000 - b776b000
> gcore: PT_LOAD[27]: b7789000 - b778a000
> gcore: PT_LOAD[28]: b778a000 - b778b000
> gcore: PT_LOAD[29]: bfd07000 - bfd1d000
> gcore: done.
> Saved core.845.bash
> crash>
> 
> So far, so good... But
> 
> Question: Are there anywhere some hints about how to use this core.<pid> file
> ?
> 
> Thanks in advance.
> Regards,
> Patrick Agrain
> 
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
> 
> 
> 
> --
> BKS
> 
> 
> --
> Crash-utility mailing list Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
> 
> 
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility