[Crash-utility] [SOLVED] Re: About the use of 'gcore'

Tue Jan 21 15:32:40 UTC 2014

Hello everybody,

This small mail to close this thread.

The use of gcore in relationship with the 'crash' utility allow us to 
debug from kernel space down (up ?) to user space.
It is worth to use it.

Thanks to all for your contribution.

Sincerely,
Patrick Agrain

Le 15/01/2014 08:39, Patrick Agrain a écrit :
> Hello everybody,
>
> After complete re-installation of the machine, I'm back on stage.
>
> Dave, to answer to your question, on gdb, the "bt" shows following:
>
>     (gdb) bt
>     #0  0xb776a416 in __kernel_vsyscall ()
>     #1  0xb7698b43 in __write_nocancel () from /lib/libc.so.6
>     #2  0xb76343b4 in _IO_new_file_write () from /lib/libc.so.6
>     #3  0xb7635c90 in _IO_new_do_write () from /lib/libc.so.6
>     #4  0xb7634e80 in _IO_new_file_overflow () from /lib/libc.so.6
>     #5  0xb7637d2a in __overflow () from /lib/libc.so.6
>     #6  0xb76312b5 in putc () from /lib/libc.so.6
>     #7  0x0809b64b in echo_builtin ()
>
> So your guess was correct, most of the functions are from libc. Also 
> stated with the 'vm' command.
>
> Performing again the whole process, I think that I now know where I 
> was wrong.
> The error message at the origin of this thread was due to the fact 
> that I do not use the correct "bash" binary when launching 'gdb'. I 
> made a mismatch between the "bash" on the crashed system and the 
> "bash" on the machine where I analyze the crash.
>
> Using the correct "bash" allows to dump the memory and disassemble the 
> code without problem.
>
> BTW, using the 'vm' and 'vtop' command (Thanks Dave for the tip) 
> offers interesting information that I will explore later.
> For now, I wrote a user-space program that will trigger the page 
> fault, so that I can more easily make the relationship between 
> assembly and C language.
>
> To be followed.
> Patrick Agrain
>
>
> Le 06/01/2014 22:08, Dave Anderson a écrit :
>
> as evidenced by the fact that you were able to at least read stack address
> bfd1b6d8 from gdb.  But it's not clear -- after you walked through the
> user stack -- that the dumpfile contains the text page that you're trying
> to disassemble.  I don't know why it wouldn't be available given that
> it's a "hot" code path, and shouldn't be swapped out or otherwise not
> mapped.
>
> I'm also presuming that your user-stack walk-through is correct.
> When you enter gdb, what does the "bt" command show?
>
> Dave
>
>> ----- Original Message -----
>>> Hello all,
>>>
>>> First of all, I wish a Happy New Year (with less crash, but still enhanced
>>> tools...)
>>>
>>> Thanks for the links, they were very useful.
>>> I dig further in the way of analyzing the User Space, but it seems that I'm
>>> linked to a dead-end way.
>>> Below is a snapshot of kernel / userland stack dump.
>>>
>>> What I've done :
>>> - Crash is triggered by a page fault inside a kernel module (write 0 in
>>> 0xFFFFFFFF, classic).
>>> - Using gcore to create the 'core.<pid>.bash (which is the user task running
>>> at time of crash).
>> I'm curious as to how the bash task was related to the module crash?
>> Did the bash task write to a procfs interface that the module created
>> to then generate the "write 0 to 0xFFFFFFFF"?  Does the crash utility
>> indicate that the bash task is the panic task?  And if so, what does
>> its "bt" show?  (i.e., the kernel-mode backtrace)
>>
>>> - Evaluating an EBP (between { }) chaining value (hypothesis), EIP value
>>> (between [ ]) is then just pushed beside
>>>
>>> The purpose of this study is to find a method to analyze futur crashes from
>>> kernel space down to user space applications.
>>>
>>> Do you have an idea about the cause of this non-dumping of the memory in
>>> user-space ?
>>> Should I use other extension as 'gcore' ?
>>>
>>> Thank in advance.
>>> Best regards,
>>> Patrick Agrain
>>>
>>>
>>> -------
>>> ===============================================================================
>>> --------------------- Go down into User Space Territory
>>> -----------------------
>>>
>>> Last pt_regs of kernel stack is:
>>> | pt_regs
>>> 00000001 094a5408 00000003 ..~......TJ..... | bx cx dx
>>> c2699fc0: 00000003 094a5408 bfd1b704 00000004 .....TJ......... | si di bp ax
>>> c2699fd0: 0000007b ffff007b c07e0000 00000033 {...{.....~.3... | ds es fs gs
>>> c2699fe0: 00000004 b776a416 00000073 00000246 ......v.s...F... | orig_eax ip
>>> cs flags
>>> c2699ff0: bfd1b6d8 0000007b | sp ss
>>> v cccccccc cccccccc ....{........... | padding
>>> |
>>> |----------------------------------------------------------------|
>>> |
>>> (gdb) x/32xw 0xbfd1b680 |
>>> 0xbfd1b680: 0xbfd1b6d0 0x0000000f 0x094b4568 0x080c90b9 |
>>> 0xbfd1b690: 0x094b4568 0x080cd160 0x00001936 0x00000001 |
>>> 0xbfd1b6a0: 0x094ab9c8 0x00000000 0x094b4b48 0xbfd1b7c8 |
>>> 0xbfd1b6b0: 0x080ce9e8 0x094b4b48 0x094b4b48 0xbfd1b728 |
>>> 0xbfd1b6c0: 0x094aed28 0x00000020 0x00000000 0x00000070 |
>>> 0xbfd1b6d0: 0x094b4588 0x080cc080 |
>>> 0xb7698b43 <--|
>>> 0xb7757ff4
>>> 0xbfd1b6e0: 0xb76343b4 0x00000001 0x094a5408 0x00000003
>>> 0xbfd1b6f0: 0xb77584e0 0x080cc080 0xbfd1b728 0xb77584e0
>>>
>>> |------------------------------------------ Hypothesis : this is an EBP
>>> |value...
>>> v
>>> 0xbfd1b700: 0x00000003 {0xbfd1b72c} [0xb7635c90] 0xb77584e0
>>> 0xbfd1b710: 0x094a5408 0x00000003 0x094b4b48 0xbfd1b7c8
>>> 0xbfd1b720: 0xb7757ff4 0xb77584e0 0x0000000a {0xbfd1b750}
>>> 0xbfd1b730: [0xb7634e80] 0xb77584e0 0x094a5408 0x00000003
>>> 0xbfd1b740: 0x0000000a 0xb7757ff4 0xb77584e0 0x0000000a
>>> 0xbfd1b750: {0xbfd1b768} [0xb7637d2a] 0xb77584e0 0x0000000a
>>> 0xbfd1b760: 0xb7757ff4 0xb77584e0 {0xbfd1b788} [0xb76312b5] >-|
>>> 0xbfd1b770: 0xb77584e0 0x0000000a 0xb75c9940 0x094a3e48 |
>>> 0xbfd1b780: 0x00000001 0x00000000 0x00000000 0x0809b64b |
>>> |
>>> Disassemble Try: EIP at 0xb76312b5
>>> <---------------------------------------------|
>>> (gdb) disassemble 0xb7631200, 0xb7631300
>>> Dump of assembler code from 0xb7631200 to 0xb7631300:
>>> 0xb7631200: Cannot access memory at address 0xb7631200
>>> (gdb)
>>> ----------
>> Anyway, I'm guessing that the 0xb76312b5 IP address is in some
>> library, probably libc?  If you do a "vm" on the active bash task
>> from within the crash utility, you will see where it comes from.
>> Try reading the user-space address from the crash utility to see
>> if it was available to copy to the core.<pid>.bash file, i.e.,
>> try this command:
>>
>>   crash> rd -u 0xb76312b5
>>
>> The command above presumes that you are in the context of the
>> "bash" task while running crash.  (i.e., if you enter "set" alone,
>> it shows that particular task)
>>
>> Dave
>>
>>   
>>> Le 17/12/2013 19:12, Buland Kumar Singh a écrit :
>>>
>>>
>>>
>>> Hi Patrick,
>>>
>>> The following links may also be helpful to understand gdb and
>>> it's usage for application core analysis.
>>>
>>> http://web.eecs.umich.edu/~sugih/pointers/gdb_core.html
>>> https://sourceware.org/gdb/onlinedocs/gdb/
>>>
>>> -- BKS
>>>
>>>
>>> On 17 December 2013 21:36, Patrick Agrain <patrick.agrain at alcatel-lucent.com
>>>> wrote:
>>>
>>> Hello all,
>>>
>>> Now that we have dumped the kernel stack, I'm intesresting in the user
>>> process from which we came just before the 'panic'.
>>> Googling around, I found mention of the 'gcore' extension.
>>>
>>> I compiled version 1.22 and installed it.
>>> Using it on crash 6.1.0-1.el6, I get a file core.845.bash on process 'bash'
>>> (in which I trigger a kernel panic) :
>>>
>>>
>>>
>>> crash> gcore -v 1 845
>>> gcore: Opening file core.845.bash ...
>>> gcore: done.
>>> gcore: Writing ELF header ...
>>> gcore: done.
>>> gcore: Retrieving and writing note information ...
>>> gcore: done.
>>> gcore: Writing PT_NOTE program header ...
>>> gcore: done.
>>> gcore: Writing PT_LOAD program headers ...
>>> gcore: done.
>>> gcore: Writing PT_LOAD segment ...
>>> gcore: PT_LOAD[0]: 8048000 - 8048000
>>> gcore: PT_LOAD[1]: 80e2000 - 80e9000
>>> gcore: PT_LOAD[2]: 80e9000 - 80ed000
>>> gcore: PT_LOAD[3]: 94a2000 - 94d1000
>>> gcore: PT_LOAD[4]: b7374000 - b7374000
>>> gcore: PT_LOAD[5]: b7375000 - b7376000
>>> gcore: PT_LOAD[6]: b7376000 - b7377000
>>> gcore: PT_LOAD[7]: b7377000 - b7377000
>>> gcore: PT_LOAD[8]: b737e000 - b737e000
>>> gcore: PT_LOAD[9]: b737f000 - b737f000
>>> gcore: PT_LOAD[10]: b73bb000 - b73bb000
>>> gcore: PT_LOAD[11]: b75bb000 - b75bb000
>>> gcore: PT_LOAD[12]: b75c7000 - b75c8000
>>> gcore: PT_LOAD[13]: b75c8000 - b75c9000
>>> gcore: PT_LOAD[14]: b75c9000 - b75ca000
>>> gcore: PT_LOAD[15]: b75ca000 - b75ca000
>>> gcore: PT_LOAD[16]: b7756000 - b7758000
>>> gcore: PT_LOAD[17]: b7758000 - b7759000
>>> gcore: PT_LOAD[18]: b7759000 - b775c000
>>> gcore: PT_LOAD[19]: b775c000 - b775c000
>>> gcore: PT_LOAD[20]: b775f000 - b7760000
>>> gcore: PT_LOAD[21]: b7760000 - b7761000
>>> gcore: PT_LOAD[22]: b7761000 - b7761000
>>> gcore: PT_LOAD[23]: b7764000 - b7765000
>>> gcore: PT_LOAD[24]: b7769000 - b776a000
>>> gcore: PT_LOAD[25]: b776a000 - b776b000
>>> gcore: PT_LOAD[26]: b776b000 - b776b000
>>> gcore: PT_LOAD[27]: b7789000 - b778a000
>>> gcore: PT_LOAD[28]: b778a000 - b778b000
>>> gcore: PT_LOAD[29]: bfd07000 - bfd1d000
>>> gcore: done.
>>> Saved core.845.bash
>>> crash>
>>>
>>> So far, so good... But
>>>
>>> Question: Are there anywhere some hints about how to use this core.<pid> file
>>> ?
>>>
>>> Thanks in advance.
>>> Regards,
>>> Patrick Agrain
>>>
>>> --
>>> Crash-utility mailing list
>>> Crash-utility at redhat.com
>>> https://www.redhat.com/mailman/listinfo/crash-utility
>>>
>>>
>>>
>>> --
>>> BKS
>>>
>>>
>>> --
>>> Crash-utility mailing listCrash-utility at redhat.com
>>> https://www.redhat.com/mailman/listinfo/crash-utility
>>>
>>>
>>> --
>>> Crash-utility mailing list
>>> Crash-utility at redhat.com
>>> https://www.redhat.com/mailman/listinfo/crash-utility
>> --
>> Crash-utility mailing list
>> Crash-utility at redhat.com
>> https://www.redhat.com/mailman/listinfo/crash-utility
>
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20140121/43957304/attachment.htm>