[Crash-utility] Re: source line numbers and modules (on x86_64)

[Dave Anderson]

Mike Snitzer wrote:
> On Wed, Apr 30, 2008 at 8:21 PM, Dave Anderson <anderson at redhat.com> wrote:
>   
>> Mike Snitzer wrote:
>>
>>     
>>> On Wed, Apr 30, 2008 at 7:41 PM, Dave Anderson <anderson at redhat.com>
>>>       
>> wrote:
>>     
>>>       
>>>> Mike Snitzer wrote:
>>>>
>>>>
>>>>
>>>>         
>>>>> On Wed, Apr 30, 2008 at 6:43 PM, Dave Anderson <anderson at redhat.com>
>>>>>
>>>>>
>>>>>           
>>>
>>>       
>>>>>>  Anyway, there are no kernel patches applied to RHEL5 kernels
>>>>>>  to make it work that I'm aware of.  That being said, I don't
>>>>>>  have an answer as to why they don't work for modules on your
>>>>>>  custom kernel (or why they don't work in RHEL4 kernels...).
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>             
>>>>> I configured my x86_64 kernel to be relocatable.  Could this be the
>>>>> reason for modules' line number debug data being off?  From:
>>>>> http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
>>>>>
>>>>> "Using /proc/kallsyms or a --reloc=[size] as a command line argument
>>>>>  is similar to using a System.map file, in that it results in the loss
>>>>>  of the use of line number debug data."
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>           
>>>>  No, that particular issue only applies to the base kernel, and is
>>>> irrelevant with respect to modules.
>>>>  (Note that the RHEL5 kernel is relocatable as well.)
>>>>
>>>>
>>>>         
>>> OK, yeah making the base kernel non-relocatable didn't help.
>>>
>>>
>>>
>>>       
>>>>  What you might confirm is whether you're getting any debuginfo data
>>>>         
>> when
>>     
>>>> you load the module?  Are you sure that your modules are being built
>>>>         
>> with -g?
>>     
>>>>         
>>> How would I confirm that I'm getting debuginfo data when I 'mod -s
>>> ext3'?  I'm using an spec file that is based heavily (at least
>>> relative to debuginfo) on the redhat kernel-2.6.spec but I'll have to
>>> verify that I'm building with -g...
>>>
>>> Mike
>>>
>>>
>>>       
>>  So your build procedure ends up with a kernel-debuginfo package that
>> contains a set of "split" <module-name>.ko.debug files?
>>     
>
> Yes.
>
>   
>> If you take the module filename displayed after the "mod -s" is done,
>> and run it through "readelf -a", you'll see a .gnu_debuglink section.
>>     
>
> My kernel's ext3.ko has:
>   [23] .gnu_debuglink    PROGBITS         0000000000000000  00036558
>        0000000000000014  0000000000000000           0     0     4
>
> which would appear fairly comparable to what the RHEL5 kernel's ext3.ko has:
>   [22] .gnu_debuglink    PROGBITS         0000000000000000  00037418
>        0000000000000014  0000000000000000           0     0     4
>
>   
>> That /lib/modules/<path-to>/module.ko module will have
>>  a link to its partner in
>> /usr/lib/debug/lib/modules/<path-to>/module.ko.debug.
>>     
>
> readelf -x $SECTION /lib/modules/<path-to>/ext3.ko shows that
> .gnu_debuglink points to ext3.ko.debug as expected.
>
> Given the "so far so good" nature of what you're helping me verify
> here it still begs the question: so why doesn't it work on my
> 2.6.22.19 x86_64 kernel? ;)
>
> Where does the line number debug data get stored in the
> module.ko.debug?  In my case the source comes up as
> "include/linux/clocksource.h: 121" for all kernel modules' symbols.
>
> Any further pointers would be greatly appreciated, thanks a lot for
> your time already!
> Mike
>   

In the crash sources, gdb-6.1/gdb/symtab.c contains several crash 
utility "hooks" into
the native gdb code.  The function called gdb_get_line_number()  calls  
the pre-existing
gdb find_pc_line()  function to get the line number information.  As to 
why it works
with some kernels and not with others is the question at hand, and to which
I have no answer.  The behavior you see is similar to what happens with
x86_64 in RHEL4 kernels, although on my RHEL4 x86_64 machine, it always
seems to return a line in include/linux/seqlock.h (?).  In any case, the 
module
debuginfo data contains a linkage between text offset values to the 
source code line
numbers.  In other words, do an "nm -Bn <module>.ko" and you'll see that 
module
text locations are offsets, as opposed to compiled-in addresses that you 
see if you
do an "nm -Bn vmlinux".  So my guess is that the problem is related to 
the fact that
the module objects are relocatable, where the module object has offset 
values applied
to each text offset, whereas the the PC values that get passed to the 
gdb code
are the kernel vmalloc()'d addresses where the module got loaded.  When the
"mod" command is done, it does a gdb "add-symbol-file" command, which
essentially transforms of the PC offset values in the module object into the
loaded vmalloc address.  But why the resultant vmalloc() addresses work for
some kernels and not others is a mystery to me.

Dave