[Crash-utility] enhance bt command

[Dave Anderson]

Ming Zhang wrote:
> On Mon, 2008-03-03 at 14:56 -0500, Dave Anderson wrote:
>> Ming Zhang wrote:
>>> On Mon, 2008-03-03 at 14:04 -0500, Dave Anderson wrote:
>>>> Ming Zhang wrote:
>>>>
>>>>> my English sucks...
>>>>>
>>>>> this is what i got. my side size-512 or size-1024 also works. just stuff
>>>>> like dentry cache or inode cache does not work.
>>>>>
>>>>>
>>>>> crash> rd 10078b2bca0 2
>>>>>      10078b2bca0:  000001007877c8d4 00000100710fd5c8   ..wx.......q....
>>>>> crash> rd -S 10078b2bca0 2
>>>>>      10078b2bca0:  [dentry_cache]   00000100710fd5c8 
>>>>> crash> kmem -s 10078b2bca0
>>>>> kmem: address is not allocated in slab subsystem: 10078b2bca0
>>>> No, your English is fine -- it's your command-entering that sucks!
>>>>
>>>> Above, when you entered "kmem -s 10078b2bca0", you're incorrectly entering
>>>> the *address* where the dentry_cache reference (1007877c8d4) is located.
>>>> And so it's telling you that 10078b2bca0 is not allocated in the slab
>>>> subsystem, which it isn't...
>>>>
>>>> But if you entered "kmem -s 1007877c8d4", you'd see the dentry_cache
>>>> information.
>>> yes. i noticed that. need some tea when handling 3 bugs at the same
>>> time...
>>>
>>> so here is my understanding. it actually shows the slab object that
>>> contain that address. it might not be the address of that object. so
>>> here is what i need to do
>>>
>>> crash> rd -x 10078b2bca0
>>>      10078b2bca0:  000001007877c8d4 
>>> crash> rd -S 10078b2bca0
>>>      10078b2bca0:  [dentry_cache]   
>>>
>>> crash> kmem -s 000001007877c8d4
>>> CACHE            NAME                 OBJSIZE  ALLOCATED     TOTAL
>>> SLABS  SSIZE
>>> 10037ffc080      dentry_cache             240       9429     10560
>>> 660     4k
>>> SLAB              MEMORY            TOTAL  ALLOCATED  FREE
>>> 1007877c040       1007877c088          16          9     7
>>> FREE / [ALLOCATED]
>>>   [1007877c8d4]
>>>
>>> this only tell me that it belongs to one dentry object, but no idea
>>> which one
>>>
>>> then i have to use kmem -S dentry_cache, find out this piece
>>>
>>> SLAB              MEMORY            TOTAL  ALLOCATED  FREE
>>> 1007877c040       1007877c088          16          9     7
>>> FREE / [ALLOCATED]
>>>   [1007877c088]
>>>   [1007877c178]
>>>    1007877c268  (cpu 1 cache)
>>>    1007877c358  (cpu 1 cache)
>>>   [1007877c448]
>>>    1007877c538  (cpu 1 cache)
>>>   [1007877c628]
>>>   [1007877c718]
>>>   [1007877c808]
>>> ~~~~~~~~~~~~~~~~~~~~ then find out it is here.
>>>   [1007877c8f8]
>>>    1007877c9e8  (cpu 1 cache)
>>>    1007877cad8  (cpu 1 cache)
>>>   [1007877cbc8]
>>>   [1007877ccb8]
>>>    1007877cda8  (cpu 1 cache)
>>>    1007877ce98  (cpu 1 cache)
>>>
>>> then i know the object contain this address is 1007877c808.
>>> then 
>>>
>>> crash> dentry.d_iname
>>> struct dentry {
>>>   [0xcc] unsigned char d_iname[36];
>>> }
>>> crash> eval c808+cc
>>> hexadecimal: c8d4  
>>>     decimal: 51412  
>>>       octal: 144324
>>>
>>> show me that variable in stack actually is d_iname.
>>>
>>> then can we have the output format as
>>>
>>>      10078b2bca0:  [000001007877c808+cc: dentry_cache]   
>>>
>>> so we know the object address, which slab it is in, and the offset,
>>> (thus can derive the raw value), all in one shot?
>>>
>> You've done a fine bit of debugging your issue...
>>
>> But I think that's a bit of overkill for each address reference.
>>
>> To do it right it would have to walk slab chains to gather all of the
>> information needed by the "kmem -S" output, which can be extremely
>> time-consuming, and potentially error-prone if the slab chain is
>> corrupt or being modified while running on a live system.
> 
> full agree. realized now.
> 
> then here is my question, how "kmem -s <addr>" find out which slab this
> address belong to? by only looking at the page?

The kmem_cache and containing slab addresses are stashed in unused
fields of the page structure of the object's virtual address.

> 
> then here is my revised suggestion. can we split this into 2 steps?
> 
> (1) rd -S show [raw address: cache name]
> 
> (this is a great to have since do rd 2 times, one with -S and another
> without -S is tedious.)

But in your case, for example, I'm presuming you've done a "bt -f",
and simply would like a symbols/slab translation of a chunk of memory
making up one of the stack frames, so you do a subsequent "rd -S" of
that memory area.  Doesn't seem that tedious to me...

> 
> (2) and kmem -s <address>
> 
> show which slab it is in and optionally (with a new option like -O) show
> the object?

Not following -- "kmem -s <address>" does show which slab it's in.
And if you're asking whether the object can be dumped as the structure
type it is (?), well, how would it be possible from the crash utility's
viewpoint to even know what type of data structure it is?

If you want to look at all of the objects in a slab represented
as data structures, you're going to have supply the knowledge of
what data structure they are.  It's simple enough, just do a "kmem -S"
into a file, delete everything except the object addresses that you're
interested in, insert "struct whatever" in front of each address, save
the file -- and run it as crash input file.

> 
> (this is nice to have).
> 
> thanks.

Look, you can have it do anything you want...

However, keep in mind that there are 4 diffenent kmem slab paradigms
supported (the "original", the first verion with per-cpu objects, the
second version with per-cpu objects, and now the completely new CONFIG_SLUB
subsystem).  All of them would have to be supported.  And the code that does
the object dumping is somewhat replicated and embedded in the "kmem -S" support
code, so it's not a simple re-use of an existing function, which I've done
to get the simple "rd -S" output.

Dave