[Crash-utility] Re: Re: Re: crash and sles 9 dumps (Dave Anderson)

[Dave Anderson]

Dave Anderson wrote:
> Daniel Li wrote:
> 
>> Dave Anderson wrote:
>>
>>> Daniel Li wrote:
>>>
>>>> It seems the problem is not one with guest dump, but the version of 
>>>> SLES.
>>>>
>>>> After upgrading my NATIVE SLES 9 system to SP 3, exactly the same 
>>>> problem happened while trying to use 'crash' on the live system, 
>>>> with a debug linux kernel ('vmlinux.dbg' below) built on the same 
>>>> system from matching 'kernel-source' package. (During this upgrade, 
>>>> the linux kernel changed from 2.6.5-7.97-smp to 2.6.5-7.244-smp, the 
>>>> same as that on the guest.)
>>>>
>>>> Has anyone else seen this?
>>>
>>>
>>>
>>> Did anything change in the task_struct between 2.6.5-7.97-smp and
>>> 2.6.5-7.244-smp?
>>>
>>> Or, more likely, anything associated with the pidhash/pid_hash-related
>>> code in the kernel?
>>>
>>> Is the output of the crash command "help -t | grep refresh_task_table"
>>> different when running against 2.6.5-7.97-smp vs. 2.6.5-7.244-smp?
>>>
>>> Dave
>>>
>> The definition of task_struct between 2.6.5-7.97-smp and 
>> 2.6.5-7.244-smp did change. There is one new 8-bytes field called 
>> 'last_ran' before the list_head for tasks. This is what I don't get: 
>> why should it matter as long as the dump and debug kernel are using 
>> the same definition?
>>
> 
> It shouldn't.
> 
> Does the output of "help -o task_struct" on the .97 vs the .244 kernels
> reflect the member offset differences as you would expect?  I.e., 
> everything
> (that's not -1) coming after the new last_ran member is bumped up by 8?
> 
> And are you sure there's nothing different w/respect to the pid_hash
> declarations/usage?
> 
> Dave
 >
 >> The output of "help -t | grep refresh_task_table" didn't change.

The reason I ask about any pid_hash-related changes is because
over the years the manner of task table handling by the crash
utility has had to change to deal with the kernel changes.
The crash-internal tt->refresh_task_table function pointer
that you see in the "help -t" output gets set during task_init()
to one of these functions:

   static void refresh_fixed_task_table(void);
   static void refresh_unlimited_task_table(void);
   static void refresh_pidhash_task_table(void);
   static void refresh_pid_hash_task_table(void);
   static void refresh_hlist_task_table(void);
   static void refresh_hlist_task_table_v2(void);

with later kernels requiring the later function in the list above.

For a 2.6.5 vintage kernel, I'm guessing that when you did
the "help -t" it showed "refresh_pid_hash_task_table()"?

Anyway, in the two kernels that you are comparing, how is the
"pid_hash" variable declared in the kernel sources?  With
respect to the crash-internal setting of tt->refresh_task_table,
it should line up like so:

kernel:  static struct list_head pid_hash[PIDTYPE_MAX][PIDHASH_SIZE];
  crash:  refresh_pid_hash_task_table()

kernel: static struct hlist_head *pid_hash[PIDTYPE_MAX];
  crash: refresh_hlist_task_table()

kernel: static struct hlist_head *pid_hash;
  crash: refresh_hlist_task_table_v2()

For whatever reason it almost looks like the task-gathering is
using the wrong function, or maybe given back-ports and such,
the SUSE kernel task-handling is now a "hybrid" that would need
its own task-gathering function in the crash utility.

With respect to the "last_ran" addition, you could always rebuild
a kernel with that field moved to the end of the task_struct,
run that kernel, and see what happens.  If the "ps" task output
is still screwed up, then it should rule that out as the problem
at hand.

Dave