[Crash-utility] crash aborts with cannot determine idle task

[Dave Anderson]

Chandru wrote:
> 
>> Look at the crash function get_idle_threads() in task.c, which is where
>> you're failing.  It runs through the history of the symbols that Linux
>> has used over the years for the run queues.  For the most recent kernels,
>> it looks for the "per_cpu__runqueues" symbol.  At least on 2.6.25-rc2,
>> the kernel still defines them in kernel/sched.c like this:
>>
>>   static DEFINE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
>>
>> So if you do an "nm -Bn vmlinux | grep runqueues", you should see:
>>
>>   # nm -Bn vmlinux-2.6.25-rc1-ext4-1 | grep runqueues
>>   ffffffff8082b700 d per_cpu__runqueues
>>   #
>>
>> I'm guessing that's not the problem -- so presuming that the symbol 
>> *does*
>> exist, find out why it's failing to increment "cnt" in this part of
>> get_idle_threads():
>>
>>        if (symbol_exists("per_cpu__runqueues") &&
>>             VALID_MEMBER(runqueue_idle)) {
>>                 runqbuf = GETBUF(SIZE(runqueue));
>>                 for (i = 0; i < nr_cpus; i++) {
>>                         if ((kt->flags & SMP) && (kt->flags & 
>> PER_CPU_OFF)) {
>>                                 runq = 
>> symbol_value("per_cpu__runqueues") +
>>                                         kt->__per_cpu_offset[i];
>>                         } else
>>                                 runq = 
>> symbol_value("per_cpu__runqueues");
>>
>>                         readmem(runq, KVADDR, runqbuf,
>>                                 SIZE(runqueue), "runqueues entry 
>> (per_cpu)",
>>                                 FAULT_ON_ERROR);
>>                         tasklist[i] = ULONG(runqbuf + 
>> OFFSET(runqueue_idle));
>>                         if (IS_KVADDR(tasklist[i]))
>>                                 cnt++;
>>                 }
>>         }
>>
>> Determine whether it even makes it to the inner for loop, whether
>> the pre-determined nr_cpus value makes sense, whether the SMP flag
>> reflects whether the kernel was compiled for SMP, whether the PER_CPU_OFF
>> flag was set, what address was calculated, etc...
>>
>> Dave
>>
> Thanks for the reply Dave.  The code makes it to the inner for loop and 
> the condition
> if (IS_KVADDR(tasklist[i]))  fails which is why 'cnt' doesn't get 
> incremented. The tasklist[i] somewhat has this value : 0x3d60657870722024.
> 
> I ran gdb on the vmcore file and printed the memory contents .
> 
> (gdb) print per_cpu__runqueues
> $1 = {lock = {raw_lock = {slock = 1431524419}}, nr_running = 
> 5283422954284598606,
>  raw_weighted_load = 5064663116585906736, cpu_load = 
> {2316051155752670036, 5929356451801411872,
>    2613857225664584019}, nr_switches = 5644502509443686462,
>  nr_uninterruptible = 2316072106569976142, expired_timestamp = 
> 5142904381182533935,
>  timestamp_last_tick = 7235439831918129227, curr = 0x5f66696c650a5243, 
> idle = 0x3d60657870722024, <<<-----
>  prev_mm = 0x5243202b20243f60, active = 0xa247b4155535443, expired = 
> 0x5352434449527d2f,
> 
> 
> Does this mean that the kernel data was corrupted when vmcore was 
> collected ?.

I don't know.

You cannot expect gdb to be able to handle it at all, unless
the kernel was configured without CONFIG_SMP.  In that case,
the per_cpu__runqueues symbol points to the singular instance
of an rq.

However, more likely your kernel is configured with CONFIG_SMP.
In that case, a per-cpu offset has to be applied to the symbol
value of per_cpu__runqueues to calculate where each cpu's instance
of its rq structure is located.  I can guarantee you that gdb
cannot do that, and that's probably why you're seeing "garbage"
data above.

So you can see that's what's happening in the get_idle_threads()
function where it's calculating the "runq" address each time
through the loop.  If the kernel is configured CONFIG_SMP,
it adds the per-cpu offset value, otherwise it uses the
symbol value of "per_cpu__runqueues" as is.

As I suggested before, you're going to have to determine why
the tasklist[i] is bogus.  The first things to determine are:

(1) what "nr_cpus" was calculated to be, and
(2) whether the SMP and PER_CPU_OFF flags are set in kt->flags.

If those variables/settings make sense, then presumably the
problem is in the determination of the per-cpu offset values.
That's done in a machine-specific way, so I can't help you
without knowing what architecture you're dealing with, not
to mention what kernel version, or whether it's configured
CONFIG_SMP or not, and whether you can run crash on the live
system that generated the dumpfile.

Dave