[Crash-utility] Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff

Wed Jun 20 14:25:46 UTC 2012

Hi Dave,

OK, makes sense. I still spend 95% of my time in the RHEL 5 kernels so it's just lack of enough time in the RHEL 6 kernels.
I did not have the debuginfo so I built a vmkernel with debug from source and used that. I was able to get what I wanted out of the dump, specifically needed to look at the net_dev structures.

Is there a write up somewhere on changes with crash for the 2.6.32.x kernels that you can share.

Thanks
Laurence

-----Original Message-----
From: crash-utility-bounces at redhat.com [mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
Sent: Wednesday, June 20, 2012 9:22 AM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] Pages excluded in dump RHEL 6.2 - set to ffffffffffffffff

----- Original Message -----
> I have a dump from RHEL 6.2 with all these masks set to all f's, so 
> obviously crash excludes them.
> 
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "present"
> WARNING: cannot read cpu_present_map
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "online"
> WARNING: cannot read cpu_online_map
> 
> typedef struct cpumask { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
> 
> #ifndef CONFIG_DISABLE_OBSOLETE_CPUMASK_FUNCTIONS
> /* These strip const, as traditionally they weren't const. */
> #define cpu_possible_map        (*(cpumask_t *)cpu_possible_mask)
> #define cpu_online_map          (*(cpumask_t *)cpu_online_mask)
> #define cpu_present_map         (*(cpumask_t *)cpu_present_mask)
> #define cpu_active_map          (*(cpumask_t *)cpu_active_mask)
> 
> Are these deprecated now and is that why they are fixed at all f's.

No, that's just a common error message for accessing the maps or masks.

When reading them, crash checks first whether there is the older kernel symbol named "cpu_xxx_map", and if it doesn't exists, then it checks for the symbol named "cpu_xxx_mask".  It then tried to read the memory that those symbols point to, which all contained values of ffffffffffffffff.  So clearly there's some kind of mis-match between the symbol values -- modified by using a System.map file -- and what's in the vmcore file. 

(FWIW, the dumpfile couldn't find the page for bogus address, and presumed that it was "excluded" since it's a filtered dump, so that's a non-issue...)

First question: it appears that you are analyzing a RHEL6 kernel, so why are you using a "System.map" argument?  Did you rebuild the kernel but when you did that, you didn't save the rebuilt vmlinux file?

Dave

> 
> [root at centos62 crash]# crash System.map vmlinux 
> va102.2012-06-14.vmcore
> 
> crash 6.0.7
> Copyright (C) 2002-2012  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation Copyright (C) 
> 1999-2006  Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012  
> Fujitsu Limited Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011  NEC Corporation Copyright (C) 1999, 2002, 
> 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public 
> License, and you are welcome to change it and/or distribute copies of 
> it under certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for 
> details.
> 
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> 
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "present"
> WARNING: cannot read cpu_present_map
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "online"
> WARNING: cannot read cpu_online_map
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "cpu_present_map"
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "cpu_present_map"
> crash: page excluded: kernel virtual address: ffffffffffffffff  type: "cpu_online_map"
>   SYSTEM MAP: System.map
> DEBUG KERNEL: vmlinux (2.6.32-220.7.1.el6)
>     DUMPFILE: va102.2012-06-14.vmcore  [PARTIAL DUMP]
>         CPUS: 32
>         DATE: Thu Jun 14 05:24:17 2012
>       UPTIME: 11:13:23
> LOAD AVERAGE: 215.00, 196.78, 135.81
>        TASKS: 1778
>     NODENAME: va1splunkindex02.va1.swtor.int
>      RELEASE: 2.6.32-220.7.1.el6.x86_64
>      VERSION: #1 SMP Tue Mar 6 13:54:51 EST 2012
>      MACHINE: x86_64  (2799 Mhz)
>       MEMORY: 48 GB
>        PANIC: "Oops: 0002 [#1] SMP " (check log for details)
>          PID: 2428
>      COMMAND: "bash"
>         TASK: ffff88063874ab40  [THREAD_INFO: ffff8806dcd1a000]
>          CPU: 0
>        STATE: TASK_RUNNING (PANIC)
> 
> Thanks
> Laurence
> 
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
> 

--
Crash-utility mailing list
Crash-utility at redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility