[Crash-utility] crash and upstream arm kernel
Dave Anderson
anderson at redhat.com
Tue Dec 1 21:49:53 UTC 2015
----- Original Message -----
> On 11/29/15 at 10:52pm, Dave Young wrote:
> [snip]
>
> With the new build it fails at later point, but I noticed I enabled
> CONFIG_DEBUG_INFO_REDUCED=y
>
> A retest without CONFIG_DEBUG_INFO_REDUCED succeed with latest latest
> crash build, but there's some read errors. I'm not sure if they matters
> but "bt" works well in crash.
>
> [snip]
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "armv7l-unknown-linux-gnueabihf"...
>
> please wait... (gathering kmem slab cache data)
> crash: read error: kernel virtual address: ff7fb414 type: "array cache
> limit"
>
> crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410
>
> crash: read error: kernel virtual address: ff7fac08 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3540: invalid array_cache pointer: ff7fac04
>
> crash: read error: kernel virtual address: ff7f7edc type: "array cache
> limit"
>
> crash: kmem_cache: df1e35c0: invalid array_cache pointer: ff7f7ed8
>
> crash: read error: kernel virtual address: ff7f7cec type: "array cache
> limit"
>
> crash: kmem_cache: df1e3640: invalid array_cache pointer: ff7f7ce8
>
> crash: read error: kernel virtual address: ff7f7c04 type: "array cache
> limit"
>
> crash: kmem_cache: df1e36c0: invalid array_cache pointer: ff7f7c00
>
> crash: read error: kernel virtual address: ff7f7894 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3740: invalid array_cache pointer: ff7f7890
>
> crash: read error: kernel virtual address: ff7f77ac type: "array cache
> limit"
>
> crash: kmem_cache: df1e37c0: invalid array_cache pointer: ff7f77a8
>
> crash: read error: kernel virtual address: ff7f76c4 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3840: invalid array_cache pointer: ff7f76c0
>
> crash: read error: kernel virtual address: ff7f74d4 type: "array cache
> limit"
>
> crash: kmem_cache: df1e38c0: invalid array_cache pointer: ff7f74d0
>
> crash: read error: kernel virtual address: ff7f73ec type: "array cache
> limit"
>
> crash: kmem_cache: df1e3940: invalid array_cache pointer: ff7f73e8
>
> crash: read error: kernel virtual address: ff7f7304 type: "array cache
> limit"
>
> crash: kmem_cache: df1e39c0: invalid array_cache pointer: ff7f7300
>
> crash: read error: kernel virtual address: ff7f707c type: "array cache
> limit"
>
> crash: kmem_cache: df1e3a40: invalid array_cache pointer: ff7f7078
>
> crash: read error: kernel virtual address: ff7f6e8c type: "array cache
> limit"
>
> crash: kmem_cache: df1e3ac0: invalid array_cache pointer: ff7f6e88
>
> crash: read error: kernel virtual address: ff7f6c9c type: "array cache
> limit"
>
> crash: kmem_cache: df1e3b40: invalid array_cache pointer: ff7f6c98
>
> crash: read error: kernel virtual address: ff7f6aac type: "array cache
> limit"
>
> crash: kmem_cache: df1e3bc0: invalid array_cache pointer: ff7f6aa8
>
> crash: read error: kernel virtual address: ff7f68bc type: "array cache
> limit"
>
> crash: kmem_cache: df1e3c40: invalid array_cache pointer: ff7f68b8
>
> crash: read error: kernel virtual address: ff7f67d4 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3cc0: invalid array_cache pointer: ff7f67d0
>
> crash: read error: kernel virtual address: ff7f65e4 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3d40: invalid array_cache pointer: ff7f65e0
>
> crash: read error: kernel virtual address: ff7f63f4 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3dc0: invalid array_cache pointer: ff7f63f0
>
> crash: read error: kernel virtual address: ff7f6204 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3e40: invalid array_cache pointer: ff7f6200
>
> crash: read error: kernel virtual address: ff7f6014 type: "array cache
> limit"
>
> crash: kmem_cache: df1e3ec0: invalid array_cache pointer: ff7f6010
>
> crash: read error: kernel virtual address: ff7f5e0c type: "array cache
> limit"
>
> crash: kmem_cache: df1e3f40: invalid array_cache pointer: ff7f5e08
>
> crash: read error: kernel virtual address: ff7f5b04 type: "array cache
> limit"
>
> crash: kmem_cache: df017040: invalid array_cache pointer: ff7f5b00
>
> crash: read error: kernel virtual address: ff7f57e0 type: "array cache
> limit"
>
> crash: kmem_cache: df0170c0: invalid array_cache pointer: ff7f57dc
>
> crash: read error: kernel virtual address: ff7f5300 type: "array cache
> limit"
>
> crash: kmem_cache: df017140: invalid array_cache pointer: ff7f52fc
>
> crash: read error: kernel virtual address: ff7f50ec type: "array cache
> limit"
>
> crash: kmem_cache: df0171c0: invalid array_cache pointer: ff7f50e8
>
> crash: read error: kernel virtual address: ff7f4004 type: "array cache
> limit"
>
> crash: kmem_cache: df017240: invalid array_cache pointer: ff7f4000
> WARNING: SLAB: cannot determine how compound pages are linked
>
> KERNEL: vmlinux
> DUMPFILE: /var/crash/127.0.0.1-2015-11-30-13:10:23/vmcore
> CPUS: 1
> DATE: Mon Nov 30 13:10:14 2015
> UPTIME: 00:00:38
> LOAD AVERAGE: 0.27, 0.08, 0.03
> TASKS: 157
> NODENAME: localhost
> RELEASE: 4.4.0-rc2+
> VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
> MACHINE: armv7l (unknown Mhz)
> MEMORY: 510 MB
> PANIC: "sysrq: SysRq : Trigger a crash"
> PID: 1368
> COMMAND: "bash"
> TASK: ddbbeb40 [THREAD_INFO: ddbc4000]
> CPU: 0
> STATE: TASK_RUNNING (SYSRQ)
>
> crash> bt
> PID: 1368 TASK: ddbbeb40 CPU: 0 COMMAND: "bash"
> #0 [<c021e23c>] (sysrq_handle_crash) from [<c021e5a5>]
> #1 [<c021e5a5>] (__handle_sysrq) from [<c021e969>]
> #2 [<c021e969>] (write_sysrq_trigger) from [<c01067df>]
> #3 [<c01067df>] (proc_reg_write) from [<c00ca5db>]
> #4 [<c00ca5db>] (__vfs_write) from [<c00cac0f>]
> #5 [<c00cac0f>] (vfs_write) from [<c00cb24b>]
> #6 [<c00cb24b>] (sys_write) from [<c000ddc1>]
> crash>
>
> > Can you send me the location of the vmlinux and vmcore files offline? I can
> > try working with them tomorrow with on an x86_64 host with a crash utility
> > built with "make target=ARM".
>
> Will save the files somewhere and let you know..
>
> Thanks a lot
> Dave
>
Dave,
OK, so the problem is that in CONFIG_SLAB kernels, the array_cache
structure that is pointed to by each kmem_cache.cpu_cache can now
be a vmalloc'd memory location. That may be a new innovation for
32-bit kernels? I don't know, but in any case, all of these error
types:
crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410
show array_cache pointers in the vmalloc region. The kmem_cache.cpu_cache
pointer is actually a per_cpu pointer, but on some of the kmem slab caches,
the translation of the per_cpu pointer result in a vmalloc() address instead
of a unity-mapped address. Perhaps there's a point at which the percpu
memory allocations switch from being unity-mapped structures to being
vmalloc()'d addresses?
In any case, at the point in time when the kmem caches are being
initialized (when the error messages get generated), the 32-bit ARM
arch has not yet set the base address of the vmalloc region in one
if its machine-dependent variables. And so the addresses are
mistakenly translated as if they are unity-mapped. The ARM arch
sets the internal vmalloc base address variable at a later point
in time, but it can be done earlier, before the kmem cache initialization
is started. So with this one-line patch, the dumpfile comes up cleanly:
--- arm.c.orig 2015-12-01 16:36:11.815775008 -0500
+++ arm.c 2015-12-01 16:25:44.580800293 -0500
@@ -1539,6 +1539,7 @@
static ulong
arm_vmalloc_start(void)
{
+ machdep->machspec->vmalloc_start_addr = vt->high_memory;
return vt->high_memory;
}
$ crash vmlinux vmcore-cp
crash 7.1.4rc21
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-elf-linux"...
KERNEL: vmlinux
DUMPFILE: vmcore-cp
CPUS: 1
DATE: Mon Nov 30 00:10:14 2015
UPTIME: 00:00:38
LOAD AVERAGE: 0.27, 0.08, 0.03
TASKS: 157
NODENAME: localhost
RELEASE: 4.4.0-rc2+
VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
MACHINE: armv7l (unknown Mhz)
MEMORY: 510 MB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 1368
COMMAND: "bash"
TASK: ddbbeb40 [THREAD_INFO: ddbc4000]
CPU: 0
STATE: TASK_RUNNING (SYSRQ)
crash>
I can only test the ELF vmcore because I don't have the 32-bit lzo library
installed for my "make target=ARM" x86 binary. But it won't be any
different from the one above.
I'll queue the patch for crash-7.1.4.
Thanks for the report,
Dave
More information about the Crash-utility
mailing list