[Crash-utility] crash and upstream arm kernel
Dave Young
dyoung at redhat.com
Fri Dec 4 09:17:35 UTC 2015
Hi, Dave
On 12/01/15 at 04:49pm, Dave Anderson wrote:
>
> ----- Original Message -----
> > On 11/29/15 at 10:52pm, Dave Young wrote:
> > [snip]
> >
> > With the new build it fails at later point, but I noticed I enabled
> > CONFIG_DEBUG_INFO_REDUCED=y
> >
> > A retest without CONFIG_DEBUG_INFO_REDUCED succeed with latest latest
> > crash build, but there's some read errors. I'm not sure if they matters
> > but "bt" works well in crash.
> >
> > [snip]
> > GNU gdb (GDB) 7.6
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "armv7l-unknown-linux-gnueabihf"...
> >
> > please wait... (gathering kmem slab cache data)
> > crash: read error: kernel virtual address: ff7fb414 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410
> >
> > crash: read error: kernel virtual address: ff7fac08 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3540: invalid array_cache pointer: ff7fac04
> >
> > crash: read error: kernel virtual address: ff7f7edc type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e35c0: invalid array_cache pointer: ff7f7ed8
> >
> > crash: read error: kernel virtual address: ff7f7cec type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3640: invalid array_cache pointer: ff7f7ce8
> >
> > crash: read error: kernel virtual address: ff7f7c04 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e36c0: invalid array_cache pointer: ff7f7c00
> >
> > crash: read error: kernel virtual address: ff7f7894 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3740: invalid array_cache pointer: ff7f7890
> >
> > crash: read error: kernel virtual address: ff7f77ac type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e37c0: invalid array_cache pointer: ff7f77a8
> >
> > crash: read error: kernel virtual address: ff7f76c4 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3840: invalid array_cache pointer: ff7f76c0
> >
> > crash: read error: kernel virtual address: ff7f74d4 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e38c0: invalid array_cache pointer: ff7f74d0
> >
> > crash: read error: kernel virtual address: ff7f73ec type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3940: invalid array_cache pointer: ff7f73e8
> >
> > crash: read error: kernel virtual address: ff7f7304 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e39c0: invalid array_cache pointer: ff7f7300
> >
> > crash: read error: kernel virtual address: ff7f707c type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3a40: invalid array_cache pointer: ff7f7078
> >
> > crash: read error: kernel virtual address: ff7f6e8c type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3ac0: invalid array_cache pointer: ff7f6e88
> >
> > crash: read error: kernel virtual address: ff7f6c9c type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3b40: invalid array_cache pointer: ff7f6c98
> >
> > crash: read error: kernel virtual address: ff7f6aac type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3bc0: invalid array_cache pointer: ff7f6aa8
> >
> > crash: read error: kernel virtual address: ff7f68bc type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3c40: invalid array_cache pointer: ff7f68b8
> >
> > crash: read error: kernel virtual address: ff7f67d4 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3cc0: invalid array_cache pointer: ff7f67d0
> >
> > crash: read error: kernel virtual address: ff7f65e4 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3d40: invalid array_cache pointer: ff7f65e0
> >
> > crash: read error: kernel virtual address: ff7f63f4 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3dc0: invalid array_cache pointer: ff7f63f0
> >
> > crash: read error: kernel virtual address: ff7f6204 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3e40: invalid array_cache pointer: ff7f6200
> >
> > crash: read error: kernel virtual address: ff7f6014 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3ec0: invalid array_cache pointer: ff7f6010
> >
> > crash: read error: kernel virtual address: ff7f5e0c type: "array cache
> > limit"
> >
> > crash: kmem_cache: df1e3f40: invalid array_cache pointer: ff7f5e08
> >
> > crash: read error: kernel virtual address: ff7f5b04 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df017040: invalid array_cache pointer: ff7f5b00
> >
> > crash: read error: kernel virtual address: ff7f57e0 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df0170c0: invalid array_cache pointer: ff7f57dc
> >
> > crash: read error: kernel virtual address: ff7f5300 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df017140: invalid array_cache pointer: ff7f52fc
> >
> > crash: read error: kernel virtual address: ff7f50ec type: "array cache
> > limit"
> >
> > crash: kmem_cache: df0171c0: invalid array_cache pointer: ff7f50e8
> >
> > crash: read error: kernel virtual address: ff7f4004 type: "array cache
> > limit"
> >
> > crash: kmem_cache: df017240: invalid array_cache pointer: ff7f4000
> > WARNING: SLAB: cannot determine how compound pages are linked
> >
> > KERNEL: vmlinux
> > DUMPFILE: /var/crash/127.0.0.1-2015-11-30-13:10:23/vmcore
> > CPUS: 1
> > DATE: Mon Nov 30 13:10:14 2015
> > UPTIME: 00:00:38
> > LOAD AVERAGE: 0.27, 0.08, 0.03
> > TASKS: 157
> > NODENAME: localhost
> > RELEASE: 4.4.0-rc2+
> > VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
> > MACHINE: armv7l (unknown Mhz)
> > MEMORY: 510 MB
> > PANIC: "sysrq: SysRq : Trigger a crash"
> > PID: 1368
> > COMMAND: "bash"
> > TASK: ddbbeb40 [THREAD_INFO: ddbc4000]
> > CPU: 0
> > STATE: TASK_RUNNING (SYSRQ)
> >
> > crash> bt
> > PID: 1368 TASK: ddbbeb40 CPU: 0 COMMAND: "bash"
> > #0 [<c021e23c>] (sysrq_handle_crash) from [<c021e5a5>]
> > #1 [<c021e5a5>] (__handle_sysrq) from [<c021e969>]
> > #2 [<c021e969>] (write_sysrq_trigger) from [<c01067df>]
> > #3 [<c01067df>] (proc_reg_write) from [<c00ca5db>]
> > #4 [<c00ca5db>] (__vfs_write) from [<c00cac0f>]
> > #5 [<c00cac0f>] (vfs_write) from [<c00cb24b>]
> > #6 [<c00cb24b>] (sys_write) from [<c000ddc1>]
> > crash>
> >
> > > Can you send me the location of the vmlinux and vmcore files offline? I can
> > > try working with them tomorrow with on an x86_64 host with a crash utility
> > > built with "make target=ARM".
> >
> > Will save the files somewhere and let you know..
> >
> > Thanks a lot
> > Dave
> >
>
>
> Dave,
>
> OK, so the problem is that in CONFIG_SLAB kernels, the array_cache
> structure that is pointed to by each kmem_cache.cpu_cache can now
> be a vmalloc'd memory location. That may be a new innovation for
> 32-bit kernels? I don't know, but in any case, all of these error
Hmm, I did not notice the CONFIG_SLAB=y
The vmalloc could be introduced by because percpu could use vmalloced area:
commit bf0dea23a9c094ae869a88bb694fbe966671bf6d
Author: Joonsoo Kim <iamjoonsoo.kim at lge.com>
Date: Thu Oct 9 15:26:27 2014 -0700
mm/slab: use percpu allocator for cpu cache
> types:
>
> crash: kmem_cache: df1e34c0: invalid array_cache pointer: ff7fb410
>
> show array_cache pointers in the vmalloc region. The kmem_cache.cpu_cache
> pointer is actually a per_cpu pointer, but on some of the kmem slab caches,
> the translation of the per_cpu pointer result in a vmalloc() address instead
> of a unity-mapped address. Perhaps there's a point at which the percpu
> memory allocations switch from being unity-mapped structures to being
> vmalloc()'d addresses?
I think you are right though I did not find the point as well.
>
> In any case, at the point in time when the kmem caches are being
> initialized (when the error messages get generated), the 32-bit ARM
> arch has not yet set the base address of the vmalloc region in one
> if its machine-dependent variables. And so the addresses are
> mistakenly translated as if they are unity-mapped. The ARM arch
> sets the internal vmalloc base address variable at a later point
> in time, but it can be done earlier, before the kmem cache initialization
> is started. So with this one-line patch, the dumpfile comes up cleanly:
>
> --- arm.c.orig 2015-12-01 16:36:11.815775008 -0500
> +++ arm.c 2015-12-01 16:25:44.580800293 -0500
> @@ -1539,6 +1539,7 @@
> static ulong
> arm_vmalloc_start(void)
> {
> + machdep->machspec->vmalloc_start_addr = vt->high_memory;
> return vt->high_memory;
> }
>
> $ crash vmlinux vmcore-cp
>
> crash 7.1.4rc21
> Copyright (C) 2002-2014 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-elf-linux"...
>
> KERNEL: vmlinux
> DUMPFILE: vmcore-cp
> CPUS: 1
> DATE: Mon Nov 30 00:10:14 2015
> UPTIME: 00:00:38
> LOAD AVERAGE: 0.27, 0.08, 0.03
> TASKS: 157
> NODENAME: localhost
> RELEASE: 4.4.0-rc2+
> VERSION: #46 SMP Mon Nov 30 12:58:47 CST 2015
> MACHINE: armv7l (unknown Mhz)
> MEMORY: 510 MB
> PANIC: "sysrq: SysRq : Trigger a crash"
> PID: 1368
> COMMAND: "bash"
> TASK: ddbbeb40 [THREAD_INFO: ddbc4000]
> CPU: 0
> STATE: TASK_RUNNING (SYSRQ)
>
> crash>
>
> I can only test the ELF vmcore because I don't have the 32-bit lzo library
> installed for my "make target=ARM" x86 binary. But it won't be any
> different from the one above.
>
> I'll queue the patch for crash-7.1.4.
Cool, Thanks a lot
Dave
More information about the Crash-utility
mailing list