[Crash-utility] The problems when running SuSE 12 on VirtualBox
Dave Anderson
anderson at redhat.com
Thu Nov 19 14:45:06 UTC 2015
----- Original Message -----
> Hi David & Dave,
>
> Executing "crash" on a physical machine (not VirtualBox):
>
> # crash
>
> crash 7.1.3
> Copyright (C) 2002-2014 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> crash: /boot/xen-4.5.gz: original filename unknown
> Use "-f /boot/xen-4.5.gz" on command line to prevent this message.
>
> WARNING: machine type mismatch:
> crash utility: X86_64
> /var/tmp/xen-4.5.gz_VIOmfp: X86
>
> crash: /boot/symtypes-3.12.49-6-default.gz: original filename unknown
> Use "-f /boot/symtypes-3.12.49-6-default.gz" on command line to prevent this message.
>
> crash: /boot/symvers-3.12.49-6-default.gz: original filename unknown
> Use "-f /boot/symvers-3.12.49-6-default.gz" on command line to prevent this message.
>
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: this kernel may be configured with CONFIG_STRICT_DEVMEM, which
> renders /dev/mem unusable as a live memory source.
> crash: trying /proc/kcore as an alternative to /dev/mem
>
> KERNEL: /boot/vmlinux-3.12.49-6-xen.gz
> DEBUGINFO: /usr/lib/debug/boot/vmlinux-3.12.49-6-xen.debug
> DUMPFILE: /proc/kcore
> CPUS: 128
> DATE: Thu Nov 19 09:37:49 2015
> UPTIME: 00:34:57
> LOAD AVERAGE: 1.77, 1.21, 1.02
> TASKS: 1328
> NODENAME: dl980-5
> RELEASE: 3.12.49-6-xen
> VERSION: #1 SMP Mon Oct 26 16:05:37 UTC 2015 (11560c3)
> MACHINE: x86_64 (1995 Mhz)
> MEMORY: 125.9 GB
> PID: 39777
> COMMAND: "crash"
> TASK: ffff881eacaaa100 [THREAD_INFO: ffff881e8ff46000]
> CPU: 3
> STATE: TASK_RUNNING (ACTIVE)
>
> crash>
>
> I can see the crash will use "/proc/kcore" instead of "/dev/mem". So I
> try the same thing on VirtualBox:
>
> # crash /boot/vmlinux-3.12.49-6-xen.gz /proc/kcore
>
> crash 7.1.3
> Copyright (C) 2002-2014 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> KERNEL: /boot/vmlinux-3.12.49-6-xen.gz
> DEBUGINFO: /usr/lib/debug/boot/vmlinux-3.12.49-6-xen.debug
> DUMPFILE: /proc/kcore
> CPUS: 1
> DATE: Thu Nov 19 01:53:01 2015
> UPTIME: 05:42:13
> LOAD AVERAGE: 0.19, 0.06, 0.06
> TASKS: 239
> NODENAME: linux-6ev3
> RELEASE: 3.12.49-6-xen
> VERSION: #1 SMP Mon Oct 26 16:05:37 UTC 2015 (11560c3)
> MACHINE: x86_64 (2594 Mhz)
> MEMORY: 855.2 MB
> PID: 3106
> COMMAND: "crash"
> TASK: ffff88002ec5c040 [THREAD_INFO: ffff88000b3e2000]
> CPU: 0
> STATE: TASK_RUNNING (ACTIVE)
>
> crash>
>
> It seems OK now.
>
> So my questions are:
>
> (1) Is it OK to use "/proc/kcore" instead of "/dev/mem" as a workaround?
> Is there any side-effect?
Yes /proc/kcore is fine, and it's not really a "workaround". Since your kernel
was configured with CONFIG_STRICT_DEVMEM, /dev/mem is unusable because it does not
allow reads above 1MB physical. And so as you saw above on the physical machine,
crash then tries to use /proc/kcore as an alternative, and it worked OK.
I don't really understand the behavior you see on VirtualBox. You are running
the same vmlinux-3.12.49-6-xen.debug kernel, but the /dev/mem restriction is
not recognized, and from your earlier posts, it appears to have continued reading
from /dev/mem, but was getting back bogus data.
Note that on some Red Hat kernel versions, /proc/kcore is also restricted, and that
is why we ship the "/dev/crash" driver for reading live memory.
> (2) Execute "crash -d8" on physical machine will cause crash utility core
> dump.
I can reproduce this, so I'll look into it. It's related to the /dev/mem "test"
to determine whether the kernel was configured with CONFIG_STRICT_DEVMEM, where
it tries to read from pfn 257 (just above the CONFIG_STRICT_DEVMEM limit), but
gets into an infinite loop when used in conjunction with -d.
Anyway, just continue to use /proc/kcore and you should be fine.
Thanks,
Dave
> Use gdb to debug it:
>
> # gdb /usr/bin/crash core-crash-11-0-0-40072-1447945769
> GNU gdb (GDB; SUSE Linux Enterprise 12) 7.9.1
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-suse-linux".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://bugs.opensuse.org/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/bin/crash...Reading symbols from
> /usr/lib/debug/usr/bin/crash.debug...done.
> done.
> [New LWP 40072]
> Core was generated by `crash -d8'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007f5119001fd0 in get_cie_encoding (cie=0x7f5119004cd8) at
> ../../../libgcc/unwind-dw2-fde.c:272
> 272 ../../../libgcc/unwind-dw2-fde.c: No such file or directory.
> (gdb)
> (gdb) bt
> #0 0x00007f5119001fd0 in get_cie_encoding (cie=0x7f5119004cd8) at
> ../../../libgcc/unwind-dw2-fde.c:272
> #1 0x00007f5119002699 in get_fde_encoding (f=0x7f5119006050) at
> ../../../libgcc/unwind-dw2-fde.c:319
> #2 _Unwind_IteratePhdrCallback (info=info at entry=0x7fff463c10e0,
> size=size at entry=64, ptr=ptr at entry=0x7fff463c1160)
> at ../../../libgcc/unwind-dw2-fde-dip.c:408
> #3 0x00007f51196a3f3c in __GI___dl_iterate_phdr
> (callback=callback at entry=0x7f5119002270 <_Unwind_IteratePhdrCallback>,
> data=data at entry=0x7fff463c1160) at dl-iteratephdr.c:76
> #4 0x00007f51190035c3 in _Unwind_Find_FDE (pc=0x7f5119001aa7
> <_Unwind_Backtrace+55>, bases=bases at entry=0x7fff463c1498)
> at ../../../libgcc/unwind-dw2-fde-dip.c:459
> #5 0x00007f5118ffff86 in uw_frame_state_for
> (context=context at entry=0x7fff463c13f0, fs=fs at entry=0x7fff463c1240)
> at ../../../libgcc/unwind-dw2.c:1241
> #6 0x00007f51190011d0 in uw_init_context_1
> (context=context at entry=0x7fff463c13f0,
> outer_cfa=outer_cfa at entry=0x7fff463c16a0,
> outer_ra=0x7f511967bf46 <__GI___backtrace+86>) at
> ../../../libgcc/unwind-dw2.c:1562
> #7 0x00007f5119001aa8 in _Unwind_Backtrace (trace=0x7f511967bdd0
> <backtrace_helper>, trace_argument=0x7fff463c16a0)
> at ../../../libgcc/unwind.inc:283
> #8 0x00007f511967bf46 in __GI___backtrace
> (array=array at entry=0x7fff463c1710, size=size at entry=4) at
> ../sysdeps/x86_64/backtrace.c:109
> #9 0x000000000047add7 in __error (type=type at entry=1,
> fmt=fmt at entry=0x853d38 "read(/dev/mem, %lx, %ld): %ld (%lx)\n") at
> tools.c:52
> #10 0x0000000000490c91 in read_dev_mem (fd=4, bufptr=0x7fff463c1f28,
> cnt=8, addr=0, paddr=1052672) at memory.c:2298
> #11 0x0000000000486398 in readmem (addr=1052672,
> memtype=memtype at entry=4, buffer=buffer at entry=0x7fff463c1f28,
> size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2198
> #12 0x000000000048704a in devmem_is_restricted () at memory.c:2414
> #13 readmem (addr=1052672, memtype=memtype at entry=4,
> buffer=buffer at entry=0x7fff463c1fb8, size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2209
> #14 0x000000000048704a in devmem_is_restricted () at memory.c:2414
> #15 readmem (addr=1052672, memtype=memtype at entry=4,
> buffer=buffer at entry=0x7fff463c2048, size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2209
> #16 0x000000000048704a in devmem_is_restricted () at memory.c:2414
> #17 readmem (addr=1052672, memtype=memtype at entry=4,
> buffer=buffer at entry=0x7fff463c20d8, size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2209
> #18 0x000000000048704a in devmem_is_restricted () at memory.c:2414
> #19 readmem (addr=1052672, memtype=memtype at entry=4,
> buffer=buffer at entry=0x7fff463c2168, size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2209
> #20 0x000000000048704a in devmem_is_restricted () at memory.c:2414
> #21 readmem (addr=1052672, memtype=memtype at entry=4,
> buffer=buffer at entry=0x7fff463c21f8, size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2209
> #22 0x000000000048704a in devmem_is_restricted () at memory.c:2414
> #23 readmem (addr=1052672, memtype=memtype at entry=4,
> buffer=buffer at entry=0x7fff463c2288, size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2209
> #24 0x000000000048704a in devmem_is_restricted () at memory.c:2414
>
> ......
>
> Below is always:
> readmem (addr=1052672, memtype=memtype at entry=4,
> buffer=buffer at entry=0x7fff463c2288, size=size at entry=8,
> type=type at entry=0x8563e1 "devmem_is_allowed - pfn 257",
> error_handle=error_handle at entry=6) at memory.c:2209
> 0x000000000048704a in devmem_is_restricted () at memory.c:2414
>
> Seems a dead-loop, but not sure.
>
> Thanks!
> Best Regards
> Nan Xiao
>
>
> On Thu, Nov 19, 2015 at 1:51 AM, David Mair <dmair at suse.com> wrote:
> > On 11/18/2015 08:00 AM, Dave Anderson wrote:
> >>
> >> <snip>
> >>
> >> Can anybody on this mailing list confirm that crash runs OK on Xen dom0
> >> kernels of this vintage?
> >>
> >
> > It does. I currently do the crash maintenance at SUSE and put together
> > the 7.1.3 case being asked about. It and the kernel being discussed are
> > part of a currently un-released service pack gold master candidate as I
> > write. Myself and one of the Xen team here were able to use crash on a
> > dom0 on bare-metal without the problems being reported by the OP.
> > VirtualBox being the cause looks like a reasonable consideration.
> >
> > --
> > David.
> >
> > --
> > Crash-utility mailing list
> > Crash-utility at redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
More information about the Crash-utility
mailing list