[Crash-utility] crash: read error: kernel virtual address: ...
Dave Anderson
anderson at redhat.com
Tue Feb 12 16:03:28 UTC 2019
----- Original Message -----
>
>
> ----- Original Message -----
> > Hi,
> >
> > I'm running vanilla Linux 4.14.91 and I cannot seem to obtain a
> > working crash dump file from any kernel panic. We've experienced
> > several recently, and each has similar output to this below when
> > attempting to use the 'crash' utility:
> >
> > --snip--
> > crash 7.2.3-2.fc29
>
> Have you tried the latest version of crash? Preferably a freshly-built version
> from github, or at least the latest Fedora build (crash-7.2.5-2.fc30)?
>
> And does crash run on the live system?
>
> Dave
Another question -- I see that your kernel is not KASLR, and there was a recent
change to the PAGE_OFFSET value. Look at your 4.14.91 kernel source tree and
find the values for __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 in
"arch/x86/include/asm/page_64_types.h".
If the 4.14.91 kernel has a backport of commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15
"x86/mm: Move LDT remap out of KASLR region on 5-level paging", then you'll need
the github version of crash, which has this patch:
commit c63d678798c21a5379f132f124181715bb1d63d4
Author: Dave Anderson <anderson at redhat.com>
Date: Fri Jan 18 14:19:18 2019 -0500
Fix for Linux 4.20 and later x86_64 kernels which are NOT
configured with CONFIG_RANDOMIZE_BASE. Linux 4.20 introduced
kernel commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15, titled
"x86/mm: Move LDT remap out of KASLR region on 5-level paging",
which modified the 4-level and 5-level paging PAGE_OFFSET values.
Without this patch, the crash session fails during initialization
with the error message "crash: read error: kernel virtual address:
<address> type: tss_struct ist array". For kernels prior to
Linux 4.20.0 which have backports of the kernel commit, the kernel's
PAGE_OFFSET value must be manually specified via the command line
option "--machdep page_offset=ffff888000000000" for kernels with
4-level page tables, or "--machdep page_offset=ff11000000000000"
for kernels with 5-level paging. (or alternatively the shorter
version "-m page_offset=<address>" may be used). The command
line option requirement may be revisited in the future.
(anderson at redhat.com)
And then you'll have to use the "--machdep page_offset=" option.
Dave
>
> > Copyright (C) 2002-2017 Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > Copyright (C) 2005, 2011 NEC Corporation
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions. Enter "help copying" to see the conditions.
> > This program has absolutely no warranty. Enter "help warranty" for
> > details.
> >
> > GNU gdb (GDB) 7.6
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> >
> > crash: read error: kernel virtual address: ffff88903f814d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f854d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f894d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f8d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f914d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f954d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f994d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f9d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fa14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fa54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fa94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fad4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fb14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fb54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fb94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fbd4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fc14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fc54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ec14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ec54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ec94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ecd4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ed14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ed54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ed94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03edd4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ee14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ee54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ee94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03eed4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ef14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ef54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03ef94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03efd4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f014d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f054d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fc94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fcd4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fd14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fd54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fd94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fdd4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fe14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fe54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fe94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903fed4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903ff14d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903ff54d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903ff94d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903ffd4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff889040014d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff889040054d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff889040094d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff8890400d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f094d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f0d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f114d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f154d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f194d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f1d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f214d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f254d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f294d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f2d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f314d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f354d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f394d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f3d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f414d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f454d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f494d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88a03f4d4d40 type:
> > "current_task (per_cpu)"
> > crash: read error: kernel virtual address: ffff88903f805024 type:
> > "tss_struct ist array"
> > --snip--
> >
> > I've read a bit about similar errors to this on the mailing list, and
> > some seem to indicate a physical memory offset issue (and/or use of
> > high memory). I see the start and align values are the same in our
> > kernel config so I don't believe that is the culprit (as others had
> > seem to indicate with similar issues):
> > CONFIG_PHYSICAL_START=0x1000000
> > CONFIG_PHYSICAL_ALIGN=0x1000000
> >
> > When running crash with '-d8' I see the READ_ERROR messages:
> > <readmem: ffff88903f814d40, KVADDR, "current_task (per_cpu)", 8,
> > (ROE), 5593c9eb7c60>
> > <read_kdump: addr: ffff88903f814d40 paddr: 903f814d40 cnt: 8>
> > read_netdump: READ_ERROR: offset not found for paddr: 903f814d40
> > crash: read error: kernel virtual address: ffff88903f814d40 type:
> > "current_task (per_cpu)"
> > <readmem: ffff88903f854d40, KVADDR, "current_task (per_cpu)", 8,
> > (ROE), 5593c9eb7c68>
> > <read_kdump: addr: ffff88903f854d40 paddr: 903f854d40 cnt: 8>
> > read_netdump: READ_ERROR: offset not found for paddr: 903f854d40
> > crash: read error: kernel virtual address: ffff88903f854d40 type:
> > "current_task (per_cpu)"
> > <readmem: ffff88903f894d40, KVADDR, "current_task (per_cpu)", 8,
> > (ROE), 5593c9eb7c70>
> > <read_kdump: addr: ffff88903f894d40 paddr: 903f894d40 cnt: 8>
> > read_netdump: READ_ERROR: offset not found for paddr: 903f894d40
> > crash: read error: kernel virtual address: ffff88903f894d40 type:
> > "current_task (per_cpu)"
> > <readmem: ffff88903f8d4d40, KVADDR, "current_task (per_cpu)", 8,
> > (ROE), 5593c9eb7c78>
> > <read_kdump: addr: ffff88903f8d4d40 paddr: 903f8d4d40 cnt: 8>
> > read_netdump: READ_ERROR: offset not found for paddr: 903f8d4d40
> > crash: read error: kernel virtual address: ffff88903f8d4d40 type:
> > "current_task (per_cpu)"
> > <readmem: ffff88903f914d40, KVADDR, "current_task (per_cpu)", 8,
> > (ROE), 5593c9eb7c80>
> > <read_kdump: addr: ffff88903f914d40 paddr: 903f914d40 cnt: 8>
> > read_netdump: READ_ERROR: offset not found for paddr: 903f914d40
> > crash: read error: kernel virtual address: ffff88903f914d40 type:
> > "current_task (per_cpu)"
> > <readmem: ffff88903f954d40, KVADDR, "current_task (per_cpu)", 8,
> > (ROE), 5593c9eb7c88>
> > <read_kdump: addr: ffff88903f954d40 paddr: 903f954d40 cnt: 8>
> > read_netdump: READ_ERROR: offset not found for paddr: 903f954d40
> > crash: read error: kernel virtual address: ffff88903f954d40 type:
> > "current_task (per_cpu)"
> > <readmem: ffff88903f994d40, KVADDR, "current_task (per_cpu)", 8,
> > (ROE), 5593c9eb7c90>
> > <read_kdump: addr: ffff88903f994d40 paddr: 903f994d40 cnt: 8>
> > read_netdump: READ_ERROR: offset not found for paddr: 903f994d40
> > ...
> >
> > I'm not sure where to go from here or what to look at next. These
> > unusable dump files are produced by 'makedumpfile' version 1.6.4; this
> > is the command used to capture the dump file in the crash dump kernel:
> > makedumpfile -E -d 31 /proc/vmcore /mnt/tmp/dumpfile-`date +"%s"`
> >
> > Any help would be greatly appreciated... this previously worked with
> > the same kernel config, the only thing that has changed that I can
> > think of is patch release increments in the Linux kernel version
> > (we've always stayed on Linux 4.14.x). But I'm not sure which one or
> > when (or if that's it at all).
> >
> > I've attached our kernel configuration file as well.
> >
> >
> > Thanks,
> >
> > Marc
> >
> > --
> > Crash-utility mailing list
> > Crash-utility at redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
More information about the Crash-utility
mailing list