[Crash-utility] crash: read error: kernel virtual address: ...
Marc Smith
msmith626 at gmail.com
Tue Feb 12 17:02:28 UTC 2019
On Tue, Feb 12, 2019 at 11:03 AM Dave Anderson <anderson at redhat.com> wrote:
>
>
>
> ----- Original Message -----
> >
> >
> > ----- Original Message -----
> > > Hi,
> > >
> > > I'm running vanilla Linux 4.14.91 and I cannot seem to obtain a
> > > working crash dump file from any kernel panic. We've experienced
> > > several recently, and each has similar output to this below when
> > > attempting to use the 'crash' utility:
> > >
> > > --snip--
> > > crash 7.2.3-2.fc29
> >
> > Have you tried the latest version of crash? Preferably a freshly-built version
> > from github, or at least the latest Fedora build (crash-7.2.5-2.fc30)?
> >
> > And does crash run on the live system?
> >
> > Dave
>
> Another question -- I see that your kernel is not KASLR, and there was a recent
> change to the PAGE_OFFSET value. Look at your 4.14.91 kernel source tree and
> find the values for __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 in
> "arch/x86/include/asm/page_64_types.h".
>
> If the 4.14.91 kernel has a backport of commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15
> "x86/mm: Move LDT remap out of KASLR region on 5-level paging", then you'll need
> the github version of crash, which has this patch:
>
> commit c63d678798c21a5379f132f124181715bb1d63d4
> Author: Dave Anderson <anderson at redhat.com>
> Date: Fri Jan 18 14:19:18 2019 -0500
>
> Fix for Linux 4.20 and later x86_64 kernels which are NOT
> configured with CONFIG_RANDOMIZE_BASE. Linux 4.20 introduced
> kernel commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15, titled
> "x86/mm: Move LDT remap out of KASLR region on 5-level paging",
> which modified the 4-level and 5-level paging PAGE_OFFSET values.
> Without this patch, the crash session fails during initialization
> with the error message "crash: read error: kernel virtual address:
> <address> type: tss_struct ist array". For kernels prior to
> Linux 4.20.0 which have backports of the kernel commit, the kernel's
> PAGE_OFFSET value must be manually specified via the command line
> option "--machdep page_offset=ffff888000000000" for kernels with
> 4-level page tables, or "--machdep page_offset=ff11000000000000"
> for kernels with 5-level paging. (or alternatively the shorter
> version "-m page_offset=<address>" may be used). The command
> line option requirement may be revisited in the future.
> (anderson at redhat.com)
>
> And then you'll have to use the "--machdep page_offset=" option.
Thanks for the quick responses. In the Linux 4.14.91 source I do not
see the __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 macros
referenced in "arch/x86/include/asm/page_64_types.h". I didn't follow
the commit to see if these values should truly end up in 4.14.x to see
how it was back-ported into 4.14.x.
I built the latest from the 'master' branch of crash and ran it
against the dump file again, received the same errors.
I then took the value from page_64_types.h for __PAGE_OFFSET_BASE (we
do not have CONFIG_X86_5LEVEL set) and used this value with the
'--machdep page_offset=' parameter and it works!
For others that might find this... used this value "ffff888000000000"
from page_64_types.h:
#ifdef CONFIG_X86_5LEVEL
#define __PAGE_OFFSET_BASE _AC(0xff11000000000000, UL)
#else
#define __PAGE_OFFSET_BASE _AC(0xffff888000000000, UL)
#endif
(We don't have CONFIG_X86_5LEVEL set.)
And ran crash with "--machdep page_offset=ffff888000000000" and it
works as expected!
Dave, thanks so much. One question: Is something abnormal in our
kernel config that requires us to use this? Anything you'd recommend
changing? If we were to enable KASLR in our kernel config, would we
then not need to use the --machdep parameter with crash? Any negative
performance impact with enabling KASLR?
--Marc
>
> Dave
>
>
> >
> > > Copyright (C) 2002-2017 Red Hat, Inc.
> > > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> > > Copyright (C) 1999-2006 Hewlett-Packard Co
> > > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> > > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> > > Copyright (C) 2005, 2011 NEC Corporation
> > > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> > > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> > > This program is free software, covered by the GNU General Public License,
> > > and you are welcome to change it and/or distribute copies of it under
> > > certain conditions. Enter "help copying" to see the conditions.
> > > This program has absolutely no warranty. Enter "help warranty" for
> > > details.
> > >
> > > GNU gdb (GDB) 7.6
> > > Copyright (C) 2013 Free Software Foundation, Inc.
> > > License GPLv3+: GNU GPL version 3 or later
> > > <http://gnu.org/licenses/gpl.html>
> > > This is free software: you are free to change and redistribute it.
> > > There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> > > and "show warranty" for details.
> > > This GDB was configured as "x86_64-unknown-linux-gnu"...
> > >
> > > crash: read error: kernel virtual address: ffff88903f814d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f854d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f894d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f8d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f914d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f954d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f994d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f9d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fa14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fa54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fa94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fad4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fb14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fb54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fb94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fbd4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fc14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fc54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ec14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ec54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ec94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ecd4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ed14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ed54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ed94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03edd4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ee14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ee54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ee94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03eed4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ef14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ef54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03ef94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03efd4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f014d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f054d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fc94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fcd4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fd14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fd54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fd94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fdd4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fe14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fe54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fe94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903fed4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903ff14d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903ff54d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903ff94d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903ffd4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff889040014d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff889040054d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff889040094d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff8890400d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f094d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f0d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f114d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f154d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f194d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f1d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f214d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f254d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f294d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f2d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f314d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f354d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f394d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f3d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f414d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f454d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f494d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88a03f4d4d40 type:
> > > "current_task (per_cpu)"
> > > crash: read error: kernel virtual address: ffff88903f805024 type:
> > > "tss_struct ist array"
> > > --snip--
> > >
> > > I've read a bit about similar errors to this on the mailing list, and
> > > some seem to indicate a physical memory offset issue (and/or use of
> > > high memory). I see the start and align values are the same in our
> > > kernel config so I don't believe that is the culprit (as others had
> > > seem to indicate with similar issues):
> > > CONFIG_PHYSICAL_START=0x1000000
> > > CONFIG_PHYSICAL_ALIGN=0x1000000
> > >
> > > When running crash with '-d8' I see the READ_ERROR messages:
> > > <readmem: ffff88903f814d40, KVADDR, "current_task (per_cpu)", 8,
> > > (ROE), 5593c9eb7c60>
> > > <read_kdump: addr: ffff88903f814d40 paddr: 903f814d40 cnt: 8>
> > > read_netdump: READ_ERROR: offset not found for paddr: 903f814d40
> > > crash: read error: kernel virtual address: ffff88903f814d40 type:
> > > "current_task (per_cpu)"
> > > <readmem: ffff88903f854d40, KVADDR, "current_task (per_cpu)", 8,
> > > (ROE), 5593c9eb7c68>
> > > <read_kdump: addr: ffff88903f854d40 paddr: 903f854d40 cnt: 8>
> > > read_netdump: READ_ERROR: offset not found for paddr: 903f854d40
> > > crash: read error: kernel virtual address: ffff88903f854d40 type:
> > > "current_task (per_cpu)"
> > > <readmem: ffff88903f894d40, KVADDR, "current_task (per_cpu)", 8,
> > > (ROE), 5593c9eb7c70>
> > > <read_kdump: addr: ffff88903f894d40 paddr: 903f894d40 cnt: 8>
> > > read_netdump: READ_ERROR: offset not found for paddr: 903f894d40
> > > crash: read error: kernel virtual address: ffff88903f894d40 type:
> > > "current_task (per_cpu)"
> > > <readmem: ffff88903f8d4d40, KVADDR, "current_task (per_cpu)", 8,
> > > (ROE), 5593c9eb7c78>
> > > <read_kdump: addr: ffff88903f8d4d40 paddr: 903f8d4d40 cnt: 8>
> > > read_netdump: READ_ERROR: offset not found for paddr: 903f8d4d40
> > > crash: read error: kernel virtual address: ffff88903f8d4d40 type:
> > > "current_task (per_cpu)"
> > > <readmem: ffff88903f914d40, KVADDR, "current_task (per_cpu)", 8,
> > > (ROE), 5593c9eb7c80>
> > > <read_kdump: addr: ffff88903f914d40 paddr: 903f914d40 cnt: 8>
> > > read_netdump: READ_ERROR: offset not found for paddr: 903f914d40
> > > crash: read error: kernel virtual address: ffff88903f914d40 type:
> > > "current_task (per_cpu)"
> > > <readmem: ffff88903f954d40, KVADDR, "current_task (per_cpu)", 8,
> > > (ROE), 5593c9eb7c88>
> > > <read_kdump: addr: ffff88903f954d40 paddr: 903f954d40 cnt: 8>
> > > read_netdump: READ_ERROR: offset not found for paddr: 903f954d40
> > > crash: read error: kernel virtual address: ffff88903f954d40 type:
> > > "current_task (per_cpu)"
> > > <readmem: ffff88903f994d40, KVADDR, "current_task (per_cpu)", 8,
> > > (ROE), 5593c9eb7c90>
> > > <read_kdump: addr: ffff88903f994d40 paddr: 903f994d40 cnt: 8>
> > > read_netdump: READ_ERROR: offset not found for paddr: 903f994d40
> > > ...
> > >
> > > I'm not sure where to go from here or what to look at next. These
> > > unusable dump files are produced by 'makedumpfile' version 1.6.4; this
> > > is the command used to capture the dump file in the crash dump kernel:
> > > makedumpfile -E -d 31 /proc/vmcore /mnt/tmp/dumpfile-`date +"%s"`
> > >
> > > Any help would be greatly appreciated... this previously worked with
> > > the same kernel config, the only thing that has changed that I can
> > > think of is patch release increments in the Linux kernel version
> > > (we've always stayed on Linux 4.14.x). But I'm not sure which one or
> > > when (or if that's it at all).
> > >
> > > I've attached our kernel configuration file as well.
> > >
> > >
> > > Thanks,
> > >
> > > Marc
> > >
> > > --
> > > Crash-utility mailing list
> > > Crash-utility at redhat.com
> > > https://www.redhat.com/mailman/listinfo/crash-utility
> >
> > --
> > Crash-utility mailing list
> > Crash-utility at redhat.com
> > https://www.redhat.com/mailman/listinfo/crash-utility
> >
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
More information about the Crash-utility
mailing list