[Crash-utility] crash: read error: kernel virtual address: ...
Marc Smith
msmith626 at gmail.com
Tue Feb 12 18:39:36 UTC 2019
On Tue, Feb 12, 2019 at 12:26 PM Dave Anderson <anderson at redhat.com> wrote:
>
>
>
> ----- Original Message -----
> > On Tue, Feb 12, 2019 at 11:03 AM Dave Anderson <anderson at redhat.com> wrote:
> > >
> > >
> > >
> > > ----- Original Message -----
> > > >
> > > >
> > > > ----- Original Message -----
> > > > > Hi,
> > > > >
> > > > > I'm running vanilla Linux 4.14.91 and I cannot seem to obtain a
> > > > > working crash dump file from any kernel panic. We've experienced
> > > > > several recently, and each has similar output to this below when
> > > > > attempting to use the 'crash' utility:
> > > > >
> > > > > --snip--
> > > > > crash 7.2.3-2.fc29
> > > >
> > > > Have you tried the latest version of crash? Preferably a freshly-built
> > > > version
> > > > from github, or at least the latest Fedora build (crash-7.2.5-2.fc30)?
> > > >
> > > > And does crash run on the live system?
> > > >
> > > > Dave
> > >
> > > Another question -- I see that your kernel is not KASLR, and there was a
> > > recent
> > > change to the PAGE_OFFSET value. Look at your 4.14.91 kernel source tree
> > > and
> > > find the values for __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 in
> > > "arch/x86/include/asm/page_64_types.h".
> > >
> > > If the 4.14.91 kernel has a backport of commit
> > > d52888aa2753e3063a9d3a0c9f72f94aa9809c15
> > > "x86/mm: Move LDT remap out of KASLR region on 5-level paging", then you'll
> > > need
> > > the github version of crash, which has this patch:
> > >
> > > commit c63d678798c21a5379f132f124181715bb1d63d4
> > > Author: Dave Anderson <anderson at redhat.com>
> > > Date: Fri Jan 18 14:19:18 2019 -0500
> > >
> > > Fix for Linux 4.20 and later x86_64 kernels which are NOT
> > > configured with CONFIG_RANDOMIZE_BASE. Linux 4.20 introduced
> > > kernel commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15, titled
> > > "x86/mm: Move LDT remap out of KASLR region on 5-level paging",
> > > which modified the 4-level and 5-level paging PAGE_OFFSET values.
> > > Without this patch, the crash session fails during initialization
> > > with the error message "crash: read error: kernel virtual address:
> > > <address> type: tss_struct ist array". For kernels prior to
> > > Linux 4.20.0 which have backports of the kernel commit, the kernel's
> > > PAGE_OFFSET value must be manually specified via the command line
> > > option "--machdep page_offset=ffff888000000000" for kernels with
> > > 4-level page tables, or "--machdep page_offset=ff11000000000000"
> > > for kernels with 5-level paging. (or alternatively the shorter
> > > version "-m page_offset=<address>" may be used). The command
> > > line option requirement may be revisited in the future.
> > > (anderson at redhat.com)
> > >
> > > And then you'll have to use the "--machdep page_offset=" option.
> >
> > Thanks for the quick responses. In the Linux 4.14.91 source I do not
> > see the __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 macros
> > referenced in "arch/x86/include/asm/page_64_types.h". I didn't follow
> > the commit to see if these values should truly end up in 4.14.x to see
> > how it was back-ported into 4.14.x.
> >
> > I built the latest from the 'master' branch of crash and ran it
> > against the dump file again, received the same errors.
> >
> > I then took the value from page_64_types.h for __PAGE_OFFSET_BASE (we
> > do not have CONFIG_X86_5LEVEL set) and used this value with the
> > '--machdep page_offset=' parameter and it works!
> >
> > For others that might find this... used this value "ffff888000000000"
> > from page_64_types.h:
> > #ifdef CONFIG_X86_5LEVEL
> > #define __PAGE_OFFSET_BASE _AC(0xff11000000000000, UL)
> > #else
> > #define __PAGE_OFFSET_BASE _AC(0xffff888000000000, UL)
> > #endif
> >
> > (We don't have CONFIG_X86_5LEVEL set.)
> >
> > And ran crash with "--machdep page_offset=ffff888000000000" and it
> > works as expected!
> >
> > Dave, thanks so much. One question: Is something abnormal in our
> > kernel config that requires us to use this? Anything you'd recommend
> > changing? If we were to enable KASLR in our kernel config, would we
> > then not need to use the --machdep parameter with crash? Any negative
> > performance impact with enabling KASLR?
> >
> > --Marc
>
> I can't answer that, but kernels configured with KASLR (CONFIG_RANDOMIZE_BASE) do
> have a handy "page_offset_base" symbol that contains the randomize PAGE_OFFSET
> value. So yes, in that case you could avoid the --machdep option.
>
> But reconfiguring the kernel seems a bit like the tail wagging the dog. Note
> that when I did the crash patch, I couldn't come up with a reasonable manner to
> determine what the non-KASLR PAGE_OFFSET value should be during initialization,
> so I defaulted to checking the Linux version where the patch was first introduced:
>
> case POST_GDB:
> + if (THIS_KERNEL_VERSION >= LINUX(4,20,0) && !(machdep->flags & RANDOMIZED)) {
> + machdep->machspec->page_offset = machdep->flags & VM_5LEVEL ?
> + PAGE_OFFSET_5LEVEL_4_20 : PAGE_OFFSET_4LEVEL_4_20;
> + machdep->kvbase = machdep->machspec->page_offset;
> + machdep->identity_map_base = machdep->machspec->page_offset;
> + }
> + /*
> + * --machdep page_offset forced override
> + */
> + if (machdep->machspec->page_offset_force) {
> + machdep->machspec->page_offset = machdep->machspec->page_offset_force;
> + machdep->kvbase = machdep->machspec->page_offset;
> + machdep->identity_map_base = machdep->machspec->page_offset;
> + }
>
> It looks like the kernel patch was backported into Linux 4.14.84, so to satisfy that
> possibility, can you apply the attached patch to the upstream github version of crash,
> and see if it works? If it does, I'll apply it upstream in github, and it will eventually
> show up in crash-7.2.6.
Yes, I applied this patch against a clone of the master branch and I
no longer need to use the --machdep page_offset= parameter when
invoking crash! Thanks so much.
--Marc
>
> Thanks,
> Dave
>
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
More information about the Crash-utility
mailing list