[Crash-utility] crash: read error: kernel virtual address: ...

[Dave Anderson]

----- Original Message -----
> On Tue, Feb 12, 2019 at 11:03 AM Dave Anderson <anderson at redhat.com> wrote:
> >
> >
> >
> > ----- Original Message -----
> > >
> > >
> > > ----- Original Message -----
> > > > Hi,
> > > >
> > > > I'm running vanilla Linux 4.14.91 and I cannot seem to obtain a
> > > > working crash dump file from any kernel panic. We've experienced
> > > > several recently, and each has similar output to this below when
> > > > attempting to use the 'crash' utility:
> > > >
> > > > --snip--
> > > > crash 7.2.3-2.fc29
> > >
> > > Have you tried the latest version of crash?  Preferably a freshly-built
> > > version
> > > from github, or at least the latest Fedora build (crash-7.2.5-2.fc30)?
> > >
> > > And does crash run on the live system?
> > >
> > > Dave
> >
> > Another question -- I see that your kernel is not KASLR, and there was a
> > recent
> > change to the PAGE_OFFSET value.  Look at your 4.14.91 kernel source tree
> > and
> > find the values for __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 in
> > "arch/x86/include/asm/page_64_types.h".
> >
> > If the 4.14.91 kernel has a backport of commit
> > d52888aa2753e3063a9d3a0c9f72f94aa9809c15
> > "x86/mm: Move LDT remap out of KASLR region on 5-level paging", then you'll
> > need
> > the github version of crash, which has this patch:
> >
> >   commit c63d678798c21a5379f132f124181715bb1d63d4
> >   Author: Dave Anderson <anderson at redhat.com>
> >   Date:   Fri Jan 18 14:19:18 2019 -0500
> >
> >     Fix for Linux 4.20 and later x86_64 kernels which are NOT
> >     configured with CONFIG_RANDOMIZE_BASE.  Linux 4.20 introduced
> >     kernel commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15, titled
> >     "x86/mm: Move LDT remap out of KASLR region on 5-level paging",
> >     which modified the 4-level and 5-level paging PAGE_OFFSET values.
> >     Without this patch, the crash session fails during initialization
> >     with the error message "crash: read error: kernel virtual address:
> >     <address>  type: tss_struct ist array".  For kernels prior to
> >     Linux 4.20.0 which have backports of the kernel commit, the kernel's
> >     PAGE_OFFSET value must be manually specified via the command line
> >     option "--machdep page_offset=ffff888000000000" for kernels with
> >     4-level page tables, or "--machdep page_offset=ff11000000000000"
> >     for kernels with 5-level paging.  (or alternatively the shorter
> >     version "-m page_offset=<address>" may be used).  The command
> >     line option requirement may be revisited in the future.
> >     (anderson at redhat.com)
> >
> > And then you'll have to use the "--machdep page_offset=" option.
> 
> Thanks for the quick responses. In the Linux 4.14.91 source I do not
> see the __PAGE_OFFSET_BASE_L4 and __PAGE_OFFSET_BASE_L5 macros
> referenced in "arch/x86/include/asm/page_64_types.h". I didn't follow
> the commit to see if these values should truly end up in 4.14.x to see
> how it was back-ported into 4.14.x.
> 
> I built the latest from the 'master' branch of crash and ran it
> against the dump file again, received the same errors.
> 
> I then took the value from page_64_types.h for __PAGE_OFFSET_BASE (we
> do not have CONFIG_X86_5LEVEL set) and used this value with the
> '--machdep page_offset=' parameter and it works!
> 
> For others that might find this... used this value "ffff888000000000"
> from page_64_types.h:
> #ifdef CONFIG_X86_5LEVEL
> #define __PAGE_OFFSET_BASE      _AC(0xff11000000000000, UL)
> #else
> #define __PAGE_OFFSET_BASE      _AC(0xffff888000000000, UL)
> #endif
> 
> (We don't have CONFIG_X86_5LEVEL set.)
> 
> And ran crash with "--machdep page_offset=ffff888000000000" and it
> works as expected!
> 
> Dave, thanks so much. One question: Is something abnormal in our
> kernel config that requires us to use this? Anything you'd recommend
> changing? If we were to enable KASLR in our kernel config, would we
> then not need to use the --machdep parameter with crash? Any negative
> performance impact with enabling KASLR?
> 
> --Marc

I can't answer that, but kernels configured with KASLR (CONFIG_RANDOMIZE_BASE) do
have a handy "page_offset_base" symbol that contains the randomize PAGE_OFFSET
value.  So yes, in that case you could avoid the --machdep option.  

But reconfiguring the kernel seems a bit like the tail wagging the dog.  Note
that when I did the crash patch, I couldn't come up with a reasonable manner to
determine what the non-KASLR PAGE_OFFSET value should be during initialization,
so I defaulted to checking the Linux version where the patch was first introduced:

        case POST_GDB:
+               if (THIS_KERNEL_VERSION >= LINUX(4,20,0) && !(machdep->flags & RANDOMIZED)) {
+                       machdep->machspec->page_offset = machdep->flags & VM_5LEVEL ?
+                               PAGE_OFFSET_5LEVEL_4_20 : PAGE_OFFSET_4LEVEL_4_20;
+                       machdep->kvbase = machdep->machspec->page_offset; 
+                       machdep->identity_map_base = machdep->machspec->page_offset; 
+               }
+               /* 
+                * --machdep page_offset forced override 
+                */
+               if (machdep->machspec->page_offset_force) {
+                       machdep->machspec->page_offset = machdep->machspec->page_offset_force;
+                       machdep->kvbase = machdep->machspec->page_offset; 
+                       machdep->identity_map_base = machdep->machspec->page_offset; 
+               }

It looks like the kernel patch was backported into Linux 4.14.84, so to satisfy that
possibility, can you apply the attached patch to the upstream github version of crash, 
and see if it works?  If it does, I'll apply it upstream in github, and it will eventually
show up in crash-7.2.6.

Thanks, 
  Dave



-------------- next part --------------
A non-text attachment was scrubbed...
Name: page_offset.patch
Type: text/x-patch
Size: 625 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/crash-utility/attachments/20190212/55c7d5cf/attachment.bin>