[Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash

[Worth, Kevin]

Dave,

I do indeed have a /dev/kmem on my live system. One other thing... is the physical limit of 0xb8000000 imposed because of the amount of memory on my system or because it is the maximum addressable memory without using PAE? My kernel does have PAE enabled (CONFIG_HIGHMEM_32G or whatever the option is), though not sure if that makes a difference, just checking. Things work just fine on an identical system that has 2GB of memory instead of 4GB. :\

-Kevin

-----Original Message-----
From: crash-utility-bounces at redhat.com [mailto:crash-utility-bounces at redhat.com] On Behalf Of Dave Anderson
Sent: Friday, October 10, 2008 1:13 PM
To: Discussion list for crash utility usage, maintenance and development
Subject: Re: [Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash


----- "Kevin Worth" <kevin.worth at hp.com> wrote:

> Hi Dave,
>
> I tried changing the PAGE_OFFSET definition in kexec-tools. Didn't
> seem to affect it- crash still fails to load the vmalloc'ed memory. If
> that seems like it absolves kexec-tools of any sins then perhaps we
> can drop the kexec-ml off the CC list.
>
> Your statement "Theoretically, anything at and above 0xb8000000 should
> fail." was accurate, which I saw on my live system (with no dump
> involved). Hoping this provides some insight.

Right -- but when you did the "module 0xf9102280", it read legitimate
data -- but the translated vmalloc address of 0xf9102280 shows a physical
address of 119b76280, i.e. well beyond the physical limit of 0xb8000000:

crash> module 0xf9102280
> struct module {
>   state = MODULE_STATE_LIVE,
>   list = {
>     next = 0xf9073d84,
>     prev = 0x403c63a4
>   },
>   name = "custom_lkm"
> ...
> crash> vtop 0xf9102280
> VIRTUAL   PHYSICAL
> f9102280  119b76280
> ...
> crash> rd -p 119b76000 30
> rd: read error: physical address: 119b76000  type: "32-bit PHYSADDR"
> ...

That being the case, I just remembered something that I had completely
forgotten about -- because of yet another Red Hat imposed restriction.
On RHEL systems, we have the restricted /dev/mem, but in addition to
that /dev/kmem has been completely removed:

  $ ls -l /dev/mem /dev/kmem
  ls: /dev/kmem: No such file or directory
  crw-r----- 1 root kmem 1, 1 Oct  6 09:17 /dev/mem
  $

However, the crash utility, if it realizes that it cannot access a physical
address because it's bigger than the high_memory limit, does this in
read_dev_mem():

        /*
         *  /dev/mem disallows anything >= __pa(high_memory)
         *
         *  However it will allow 64-bit lseeks to anywhere, and when followed
         *  by pulling a 32-bit address from the 64-bit file position, it
         *  quietly returns faulty data from the (wrapped-around) address.
         */
        if (vt->high_memory && (paddr >= (physaddr_t)(VTOP(vt->high_memory)))) {
                readcnt = 0;
                errno = 0;
                goto try_dev_kmem;
        }

So the vmalloc read of 0xf9102280, whose physical address is 0x119b76280 is
greater than the VTOP of 0xf8000000, or b8000000, will goto try_dev_kmem.

First, do you have a /dev/kmem on your system?  If so, the read attempt
continues like so if the passed-in address was a vmalloc address:

        if ((readcnt != cnt) && BITS32() && !readcnt && !errno &&
            IS_VMALLOC_ADDR(addr))
                readcnt = read_dev_kmem(addr, bufptr, cnt);

You'll have to debug crash's memory.c:read_dev_mem() function to determine
whether it followed that path, and successfully read_dev_kmem() above to get
the legitimate module data.  That would be a possible (the only?) explanation
for what you're seeing.

Now, that all being said, debugging the above offers nothing towards the
debugging of the kdump/dumpfile issue.

Dave














--
Crash-utility mailing list
Crash-utility at redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility