[Crash-utility] Crash fails with the error: "crash: read error: kernel virtual address:..."

[Dave Anderson]

----- "Adhiraj Joshi" <adhiraj at linsyssoft.com> wrote:

> Hi Dave,
> 
> Thanks for the quick reply! I followed approach 1 of disabling
> CONFIG_STRICT_DEVMEM and the problem disappeared. Now a new problem
> :-) On invoking crash, I get the following warning:
> WARNING: cannot access vmalloc'd module memory
> 
> Because of this I am not able to run commands like "mod":
> crash> mod
> mod: cannot access vmalloc'd module memory
> crash>

What you're running into now is yet another problem with the /dev/mem
driver on the 32-bit x86 architecture when used on a live system with
more than 896MB of physical memory.  Here's the beginning of the
/dev/mem driver's read routine in ./drivers/char/mem.c:

  static ssize_t read_mem(struct file * file, char __user * buf,
                          size_t count, loff_t *ppos)
  {
          unsigned long p = *ppos;
          ssize_t read, sz;
          char *ptr;

          if (!valid_phys_addr_range(p, count))
                  return -EFAULT;

where on x86, valid_phys_addr_range() looks like this:

  static inline int valid_phys_addr_range(unsigned long addr, size_t count)
  {
          if (addr + count > __pa(high_memory))
                  return 0;

          return 1;
  }

And "high_memory" is the dividing line between "lowmem" and "highmem",
which is 896MB physical on your machine.  So any reference to a physical
address above 896MB is rejected. 

Kernel module memory is vmalloc'd with the __GFP_HIGHMEM modifier set,
so the allocation is biased to use physical memory from the highmem zone.
That being the case, and since your machine apparently has physical memory
above 896MB, any crash utility request to read module memory that
exists up there fails.  Hence the "cannot access vmalloc'd module
memory" warning.  You'll also see failures to access user memory,
and the PTEs to translate them as well.  But given that the crucial
kernel memory is always located in lowmem, at least you can run a 
crash session.

> Should the approach 2 or 3 that you mentioned below solve this
> problem? I have upgraded crash to the latest version (4.0-7.7).

... [ snip ] ...
 
> > For your kernel, you've got 3 options:
> > 
> > (1) Rebuild your kernel without the CONFIG_STRICT_DEVMEM restriction.
> > (2) Port the Fedora /dev/crash driver (./drivers/char/crash.c) to your kernel.
> > (3) Write a kretprobe module that tinkers with the return value of the
> >     kernel's devmem_is_allowed() function such that it always returns 1.

(3) won't help since you can't get that far -- and since you've turned
off CONFIG_STRICT_DEVMEM, it's unnecessary anyway.

You also can't kludge/kretprobe valid_phys_addr_range() to return 1, because
later on in read_mem() the physical address is translated into a unity-mapped
kernel virtual address here:

                ptr = xlate_dev_mem_ptr(p);
                if (!ptr)
                        return -EFAULT;

                if (copy_to_user(buf, ptr, sz)) {
                        unxlate_dev_mem_ptr(p, ptr);
                        return -EFAULT;
                }

where for x86, xlate_dev_mem_ptr() simply does a __va() on the
physical address:

  void *xlate_dev_mem_ptr(unsigned long phys)
  {
          void *addr;
          unsigned long start = phys & PAGE_MASK;

          /* If page is RAM, we can use __va. Otherwise ioremap and unmap. */
          if (page_is_ram(start >> PAGE_SHIFT))
                  return __va(phys);

and __va() simply adds PAGE_OFFSET to the physical address to make a
unity-mapped kernel virtual address -- which does not apply to "highmem"
physical addresses.

So that basically leaves (2) as the best option.  That driver uses
kmap() on the physical address, and so it can handle any physical
memory request.

The driver should be portable, one caveat being is that it
uses page_is_ram() as a physical memory qualifier, and therefore
requires that the function be EXPORT_SYMBOL()'d to build.  And in
upstream kernels, it currently is not.  

It looks like you may be able to replace page_is_ram() with the
e820_any_mapped() function -- which is exported -- and look for
the E820_RAM "type":

  int
  e820_any_mapped(u64 start, u64 end, unsigned type)
  {
          int i;

          for (i = 0; i < e820.nr_map; i++) {
                  struct e820entry *ei = &e820.map[i];
 
                  if (type && ei->type != type)
                          continue;
                  if (ei->addr >= end || ei->addr + ei->size <= start)
                          continue;
                  return 1;
          }
          return 0;
  }
  EXPORT_SYMBOL_GPL(e820_any_mapped);

Or you can make page_is_ram() EXPORT_SYMBOL_GPL().

Or you can just comment out the page_is_ram() call in the crash
driver's crash.h file.

Another issue with the port is that the driver was written for
kernels prior to the x86/x86_64 merge.  So the driver's
"./include/asm-i386/crash.h" and "./include/asm-x86_64/crash.h"
header files will need to be merged. 

Or you can always boot your kernel with "mem=896M" on the
kernel command line, and the problem goes away...  ;-)

Dave