[Crash-utility] crash: page excluded: kernel virtual address: ffffffff81e3da50 type: "page_offset_base"
Dave Anderson
anderson at prospeed.net
Thu Nov 23 14:03:29 UTC 2017
> Hi Dave,
> Â Because I didn't subscribe the list, so please CC me when reply:) My
> colleague help to forward your reply to me, so I decide to reply
> directly in order to keep this mail in the thread other than new a thread.
>
> On 11/23/2017 02:04 PM, Fei, Jie/è´¹ æ° wrote:
>>
>>
>>
>> -------- Forwarded Message --------
>> Subject: Re: [Crash-utility] crash: page excluded: kernel virtual
>> address: ffffffff81e3da50 type: "page_offset_base"
>> Date: Wed, 22 Nov 2017 09:35:40 -0500
>> From: Dave Anderson <anderson at redhat.com>
>> Reply-To: Discussion list for crash utility usage, maintenance and
>> development <crash-utility at redhat.com>
>> To: Discussion list for crash utility usage, maintenance and
>> development <crash-utility at redhat.com>
>>
>>
>>
>> ----- Original Message -----
>>> Hi,
>>>
>>> I am using the latest crash tool & kernel 4.14 compiled from source,
>>> and
>>> I got the following error message. As I searched, this is fixed in
>>> crash
>>> 7.2.0, but I still have it here. So, is anyone has a clue?
>>
>> It's always going to be a crap-shoot with very recent upstream kernels,
>> but you haven't given enough information to determine what the issue is.
>>
>
> Sorry, I am to new dump/crash area.
>
>> If the vmcore was created by "virsh dump", and the kernel has KASLR
>> enabled,
>> then it's just not supported at this time. Otherwise, perhaps the
>> output of
>> "crash -d8" may yield some clues.
>>
>
> I am using kdump to create the vmcore. Actually, I did solve this issue
> by add "nokaslr" to kernel parameter when I used the built-in crash of
> Fedora. But when turn to the compiled version of my own, that solution
> don't work. here is the output of `crash -d8`:
>
> $ sudo ./crash -d8 /var/crash/127.0.0.1-2017-11-22-18\:21\:42/vmcore
> ../linux/vmlinux
>
> crash 7.2.0++
> Copyright (C) 2002-2017 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> compressed kdump: header->utsname.machine: x86_64
> compressed kdump: memory bitmap offset: 2000
> diskdump_data:
> filename: /var/crash/127.0.0.1-2017-11-22-18:21:42/vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 1013930
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: IAAS1
> release: 4.14.0
> version: #1 SMP Wed Nov 15 10:32:46 CST 2017
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5a154fac
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 80
> max_mapnr: 1310208
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 4
> tasks[nr_cpus]: 0
> 0
> 0
> 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 1014940
> phys_base: 0
> dump_level: 31 (0x1f)
> (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 5648 (0x1610)
> size_vmcoreinfo: 1883 (0x75b)
> OSRELEASE=4.14.0
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffff81e10280
> SYMBOL(node_online_map)=ffffffff82030e80
> SYMBOL(swapper_pg_dir)=ffffffff81e09000
> SYMBOL(_stext)=ffffffff81000000
> SYMBOL(vmap_area_list)=ffffffff81efc470
> SYMBOL(mem_section)=ffffffff82401dc0
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=172864
> SIZE(zone)=1664
> SIZE(free_area)=104
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=28
> OFFSET(page.mapping)=8
> OFFSET(page.lru)=32
> OFFSET(page._mapcount)=24
> OFFSET(page.private)=48
> OFFSET(page.compound_dtor)=40
> OFFSET(page.compound_order)=44
> OFFSET(page.compound_head)=32
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=172192
> OFFSET(pglist_data.node_start_pfn)=172200
> OFFSET(pglist_data.node_spanned_pages)=172216
> OFFSET(pglist_data.node_id)=172224
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1472
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffff81e58480
> SYMBOL(log_buf_len)=ffffffff81e5847c
> SYMBOL(log_first_idx)=ffffffff823340f8
> SYMBOL(clear_idx)=ffffffff823340cc
> SYMBOL(log_next_idx)=ffffffff823340e8
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=6
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(phys_base)=0
> SYMBOL(init_top_pgt)=ffffffff81e09000
> SYMBOL(node_data)=ffffffff8202c6c0
> LENGTH(node_data)=1024
> KERNELOFFSET=0
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> CRASHTIME=1511346092
> offset_note: 4200 (0x1068)
> size_note: 3332 (0xd04)
> notes_buf: 1015950
> num_prstatus_notes: 4
> notes[0]: 1015950 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 114 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: ffffffff82445b80 R14: ffffffff822f3422
> R13: 0000000000000020 R12: 00000000000026f5
> RBP: ffffc90000adbce0 RBX: ffffffff82445b80
> R11: ffffffff822f342d R10: 0000000000000000
> R9: 000000000000000f R8: 0000000000000000
> RAX: 000000799065724b RCX: 0000007990656c1b
> RDX: 0000000000000000 RSI: 0000000000000000
> RDI: 0000000000000cdd RIP: ffffffff8188600f
> RFLAGS: 0000000000000097 RSP: ffffc90000adbce0
> FS_BASE: 0000000000000000
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> notes[1]: 1015ab4 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 2030 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: ffff880131999500 R14: ffffffff81f76fa0
> R13: 0000000000000000 R12: 0000000000000007
> RBP: ffffc90001dabdd8 RBX: 0000000000000063
> R11: ffffffff822f342d R10: 0000000000000001
> R9: 0000000000000007 R8: 00000000000002d9
> RAX: 000000000000000f RCX: 0000000000000000
> RDX: 0000000000000000 RSI: ffff88013fa8e138
> RDI: 0000000000000063 RIP: ffffffff81532ad6
> RFLAGS: 0000000000010282 RSP: ffffc90001dabdd8
> FS_BASE: 00007fe58251fb80
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> notes[2]: 1015c18 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 0 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: ffffffff81f65618 R14: 0000000000000020
> R13: 0000000000000004 R12: 0000000000000003
> RBP: ffffc900006a3e60 RBX: 0000000000000008
> R11: 000000000000298b R10: ffffc900006a3e40
> R9: 0000000000000018 R8: 00000000000054a2
> RAX: 0000000000000020 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffffffff81f65480
> RDI: 0000000000000002 RIP: ffffffff81895ddc
> RFLAGS: 0000000000000046 RSP: ffffc900006a3e40
> FS_BASE: 0000000000000000
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> notes[3]: 1015d7c (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 0 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: ffffffff81f65618 R14: 0000000000000020
> R13: 0000000000000004 R12: 0000000000000003
> RBP: ffffc900006abe60 RBX: 0000000000000008
> R11: 000000000000037e R10: ffffc900006abe40
> R9: 0000000000000008 R8: 00000000ffffffff
> RAX: 0000000000000020 RCX: 0000000000000001
> RDX: 0000000000000000 RSI: ffffffff81f65480
> RDI: 0000000000000003 RIP: ffffffff81895ddc
> RFLAGS: 0000000000000046 RSP: ffffc900006abe40
> FS_BASE: 0000000000000000
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> 11cc (NT_PRSTATUS)
> 1330 (NT_PRSTATUS)
> 1494 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 1310208 (0x13fe00)
>
> data_offset: 52000
> block_size: 4096
> block_shift: 12
> bitmap: 7f8ff466c010
> bitmap_len: 327680
> max_mapnr: 1310208 (0x13fe00)
> dumpable_bitmap: 7f8ff461b010
> byte: 0
> bit: 0
> compressed_page: 10470a0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1037090
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1038090
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1039090
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 103a090
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 103b090
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 103c090
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 103d090
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 103e090
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 103f090
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1040090
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1041090
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1042090
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1043090
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1044090
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1045090
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 1046090
> pg_hit_count: 0
>
> page_cache_buf: 1037090
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 1036680
> readmem: read_diskdump()
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 0 -> 0
> gdb ../linux/vmlinux
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(288 -> 0)
> GETBUF(1500 -> 1)
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffff81e3da50, KVADDR, "page_offset_base", 8, (FOE), d83568>
> <read_diskdump: addr: ffffffff81e3da50 paddr: 1e3da50 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 1e3da50/1e3d
> crash: page excluded: kernel virtual address: ffffffff81e3da50 type:
> "page_offset_base"
>
> --
> Sincerely,
> Cao jin
OK, so you seem to be describing two issues:
(1) With KASLR enabled, the vmcore cannot be read by either
Fedora's 7.2.0-1 or upstream 7.2.0++.
(2) With KASLR disabled, the vmcore can be read by Fedora's
7.2.0-1, but cannot be read by 7.2.0++,
When you run crash on the live system, what happens when:
(1) KASLR is enabled, running Fedora's 7.2.0-1
(2) KASLR is enabled, running upstream 7.2.0++
(2) KASLR is disabled, running Fedora's 7.2.0-1
(3) KASLR is disabled, running upstream 7.2.0++
I don't see anything unusual in the output above, presuming the
the phys_base of 0 is correct. When you run crash on the live
system, what is the output of "help -m | grep phys_base"?
I need to compare the crash -d8 output above with the same
crash -d8 output from the working session using the same
KASLR disabled vmcore, showing at least up until the point that
it does the readmem of "page_offset_base" above?
Also, if you can make the 2 vmcores available, please send me
the details of how to download them in a private email to
anderson at redhat.com or anderson at prospeed.net. If you have
modified/rebuilt the Fedora kernel, I would also need a copy
of the vmlinux file as well.
We are on Thanksgiving holiday until next Monday, so I won't
be able to spend any time looking at this issue until then.
Dave
More information about the Crash-utility
mailing list