[Crash-utility] crash: page excluded: kernel virtual address: ffffffff81e3da50 type: "page_offset_base"
Cao jin
caoj.fnst at cn.fujitsu.com
Mon Nov 27 10:41:57 UTC 2017
Hi Dave,
Information update: I think I found the root cause.
During the test in last week, I always use makedumpfile, use
/etc/kdump.conf as-is:
core_collector makedumpfile -l --message-level 1 -d 31
And now I am suggested to dump without makedumpfile by using:
core_collector cp
I get big vmcore file of 3.5G, but crash 7.2.0++ can deal with it
without any error, whether with or without "nokaslr" kernel parameter.
So it seems makedumpfile has problem, and if it does is, I am curious
about that there is no other people report this kind of issue, because
it seems easy to reproduce.
--
Sincerely,
Cao jin
On 11/23/2017 10:03 PM, Dave Anderson wrote:
>> Hi Dave,
>> Â Because I didn't subscribe the list, so please CC me when reply:) My
>> colleague help to forward your reply to me, so I decide to reply
>> directly in order to keep this mail in the thread other than new a thread.
>>
>> On 11/23/2017 02:04 PM, Fei, Jie/è´¹ æ° wrote:
>>>
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject: Re: [Crash-utility] crash: page excluded: kernel virtual
>>> address: ffffffff81e3da50 type: "page_offset_base"
>>> Date: Wed, 22 Nov 2017 09:35:40 -0500
>>> From: Dave Anderson <anderson at redhat.com>
>>> Reply-To: Discussion list for crash utility usage, maintenance and
>>> development <crash-utility at redhat.com>
>>> To: Discussion list for crash utility usage, maintenance and
>>> development <crash-utility at redhat.com>
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> Hi,
>>>>
>>>> I am using the latest crash tool & kernel 4.14 compiled from source,
>>>> and
>>>> I got the following error message. As I searched, this is fixed in
>>>> crash
>>>> 7.2.0, but I still have it here. So, is anyone has a clue?
>>>
>>> It's always going to be a crap-shoot with very recent upstream kernels,
>>> but you haven't given enough information to determine what the issue is.
>>>
>>
>> Sorry, I am to new dump/crash area.
>>
>>> If the vmcore was created by "virsh dump", and the kernel has KASLR
>>> enabled,
>>> then it's just not supported at this time. Otherwise, perhaps the
>>> output of
>>> "crash -d8" may yield some clues.
>>>
>>
>> I am using kdump to create the vmcore. Actually, I did solve this issue
>> by add "nokaslr" to kernel parameter when I used the built-in crash of
>> Fedora. But when turn to the compiled version of my own, that solution
>> don't work. here is the output of `crash -d8`:
>>
>> $ sudo ./crash -d8 /var/crash/127.0.0.1-2017-11-22-18\:21\:42/vmcore
>> ../linux/vmlinux
>>
>> crash 7.2.0++
>> Copyright (C) 2002-2017 Red Hat, Inc.
>> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
>> Copyright (C) 1999-2006 Hewlett-Packard Co
>> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
>> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
>> Copyright (C) 2005, 2011 NEC Corporation
>> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
>> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
>> This program is free software, covered by the GNU General Public License,
>> and you are welcome to change it and/or distribute copies of it under
>> certain conditions. Enter "help copying" to see the conditions.
>> This program has absolutely no warranty. Enter "help warranty" for
>> details.
>>
>> compressed kdump: header->utsname.machine: x86_64
>> compressed kdump: memory bitmap offset: 2000
>> diskdump_data:
>> filename: /var/crash/127.0.0.1-2017-11-22-18:21:42/vmcore
>> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
>> dfd: 3
>> ofp: 0
>> machine_type: 62 (EM_X86_64)
>>
>> header: 1013930
>> signature: "KDUMP "
>> header_version: 6
>> utsname:
>> sysname: Linux
>> nodename: IAAS1
>> release: 4.14.0
>> version: #1 SMP Wed Nov 15 10:32:46 CST 2017
>> machine: x86_64
>> domainname: (none)
>> timestamp:
>> tv_sec: 5a154fac
>> tv_usec: 0
>> status: 2 (DUMP_DH_COMPRESSED_LZO)
>> block_size: 4096
>> sub_hdr_size: 1
>> bitmap_blocks: 80
>> max_mapnr: 1310208
>> total_ram_blocks: 0
>> device_blocks: 0
>> written_blocks: 0
>> current_cpu: 0
>> nr_cpus: 4
>> tasks[nr_cpus]: 0
>> 0
>> 0
>> 0
>>
>> sub_header: 0 (n/a)
>>
>> sub_header_kdump: 1014940
>> phys_base: 0
>> dump_level: 31 (0x1f)
>> (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
>> split: 0
>> start_pfn: (unused)
>> end_pfn: (unused)
>> offset_vmcoreinfo: 5648 (0x1610)
>> size_vmcoreinfo: 1883 (0x75b)
>> OSRELEASE=4.14.0
>> PAGESIZE=4096
>> SYMBOL(init_uts_ns)=ffffffff81e10280
>> SYMBOL(node_online_map)=ffffffff82030e80
>> SYMBOL(swapper_pg_dir)=ffffffff81e09000
>> SYMBOL(_stext)=ffffffff81000000
>> SYMBOL(vmap_area_list)=ffffffff81efc470
>> SYMBOL(mem_section)=ffffffff82401dc0
>> LENGTH(mem_section)=2048
>> SIZE(mem_section)=16
>> OFFSET(mem_section.section_mem_map)=0
>> SIZE(page)=64
>> SIZE(pglist_data)=172864
>> SIZE(zone)=1664
>> SIZE(free_area)=104
>> SIZE(list_head)=16
>> SIZE(nodemask_t)=128
>> OFFSET(page.flags)=0
>> OFFSET(page._refcount)=28
>> OFFSET(page.mapping)=8
>> OFFSET(page.lru)=32
>> OFFSET(page._mapcount)=24
>> OFFSET(page.private)=48
>> OFFSET(page.compound_dtor)=40
>> OFFSET(page.compound_order)=44
>> OFFSET(page.compound_head)=32
>> OFFSET(pglist_data.node_zones)=0
>> OFFSET(pglist_data.nr_zones)=172192
>> OFFSET(pglist_data.node_start_pfn)=172200
>> OFFSET(pglist_data.node_spanned_pages)=172216
>> OFFSET(pglist_data.node_id)=172224
>> OFFSET(zone.free_area)=192
>> OFFSET(zone.vm_stat)=1472
>> OFFSET(zone.spanned_pages)=112
>> OFFSET(free_area.free_list)=0
>> OFFSET(list_head.next)=0
>> OFFSET(list_head.prev)=8
>> OFFSET(vmap_area.va_start)=0
>> OFFSET(vmap_area.list)=48
>> LENGTH(zone.free_area)=11
>> SYMBOL(log_buf)=ffffffff81e58480
>> SYMBOL(log_buf_len)=ffffffff81e5847c
>> SYMBOL(log_first_idx)=ffffffff823340f8
>> SYMBOL(clear_idx)=ffffffff823340cc
>> SYMBOL(log_next_idx)=ffffffff823340e8
>> SIZE(printk_log)=16
>> OFFSET(printk_log.ts_nsec)=0
>> OFFSET(printk_log.len)=8
>> OFFSET(printk_log.text_len)=10
>> OFFSET(printk_log.dict_len)=12
>> LENGTH(free_area.free_list)=6
>> NUMBER(NR_FREE_PAGES)=0
>> NUMBER(PG_lru)=5
>> NUMBER(PG_private)=12
>> NUMBER(PG_swapcache)=9
>> NUMBER(PG_slab)=8
>> NUMBER(PG_hwpoison)=22
>> NUMBER(PG_head_mask)=32768
>> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-128
>> NUMBER(HUGETLB_PAGE_DTOR)=2
>> NUMBER(phys_base)=0
>> SYMBOL(init_top_pgt)=ffffffff81e09000
>> SYMBOL(node_data)=ffffffff8202c6c0
>> LENGTH(node_data)=1024
>> KERNELOFFSET=0
>> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
>> CRASHTIME=1511346092
>> offset_note: 4200 (0x1068)
>> size_note: 3332 (0xd04)
>> notes_buf: 1015950
>> num_prstatus_notes: 4
>> notes[0]: 1015950 (NT_PRSTATUS)
>> si.signo: 0 si.code: 0 si.errno: 0
>> cursig: 0 sigpend: 0 sighold: 0
>> pid: 114 ppid: 0 pgrp: 0 sid:0
>> utime: 0.000000 stime: 0.000000
>> cutime: 0.000000 cstime: 0.000000
>> ORIG_RAX: ffffffffffffffff fpvalid: 0
>> R15: ffffffff82445b80 R14: ffffffff822f3422
>> R13: 0000000000000020 R12: 00000000000026f5
>> RBP: ffffc90000adbce0 RBX: ffffffff82445b80
>> R11: ffffffff822f342d R10: 0000000000000000
>> R9: 000000000000000f R8: 0000000000000000
>> RAX: 000000799065724b RCX: 0000007990656c1b
>> RDX: 0000000000000000 RSI: 0000000000000000
>> RDI: 0000000000000cdd RIP: ffffffff8188600f
>> RFLAGS: 0000000000000097 RSP: ffffc90000adbce0
>> FS_BASE: 0000000000000000
>> GS_BASE: 0000000000000000
>> CS: 0010 SS: 0018 DS: 0000
>> ES: 0000 FS: 0000 GS: 0000
>> notes[1]: 1015ab4 (NT_PRSTATUS)
>> si.signo: 0 si.code: 0 si.errno: 0
>> cursig: 0 sigpend: 0 sighold: 0
>> pid: 2030 ppid: 0 pgrp: 0 sid:0
>> utime: 0.000000 stime: 0.000000
>> cutime: 0.000000 cstime: 0.000000
>> ORIG_RAX: ffffffffffffffff fpvalid: 0
>> R15: ffff880131999500 R14: ffffffff81f76fa0
>> R13: 0000000000000000 R12: 0000000000000007
>> RBP: ffffc90001dabdd8 RBX: 0000000000000063
>> R11: ffffffff822f342d R10: 0000000000000001
>> R9: 0000000000000007 R8: 00000000000002d9
>> RAX: 000000000000000f RCX: 0000000000000000
>> RDX: 0000000000000000 RSI: ffff88013fa8e138
>> RDI: 0000000000000063 RIP: ffffffff81532ad6
>> RFLAGS: 0000000000010282 RSP: ffffc90001dabdd8
>> FS_BASE: 00007fe58251fb80
>> GS_BASE: 0000000000000000
>> CS: 0010 SS: 0018 DS: 0000
>> ES: 0000 FS: 0000 GS: 0000
>> notes[2]: 1015c18 (NT_PRSTATUS)
>> si.signo: 0 si.code: 0 si.errno: 0
>> cursig: 0 sigpend: 0 sighold: 0
>> pid: 0 ppid: 0 pgrp: 0 sid:0
>> utime: 0.000000 stime: 0.000000
>> cutime: 0.000000 cstime: 0.000000
>> ORIG_RAX: ffffffffffffffff fpvalid: 0
>> R15: ffffffff81f65618 R14: 0000000000000020
>> R13: 0000000000000004 R12: 0000000000000003
>> RBP: ffffc900006a3e60 RBX: 0000000000000008
>> R11: 000000000000298b R10: ffffc900006a3e40
>> R9: 0000000000000018 R8: 00000000000054a2
>> RAX: 0000000000000020 RCX: 0000000000000001
>> RDX: 0000000000000000 RSI: ffffffff81f65480
>> RDI: 0000000000000002 RIP: ffffffff81895ddc
>> RFLAGS: 0000000000000046 RSP: ffffc900006a3e40
>> FS_BASE: 0000000000000000
>> GS_BASE: 0000000000000000
>> CS: 0010 SS: 0018 DS: 0000
>> ES: 0000 FS: 0000 GS: 0000
>> notes[3]: 1015d7c (NT_PRSTATUS)
>> si.signo: 0 si.code: 0 si.errno: 0
>> cursig: 0 sigpend: 0 sighold: 0
>> pid: 0 ppid: 0 pgrp: 0 sid:0
>> utime: 0.000000 stime: 0.000000
>> cutime: 0.000000 cstime: 0.000000
>> ORIG_RAX: ffffffffffffffff fpvalid: 0
>> R15: ffffffff81f65618 R14: 0000000000000020
>> R13: 0000000000000004 R12: 0000000000000003
>> RBP: ffffc900006abe60 RBX: 0000000000000008
>> R11: 000000000000037e R10: ffffc900006abe40
>> R9: 0000000000000008 R8: 00000000ffffffff
>> RAX: 0000000000000020 RCX: 0000000000000001
>> RDX: 0000000000000000 RSI: ffffffff81f65480
>> RDI: 0000000000000003 RIP: ffffffff81895ddc
>> RFLAGS: 0000000000000046 RSP: ffffc900006abe40
>> FS_BASE: 0000000000000000
>> GS_BASE: 0000000000000000
>> CS: 0010 SS: 0018 DS: 0000
>> ES: 0000 FS: 0000 GS: 0000
>> snapshot_task: 0
>> num_qemu_notes: 0
>> NOTE offsets: 1068 (NT_PRSTATUS)
>> 11cc (NT_PRSTATUS)
>> 1330 (NT_PRSTATUS)
>> 1494 (NT_PRSTATUS)
>> offset_eraseinfo: 0 (0x0)
>> size_eraseinfo: 0 (0x0)
>> start_pfn_64: (unused)
>> end_pfn_64: (unused)
>> max_mapnr_64: 1310208 (0x13fe00)
>>
>> data_offset: 52000
>> block_size: 4096
>> block_shift: 12
>> bitmap: 7f8ff466c010
>> bitmap_len: 327680
>> max_mapnr: 1310208 (0x13fe00)
>> dumpable_bitmap: 7f8ff461b010
>> byte: 0
>> bit: 0
>> compressed_page: 10470a0
>> curbufptr: 0
>>
>> page_cache_hdr[0]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1037090
>> pg_hit_count: 0
>> page_cache_hdr[1]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1038090
>> pg_hit_count: 0
>> page_cache_hdr[2]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1039090
>> pg_hit_count: 0
>> page_cache_hdr[3]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 103a090
>> pg_hit_count: 0
>> page_cache_hdr[4]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 103b090
>> pg_hit_count: 0
>> page_cache_hdr[5]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 103c090
>> pg_hit_count: 0
>> page_cache_hdr[6]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 103d090
>> pg_hit_count: 0
>> page_cache_hdr[7]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 103e090
>> pg_hit_count: 0
>> page_cache_hdr[8]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 103f090
>> pg_hit_count: 0
>> page_cache_hdr[9]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1040090
>> pg_hit_count: 0
>> page_cache_hdr[10]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1041090
>> pg_hit_count: 0
>> page_cache_hdr[11]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1042090
>> pg_hit_count: 0
>> page_cache_hdr[12]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1043090
>> pg_hit_count: 0
>> page_cache_hdr[13]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1044090
>> pg_hit_count: 0
>> page_cache_hdr[14]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1045090
>> pg_hit_count: 0
>> page_cache_hdr[15]:
>> pg_flags: 0 ()
>> pg_addr: 0
>> pg_bufptr: 1046090
>> pg_hit_count: 0
>>
>> page_cache_buf: 1037090
>> evict_index: 0
>> evictions: 0
>> accesses: 0
>> cached_reads: 0
>> valid_pages: 1036680
>> readmem: read_diskdump()
>> crash: pv_init_ops exists: ARCH_PVOPS
>> VMCOREINFO: NUMBER(phys_base): 0 -> 0
>> gdb ../linux/vmlinux
>> GNU gdb (GDB) 7.6
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later
>> <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-unknown-linux-gnu"...
>> GETBUF(288 -> 0)
>> GETBUF(1500 -> 1)
>>
>> FREEBUF(1)
>> FREEBUF(0)
>> <readmem: ffffffff81e3da50, KVADDR, "page_offset_base", 8, (FOE), d83568>
>> <read_diskdump: addr: ffffffff81e3da50 paddr: 1e3da50 cnt: 8>
>> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 1e3da50/1e3d
>> crash: page excluded: kernel virtual address: ffffffff81e3da50 type:
>> "page_offset_base"
>>
>> --
>> Sincerely,
>> Cao jin
>
> OK, so you seem to be describing two issues:
>
> (1) With KASLR enabled, the vmcore cannot be read by either
> Fedora's 7.2.0-1 or upstream 7.2.0++.
> (2) With KASLR disabled, the vmcore can be read by Fedora's
> 7.2.0-1, but cannot be read by 7.2.0++,
>
> When you run crash on the live system, what happens when:
>
> (1) KASLR is enabled, running Fedora's 7.2.0-1
> (2) KASLR is enabled, running upstream 7.2.0++
> (2) KASLR is disabled, running Fedora's 7.2.0-1
> (3) KASLR is disabled, running upstream 7.2.0++
>
> I don't see anything unusual in the output above, presuming the
> the phys_base of 0 is correct. When you run crash on the live
> system, what is the output of "help -m | grep phys_base"?
>
> I need to compare the crash -d8 output above with the same
> crash -d8 output from the working session using the same
> KASLR disabled vmcore, showing at least up until the point that
> it does the readmem of "page_offset_base" above?
>
> Also, if you can make the 2 vmcores available, please send me
> the details of how to download them in a private email to
> anderson at redhat.com or anderson at prospeed.net. If you have
> modified/rebuilt the Fedora kernel, I would also need a copy
> of the vmlinux file as well.
>
> We are on Thanksgiving holiday until next Monday, so I won't
> be able to spend any time looking at this issue until then.
>
> Dave
>
>
>
>
> .
>
More information about the Crash-utility
mailing list