[Crash-utility] EXT: Re: crash start in CentOS 8
Agrain Patrick
patrick.agrain at al-enterprise.com
Mon Jul 6 13:03:14 UTC 2020
Hello all,
Back on this topic:
I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
I made a crash on this same VM.
I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal machine.
The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces following logs, but still failing to get the crash> prompt.
Still KASLR ?
Thanks in advance for any information.
Best regards,
Patrick Agrain
-- Logs:
crash 7.2.6-2.el8
Copyright (C) 2002-2019 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
compressed kdump: header->utsname.machine: x86_64
compressed kdump: memory bitmap offset: 2000
diskdump_data:
filename: vmcore
flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
dfd: 3
ofp: 0
machine_type: 62 (EM_X86_64)
header: 212b5d0
signature: "KDUMP "
header_version: 6
utsname:
sysname: Linux
nodename: localhost.localdomain
release: 4.18.0-147.el8.x86_64
version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
machine: x86_64
domainname: (none)
timestamp:
tv_sec: 5ee72d23
tv_usec: 0
status: 2 (DUMP_DH_COMPRESSED_LZO)
block_size: 4096
sub_hdr_size: 1
bitmap_blocks: 32
max_mapnr: 524288
total_ram_blocks: 0
device_blocks: 0
written_blocks: 0
current_cpu: 0
nr_cpus: 1
tasks[nr_cpus]: 0
sub_header: 0 (n/a)
sub_header_kdump: 212c5e0
phys_base: 7c00000
dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
split: 0
start_pfn: (unused)
end_pfn: (unused)
offset_vmcoreinfo: 4580 (0x11e4)
size_vmcoreinfo: 2025 (0x7e9)
OSRELEASE=4.18.0-147.el8.x86_64
PAGESIZE=4096
SYMBOL(init_uts_ns)=ffffffffb5a12540
SYMBOL(node_online_map)=ffffffffb5c06d60
SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
SYMBOL(_stext)=ffffffffb4800000
SYMBOL(vmap_area_list)=ffffffffb5ad2a90
SYMBOL(mem_section)=ffff89df7ffd2000
LENGTH(mem_section)=2048
SIZE(mem_section)=16
OFFSET(mem_section.section_mem_map)=0
SIZE(page)=64
SIZE(pglist_data)=171968
SIZE(zone)=1472
SIZE(free_area)=88
SIZE(list_head)=16
SIZE(nodemask_t)=128
OFFSET(page.flags)=0
OFFSET(page._refcount)=52
OFFSET(page.mapping)=24
OFFSET(page.lru)=8
OFFSET(page._mapcount)=48
OFFSET(page.private)=40
OFFSET(page.compound_dtor)=16
OFFSET(page.compound_order)=17
OFFSET(page.compound_head)=8
OFFSET(pglist_data.node_zones)=0
OFFSET(pglist_data.nr_zones)=171232
OFFSET(pglist_data.node_start_pfn)=171240
OFFSET(pglist_data.node_spanned_pages)=171256
OFFSET(pglist_data.node_id)=171264
OFFSET(zone.free_area)=192
OFFSET(zone.vm_stat)=1296
OFFSET(zone.spanned_pages)=112
OFFSET(free_area.free_list)=0
OFFSET(list_head.next)=0
OFFSET(list_head.prev)=8
OFFSET(vmap_area.va_start)=0
OFFSET(vmap_area.list)=48
LENGTH(zone.free_area)=11
SYMBOL(log_buf)=ffffffffb5a5a280
SYMBOL(log_buf_len)=ffffffffb5a5a27c
SYMBOL(log_first_idx)=ffffffffb633a770
SYMBOL(clear_idx)=ffffffffb633a744
SYMBOL(log_next_idx)=ffffffffb633a760
SIZE(printk_log)=16
OFFSET(printk_log.ts_nsec)=0
OFFSET(printk_log.len)=8
OFFSET(printk_log.text_len)=10
OFFSET(printk_log.dict_len)=12
LENGTH(free_area.free_list)=5
NUMBER(NR_FREE_PAGES)=0
NUMBER(PG_lru)=5
NUMBER(PG_private)=12
NUMBER(PG_swapcache)=9
NUMBER(PG_swapbacked)=18
NUMBER(PG_slab)=8
NUMBER(PG_hwpoison)=22
NUMBER(PG_head_mask)=32768
NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
NUMBER(HUGETLB_PAGE_DTOR)=2
NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
NUMBER(phys_base)=130023424
SYMBOL(init_top_pgt)=ffffffffb5a0a000
NUMBER(pgtable_l5_enabled)=0
SYMBOL(node_data)=ffffffffb5c02580
LENGTH(node_data)=1024
KERNELOFFSET=33800000
NUMBER(KERNEL_IMAGE_SIZE)=1073741824
NUMBER(sme_mask)=0
CRASHTIME=1592208675
offset_note: 4200 (0x1068)
size_note: 2408 (0x968)
notes_buf: 2129460
num_vmcoredd_notes: 0
num_prstatus_notes: 1
notes[0]: 2129460 (NT_PRSTATUS)
si.signo: 0 si.code: 0 si.errno: 0
cursig: 0 sigpend: 0 sighold: 0
pid: 2846 ppid: 0 pgrp: 0 sid:0
utime: 0.000000 stime: 0.000000
cutime: 0.000000 cstime: 0.000000
ORIG_RAX: ffffffffffffffff fpvalid: 0
R15: 0000000000000000 R14: 00005616bb1208f0
R13: ffffffffb5b36fc0 R12: 0000000000000000
RBP: 0000000000000004 RBX: 0000000000000063
R11: 6873617263206120 R10: 7265676769725420
R9: 54203a2071527379 R8: 000000000000056a
RAX: ffffffffb4d12d10 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000086
RDI: 0000000000000063 RIP: ffffffffb4d12d22
RFLAGS: 0000000000010246 RSP: ffffb03903147e80
FS_BASE: 00007f5d6e06f740
GS_BASE: 0000000000000000
CS: 0010 SS: 0018 DS: 0000
ES: 0000 FS: 0000 GS: 0000
snapshot_task: 0
num_qemu_notes: 0
NOTE offsets: 1068 (NT_PRSTATUS)
offset_eraseinfo: 0 (0x0)
size_eraseinfo: 0 (0x0)
start_pfn_64: (unused)
end_pfn_64: (unused)
max_mapnr_64: 524288 (0x80000)
data_offset: 22000
block_size: 4096
block_shift: 12
bitmap: 212f190
bitmap_len: 131072
max_mapnr: 524288 (0x80000)
dumpable_bitmap: 7f937ad93010
byte: 0
bit: 0
compressed_page: 212d5f0
curbufptr: 0
page_cache_hdr[0]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 217f1d0
pg_hit_count: 0
page_cache_hdr[1]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21801d0
pg_hit_count: 0
page_cache_hdr[2]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21811d0
pg_hit_count: 0
page_cache_hdr[3]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21821d0
pg_hit_count: 0
page_cache_hdr[4]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21831d0
pg_hit_count: 0
page_cache_hdr[5]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21841d0
pg_hit_count: 0
page_cache_hdr[6]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21851d0
pg_hit_count: 0
page_cache_hdr[7]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21861d0
pg_hit_count: 0
page_cache_hdr[8]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21871d0
pg_hit_count: 0
page_cache_hdr[9]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21881d0
pg_hit_count: 0
page_cache_hdr[10]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 21891d0
pg_hit_count: 0
page_cache_hdr[11]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218a1d0
pg_hit_count: 0
page_cache_hdr[12]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218b1d0
pg_hit_count: 0
page_cache_hdr[13]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218c1d0
pg_hit_count: 0
page_cache_hdr[14]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218d1d0
pg_hit_count: 0
page_cache_hdr[15]:
pg_flags: 0 ()
pg_addr: 0
pg_bufptr: 218e1d0
pg_hit_count: 0
page_cache_buf: 217f1d0
evict_index: 0
evictions: 0
accesses: 0
cached_reads: 0
valid_pages: 212a3b0
readmem: read_diskdump()
KASLR:
_stext from vmlinux.debuginfo: ffffffff81000000
_stext from vmcoreinfo: ffffffffb4800000
relocate: 33800000 (824MB)
crash: pv_init_ops exists: ARCH_PVOPS
VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000
gdb vmlinux.debuginfo
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
GETBUF(328 -> 0)
GETBUF(1500 -> 1)
WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol values
FREEBUF(1)
FREEBUF(0)
<readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q), d8b948>
<read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page: 3ecdf000
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(1024 -> 0)
<readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type: "possible"
WARNING: cannot read cpu_possible_map
<readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type: "present"
WARNING: cannot read cpu_present_map
<readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type: "online"
WARNING: cannot read cpu_online_map
<readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
<read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type: "active"
WARNING: cannot read cpu_active_map
FREEBUF(0)
<readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE), 7ffd3655d640>
<read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page: 3ef1a000
GETBUF(328 -> 0)
FREEBUF(0)
GETBUF(328 -> 0)
FREEBUF(0)
<readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE), 7ffd3655d5f0>
<read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type: "shadow_timekeeper xtime_sec"
xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
<readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE), d71b7c>
<read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390>
read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page: 3ee21000
utsname:
sysname:
nodename:
release:
version:
machine:
domainname:
base kernel version: 0.0.0
<readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q), 7ffd3655af30>
<read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
crash: page excluded: kernel virtual address: ffffffffb6a016c0 type: "accessible check"
crash: vmlinux.debuginfo and vmcore do not match!
Usage:
crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
crash [OPTION]... [NAMELIST] (live system form)
Enter "crash -h" for details.
-----Message d'origine-----
De : crash-utility-bounces at redhat.com [mailto:crash-utility-bounces at redhat.com] De la part de HAGIO KAZUHITO(?? ??)
Envoyé : mercredi 17 juin 2020 17:17
À : Discussion list for crash utility usage, maintenance and development <crash-utility at redhat.com>
Objet : EXT: Re: [Crash-utility] crash start in CentOS 8
** External email - Please consider with caution **
Hi Bhupesh,
-----Original Message-----
> > Just FYI, with respect to the System.map file, the following Dave's
> > post would be helpful:
> > https://www.redhat.com/archives/crash-utility/2018-June/msg00002.htm
> > l
> >
> > If you have the vmlinux corresponding to a running kernel, you don't
> > need the System.map file.
>
> Indeed, I remember discussing this with Dave A. some months back.
> Infact with newer KASLR enabled kernels it makes little sense to use
> System.map file as it contains the non-relocated symbol values that
> were compiled into the vmlinux file - that's why I suggested just to
> use to the vmlinux and vmcore file as an arguments while invoking the
> crash tool to Patrick.
>
> However, going back to the discussion I had with Dave A., I think the
> main point of confusion was the 'crash_whitepaper' which is still
> located at [1], and which provides the following example to invoke the
> crash utility with the System.map file, vmlinux and vmcore
> combination:
>
> Examples when running on a dumpfile:
>
> $ crash /boot/System.map vmlinux.dbg vmcore
>
> I think while this was ok for older kernels which did not support
> KASLR, for newer kernels this needs an update.
> I am trying to see if I can modify the 'crash_whitepaper' and push the
> same to github, so that users are not confused with the System.map
> option.
>
> What's your views on the same? Please let me know.
>
> [1]. https://crash-utility.github.io/crash_whitepaper.html
There are several command lines with System.map in "4. Invocation" chapter and I think that the ones in "RHEL2.1 Kernels (or kernels built without -g flag)"
section are no problem because the reason why it's needed is explained well.
The others in the "crash -h" output in "Invocation output" section might be a little confusing as you say if they see only the -S option's explanation..
I think it would be fine to omit the part of the output below the [mapfile] argument and itself because the output looks to be intended almost only to introduce the "crash -h" option. Then users see the current crash -h output which doesn't have command lines with System.map. So, for instance:
diff --git a/crash_whitepaper.html b/crash_whitepaper.html index 9b2e88c8acb4..7cb2608e6062 100644
--- a/crash_whitepaper.html
+++ b/crash_whitepaper.html
@@ -1166,7 +1166,8 @@ Usage:
file. If the [dumpfile] argument is not entered, the session will be
invoked on the live system using /dev/mem, which usually requires root
privileges.
-
+...
+<!-- Omit this part so that users don't use System.map needlessly.
[mapfile]
If the live system kernel, or the kernel from which the [dumpfile]
was derived, was not compiled with the -g switch, then the additional @@ -1215,7 +1216,7 @@ Usage:
[-d num]
Set crash debug level [num]. The higher the number, the more debug data
will be printed during crash runtime.
-
+-->
</pre>
</td>
</tr>
How about this?
Thanks,
Kazu
--
Crash-utility mailing list
Crash-utility at redhat.com
https://www.redhat.com/mailman/listinfo/crash-utility
More information about the Crash-utility
mailing list