[Crash-utility] EXT: Re: crash start in CentOS 8
Santosh
ysan99 at gmail.com
Mon Jul 6 21:53:55 UTC 2020
On Mon, Jul 6, 2020 at 6:03 AM Agrain Patrick
<patrick.agrain at al-enterprise.com> wrote:
>
> Hello all,
>
> Back on this topic:
> I compiled a kernel with debuginfo on the Virtual Machine with CentOS8.
> I made a crash on this same VM.
> I transferred the vmcore and vmlinux.debuginfo files on another CentOS8 baremetal machine.
>
> The command '/usr/bin/crash -d 15 vmlinux.debuginfo vmcore' produces following logs, but still failing to get the crash> prompt.
> Still KASLR ?
>
> Thanks in advance for any information.
> Best regards,
> Patrick Agrain
>
> -- Logs:
> crash 7.2.6-2.el8
You may try the latest release 7.2.8 or even better the tip repo
https://github.com/crash-utility/crash ?
Thanks,
Santosh
> Copyright (C) 2002-2019 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> compressed kdump: header->utsname.machine: x86_64
> compressed kdump: memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 212b5d0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: localhost.localdomain
> release: 4.18.0-147.el8.x86_64
> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5ee72d23
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 32
> max_mapnr: 524288
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 212c5e0
> phys_base: 7c00000
> dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 4580 (0x11e4)
> size_vmcoreinfo: 2025 (0x7e9)
> OSRELEASE=4.18.0-147.el8.x86_64
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffffb5a12540
> SYMBOL(node_online_map)=ffffffffb5c06d60
> SYMBOL(swapper_pg_dir)=ffffffffb5a0a000
> SYMBOL(_stext)=ffffffffb4800000
> SYMBOL(vmap_area_list)=ffffffffb5ad2a90
> SYMBOL(mem_section)=ffff89df7ffd2000
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=171968
> SIZE(zone)=1472
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=52
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=8
> OFFSET(page._mapcount)=48
> OFFSET(page.private)=40
> OFFSET(page.compound_dtor)=16
> OFFSET(page.compound_order)=17
> OFFSET(page.compound_head)=8
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=171232
> OFFSET(pglist_data.node_start_pfn)=171240
> OFFSET(pglist_data.node_spanned_pages)=171256
> OFFSET(pglist_data.node_id)=171264
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1296
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffffb5a5a280
> SYMBOL(log_buf_len)=ffffffffb5a5a27c
> SYMBOL(log_first_idx)=ffffffffb633a770
> SYMBOL(clear_idx)=ffffffffb633a744
> SYMBOL(log_next_idx)=ffffffffb633a760
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_swapbacked)=18
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> NUMBER(phys_base)=130023424
> SYMBOL(init_top_pgt)=ffffffffb5a0a000
> NUMBER(pgtable_l5_enabled)=0
> SYMBOL(node_data)=ffffffffb5c02580
> LENGTH(node_data)=1024
> KERNELOFFSET=33800000
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> NUMBER(sme_mask)=0
> CRASHTIME=1592208675
> offset_note: 4200 (0x1068)
> size_note: 2408 (0x968)
> notes_buf: 2129460
> num_vmcoredd_notes: 0
> num_prstatus_notes: 1
> notes[0]: 2129460 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 2846 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: 0000000000000000 R14: 00005616bb1208f0
> R13: ffffffffb5b36fc0 R12: 0000000000000000
> RBP: 0000000000000004 RBX: 0000000000000063
> R11: 6873617263206120 R10: 7265676769725420
> R9: 54203a2071527379 R8: 000000000000056a
> RAX: ffffffffb4d12d10 RCX: 0000000000000006
> RDX: 0000000000000000 RSI: 0000000000000086
> RDI: 0000000000000063 RIP: ffffffffb4d12d22
> RFLAGS: 0000000000010246 RSP: ffffb03903147e80
> FS_BASE: 00007f5d6e06f740
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 524288 (0x80000)
>
> data_offset: 22000
> block_size: 4096
> block_shift: 12
> bitmap: 212f190
> bitmap_len: 131072
> max_mapnr: 524288 (0x80000)
> dumpable_bitmap: 7f937ad93010
> byte: 0
> bit: 0
> compressed_page: 212d5f0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 217f1d0
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21801d0
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21811d0
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21821d0
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21831d0
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21841d0
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21851d0
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21861d0
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21871d0
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21881d0
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 21891d0
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218a1d0
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218b1d0
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218c1d0
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218d1d0
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 218e1d0
> pg_hit_count: 0
>
> page_cache_buf: 217f1d0
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 212a3b0
> readmem: read_diskdump()
> KASLR:
> _stext from vmlinux.debuginfo: ffffffff81000000
> _stext from vmcoreinfo: ffffffffb4800000
> relocate: 33800000 (824MB)
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 130023424 -> 7c00000
> gdb vmlinux.debuginfo
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(328 -> 0)
> GETBUF(1500 -> 1)
>
> WARNING: kernel relocated [824MB]: patching 109676 gdb minimal_symbol values
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffffb70df958, KVADDR, "page_offset_base", 8, (FOE|Q), d8b948>
> <read_diskdump: addr: ffffffffb70df958 paddr: 3ecdf958 cnt: 8>
> read_diskdump: paddr/pfn: 3ecdf958/3ecdf -> cache physical page: 3ecdf000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(1024 -> 0)
> <readmem: ffffffffb7f0c5e0, KVADDR, "possible", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c5e0 paddr: 3fb0c5e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c5e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c5e0 type: "possible"
> WARNING: cannot read cpu_possible_map
> <readmem: ffffffffb7f0bde0, KVADDR, "present", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0bde0 paddr: 3fb0bde0 cnt: 544>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0bde0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0bde0 type: "present"
> WARNING: cannot read cpu_present_map
> <readmem: ffffffffb7f0c1e0, KVADDR, "online", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0c1e0 paddr: 3fb0c1e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0c1e0/3fb0c
> crash: page excluded: kernel virtual address: ffffffffb7f0c1e0 type: "online"
> WARNING: cannot read cpu_online_map
> <readmem: ffffffffb7f0b9e0, KVADDR, "active", 1024, (ROE), fcd120>
> <read_diskdump: addr: ffffffffb7f0b9e0 paddr: 3fb0b9e0 cnt: 1024>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3fb0b9e0/3fb0b
> crash: page excluded: kernel virtual address: ffffffffb7f0b9e0 type: "active"
> WARNING: cannot read cpu_active_map
> FREEBUF(0)
> <readmem: ffffffffb731a5b0, KVADDR, "pv_init_ops", 8, (ROE), 7ffd3655d640>
> <read_diskdump: addr: ffffffffb731a5b0 paddr: 3ef1a5b0 cnt: 8>
> read_diskdump: paddr/pfn: 3ef1a5b0/3ef1a -> cache physical page: 3ef1a000
> GETBUF(328 -> 0)
> FREEBUF(0)
> GETBUF(328 -> 0)
> FREEBUF(0)
> <readmem: ffffffffb9bad1d0, KVADDR, "shadow_timekeeper xtime_sec", 8, (ROE), 7ffd3655d5f0>
> <read_diskdump: addr: ffffffffb9bad1d0 paddr: 417ad1d0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 417ad1d0/417ad
> crash: page excluded: kernel virtual address: ffffffffb9bad1d0 type: "shadow_timekeeper xtime_sec"
> xtime timespec.tv_sec: 900f27: Mon Apr 20 07:31:03 1970
> <readmem: ffffffffb7221d64, KVADDR, "init_uts_ns", 390, (ROE), d71b7c>
> <read_diskdump: addr: ffffffffb7221d64 paddr: 3ee21d64 cnt: 390>
> read_diskdump: paddr/pfn: 3ee21d64/3ee21 -> cache physical page: 3ee21000
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffffffffb6a016c0, KVADDR, "accessible check", 8, (ROE|Q), 7ffd3655af30>
> <read_diskdump: addr: ffffffffb6a016c0 paddr: 3e6016c0 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 3e6016c0/3e601
> crash: page excluded: kernel virtual address: ffffffffb6a016c0 type: "accessible check"
> crash: vmlinux.debuginfo and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
> -----Message d'origine-----
> De : crash-utility-bounces at redhat.com [mailto:crash-utility-bounces at redhat.com] De la part de HAGIO KAZUHITO(?? ??)
> Envoyé : mercredi 17 juin 2020 17:17
> À : Discussion list for crash utility usage, maintenance and development <crash-utility at redhat.com>
> Objet : EXT: Re: [Crash-utility] crash start in CentOS 8
>
>
> ** External email - Please consider with caution **
>
>
> Hi Bhupesh,
>
> -----Original Message-----
> > > Just FYI, with respect to the System.map file, the following Dave's
> > > post would be helpful:
> > > https://www.redhat.com/archives/crash-utility/2018-June/msg00002.htm
> > > l
> > >
> > > If you have the vmlinux corresponding to a running kernel, you don't
> > > need the System.map file.
> >
> > Indeed, I remember discussing this with Dave A. some months back.
> > Infact with newer KASLR enabled kernels it makes little sense to use
> > System.map file as it contains the non-relocated symbol values that
> > were compiled into the vmlinux file - that's why I suggested just to
> > use to the vmlinux and vmcore file as an arguments while invoking the
> > crash tool to Patrick.
> >
> > However, going back to the discussion I had with Dave A., I think the
> > main point of confusion was the 'crash_whitepaper' which is still
> > located at [1], and which provides the following example to invoke the
> > crash utility with the System.map file, vmlinux and vmcore
> > combination:
> >
> > Examples when running on a dumpfile:
> >
> > $ crash /boot/System.map vmlinux.dbg vmcore
> >
> > I think while this was ok for older kernels which did not support
> > KASLR, for newer kernels this needs an update.
> > I am trying to see if I can modify the 'crash_whitepaper' and push the
> > same to github, so that users are not confused with the System.map
> > option.
> >
> > What's your views on the same? Please let me know.
> >
> > [1]. https://crash-utility.github.io/crash_whitepaper.html
>
> There are several command lines with System.map in "4. Invocation" chapter and I think that the ones in "RHEL2.1 Kernels (or kernels built without -g flag)"
> section are no problem because the reason why it's needed is explained well.
>
> The others in the "crash -h" output in "Invocation output" section might be a little confusing as you say if they see only the -S option's explanation..
>
> I think it would be fine to omit the part of the output below the [mapfile] argument and itself because the output looks to be intended almost only to introduce the "crash -h" option. Then users see the current crash -h output which doesn't have command lines with System.map. So, for instance:
>
> diff --git a/crash_whitepaper.html b/crash_whitepaper.html index 9b2e88c8acb4..7cb2608e6062 100644
> --- a/crash_whitepaper.html
> +++ b/crash_whitepaper.html
> @@ -1166,7 +1166,8 @@ Usage:
> file. If the [dumpfile] argument is not entered, the session will be
> invoked on the live system using /dev/mem, which usually requires root
> privileges.
> -
> +...
> +<!-- Omit this part so that users don't use System.map needlessly.
> [mapfile]
> If the live system kernel, or the kernel from which the [dumpfile]
> was derived, was not compiled with the -g switch, then the additional @@ -1215,7 +1216,7 @@ Usage:
> [-d num]
> Set crash debug level [num]. The higher the number, the more debug data
> will be printed during crash runtime.
> -
> +-->
> </pre>
> </td>
> </tr>
>
> How about this?
>
> Thanks,
> Kazu
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
More information about the Crash-utility
mailing list