[Crash-utility] EXT: Re: crash start in CentOS 8
Bhupesh Sharma
bhsharma at redhat.com
Mon Jun 15 08:40:33 UTC 2020
Hi Patrick,
On Mon, Jun 15, 2020 at 12:51 PM Agrain Patrick
<patrick.agrain at al-enterprise.com> wrote:
> -----Message d'origine-----
> De : crash-utility-bounces at redhat.com [mailto:crash-utility-bounces at redhat.com] De la part de Bhupesh Sharma
> Envoyé : lundi 15 juin 2020 09:13
> À : Discussion list for crash utility usage, maintenance and development <crash-utility at redhat.com>
> Objet : EXT: Re: [Crash-utility] crash start in CentOS 8
>
>
> ** External email - Please consider with caution **
>
>
> Hello Patrick,
>
> On Mon, Jun 15, 2020 at 12:28 PM Agrain Patrick <patrick.agrain at al-enterprise.com> wrote:
> >
> > Hello,
> >
> >
> >
> > I tried to test the crash utility from CentOS 8 with following requirements:
> >
> > First machine (VMware) on which CentOS 8 is installed and on which I performed a fake crash using SysRq. After reboot, The /var/crash was populated with the vmcore.
> > Second machine (Server) on which CentOS 8 is also installed and on which I wish to analyze the crash.
> >
> >
> >
> > Both machine have the same CentOS build installed.
> >
> > I imported System.map-4.18.0-147.el8.x86_64, vmcore and vmlinux.debuginfo from first machine to a directory on the second machine.
> >
> > When I run ‘crash’, I have following messages (see below), but no ‘crash>’ prompt.
> >
> >
> >
> > What have I missed ? What Should I check ?
> >
> >
> >
> > Thanks.
> >
> > Best regards,
> >
> > Patrick Agrain
> >
> >
> >
> > [root at lxkmg-pag-ale test64]# /usr/bin/crash
> > System.map-4.18.0-147.el8.x86_64 vmlinux.debuginfo vmcore
> >
> >
> >
> > crash 7.2.6-2.el8
> >
> > Copyright (C) 2002-2019 Red Hat, Inc.
> >
> > Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> >
> > Copyright (C) 1999-2006 Hewlett-Packard Co
> >
> > Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> >
> > Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> >
> > Copyright (C) 2005, 2011 NEC Corporation
> >
> > Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> >
> > Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> >
> > This program is free software, covered by the GNU General Public
> > License,
> >
> > and you are welcome to change it and/or distribute copies of it under
> >
> > certain conditions. Enter "help copying" to see the conditions.
> >
> > This program has absolutely no warranty. Enter "help warranty" for details.
> >
> >
> >
> > GNU gdb (GDB) 7.6
> >
> > Copyright (C) 2013 Free Software Foundation, Inc.
> >
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> >
> > This is free software: you are free to change and redistribute it.
> >
> > There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> >
> > and "show warranty" for details.
> >
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> >
> >
> >
> > please wait... (patching 92149 gdb minimal_symbol values) [Vanished
> > after a few seconds]
> >
> >
> >
> > [root at lxkmg-pag-ale test64]#
>
> Can you please increase the verbosity level by specifying the -d parameter while invoking the crash command?
> >From CRASH(8) man page:
>
> -d num: Set the internal debug level. The higher the number, the more debugging data will be printed when crash initializes and runs.
>
> I normally would run crash with -d 15 to get verbose logs (but not too verbose to overwhelm the console :)).
> Can you please share the output with the same?
>
> Thanks,
> Bhupesh
>
>
> Thanks Bhupesh for taking some time.
> Please find the logs with -d 15 verbosity.
>
> Thanks.
> Best regards,
> Patrick Agrain
>
> [root at lxkmg-pag-ale test64]# /usr/bin/crash -d 15 System.map-4.18.0-147.el8.x86_64 vmlinux.debuginfo vmcore
>
> crash 7.2.6-2.el8
> Copyright (C) 2002-2019 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for details.
>
> compressed kdump: header->utsname.machine: x86_64
> compressed kdump: memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 3
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: 2d006a0
> signature: "KDUMP "
> header_version: 6
> utsname:
> sysname: Linux
> nodename: localhost.localdomain
> release: 4.18.0-147.el8.x86_64
> version: #1 SMP Wed Dec 4 21:51:45 UTC 2019
> machine: x86_64
> domainname: (none)
> timestamp:
> tv_sec: 5ea68afe
> tv_usec: 0
> status: 2 (DUMP_DH_COMPRESSED_LZO)
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 32
> max_mapnr: 524288
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: 2d016b0
> phys_base: 35a00000
> dump_level: 31 (0x1f) (DUMP_EXCLUDE_ZERO|DUMP_EXCLUDE_CACHE|DUMP_EXCLUDE_CACHE_PRI|DUMP_EXCLUDE_USER_DATA|DUMP_EXCLUDE_FREE)
> split: 0
> start_pfn: (unused)
> end_pfn: (unused)
> offset_vmcoreinfo: 4580 (0x11e4)
> size_vmcoreinfo: 2024 (0x7e8)
> OSRELEASE=4.18.0-147.el8.x86_64
> PAGESIZE=4096
> SYMBOL(init_uts_ns)=ffffffff86812540
> SYMBOL(node_online_map)=ffffffff86a06d60
> SYMBOL(swapper_pg_dir)=ffffffff8680a000
> SYMBOL(_stext)=ffffffff85600000
> SYMBOL(vmap_area_list)=ffffffff868d2a90
> SYMBOL(mem_section)=ffff95a67ffd2000
> LENGTH(mem_section)=2048
> SIZE(mem_section)=16
> OFFSET(mem_section.section_mem_map)=0
> SIZE(page)=64
> SIZE(pglist_data)=171968
> SIZE(zone)=1472
> SIZE(free_area)=88
> SIZE(list_head)=16
> SIZE(nodemask_t)=128
> OFFSET(page.flags)=0
> OFFSET(page._refcount)=52
> OFFSET(page.mapping)=24
> OFFSET(page.lru)=8
> OFFSET(page._mapcount)=48
> OFFSET(page.private)=40
> OFFSET(page.compound_dtor)=16
> OFFSET(page.compound_order)=17
> OFFSET(page.compound_head)=8
> OFFSET(pglist_data.node_zones)=0
> OFFSET(pglist_data.nr_zones)=171232
> OFFSET(pglist_data.node_start_pfn)=171240
> OFFSET(pglist_data.node_spanned_pages)=171256
> OFFSET(pglist_data.node_id)=171264
> OFFSET(zone.free_area)=192
> OFFSET(zone.vm_stat)=1296
> OFFSET(zone.spanned_pages)=112
> OFFSET(free_area.free_list)=0
> OFFSET(list_head.next)=0
> OFFSET(list_head.prev)=8
> OFFSET(vmap_area.va_start)=0
> OFFSET(vmap_area.list)=48
> LENGTH(zone.free_area)=11
> SYMBOL(log_buf)=ffffffff8685a280
> SYMBOL(log_buf_len)=ffffffff8685a27c
> SYMBOL(log_first_idx)=ffffffff8713a770
> SYMBOL(clear_idx)=ffffffff8713a744
> SYMBOL(log_next_idx)=ffffffff8713a760
> SIZE(printk_log)=16
> OFFSET(printk_log.ts_nsec)=0
> OFFSET(printk_log.len)=8
> OFFSET(printk_log.text_len)=10
> OFFSET(printk_log.dict_len)=12
> LENGTH(free_area.free_list)=5
> NUMBER(NR_FREE_PAGES)=0
> NUMBER(PG_lru)=5
> NUMBER(PG_private)=12
> NUMBER(PG_swapcache)=9
> NUMBER(PG_swapbacked)=18
> NUMBER(PG_slab)=8
> NUMBER(PG_hwpoison)=22
> NUMBER(PG_head_mask)=32768
> NUMBER(PAGE_BUDDY_MAPCOUNT_VALUE)=-129
> NUMBER(HUGETLB_PAGE_DTOR)=2
> NUMBER(PAGE_OFFLINE_MAPCOUNT_VALUE)=-257
> NUMBER(phys_base)=899678208
> SYMBOL(init_top_pgt)=ffffffff8680a000
> NUMBER(pgtable_l5_enabled)=0
> SYMBOL(node_data)=ffffffff86a02580
> LENGTH(node_data)=1024
> KERNELOFFSET=4600000
> NUMBER(KERNEL_IMAGE_SIZE)=1073741824
> NUMBER(sme_mask)=0
> CRASHTIME=1587972862
> offset_note: 4200 (0x1068)
> size_note: 2404 (0x964)
> notes_buf: 2cff690
> num_vmcoredd_notes: 0
> num_prstatus_notes: 1
> notes[0]: 2cff690 (NT_PRSTATUS)
> si.signo: 0 si.code: 0 si.errno: 0
> cursig: 0 sigpend: 0 sighold: 0
> pid: 7020 ppid: 0 pgrp: 0 sid:0
> utime: 0.000000 stime: 0.000000
> cutime: 0.000000 cstime: 0.000000
> ORIG_RAX: ffffffffffffffff fpvalid: 0
> R15: 0000000000000000 R14: 000056500b98d6a0
> R13: ffffffff86936fc0 R12: 0000000000000000
> RBP: 0000000000000004 RBX: 0000000000000063
> R11: 6873617263206120 R10: 7265676769725420
> R9: 54203a2071527379 R8: 0000000000000556
> RAX: ffffffff85b12d10 RCX: 0000000000000006
> RDX: 0000000000000000 RSI: 0000000000000086
> RDI: 0000000000000063 RIP: ffffffff85b12d22
> RFLAGS: 0000000000010246 RSP: ffffbd9943093e80
> FS_BASE: 00007f2d9a1f6740
> GS_BASE: 0000000000000000
> CS: 0010 SS: 0018 DS: 0000
> ES: 0000 FS: 0000 GS: 0000
> snapshot_task: 0
> num_qemu_notes: 0
> NOTE offsets: 1068 (NT_PRSTATUS)
> offset_eraseinfo: 0 (0x0)
> size_eraseinfo: 0 (0x0)
> start_pfn_64: (unused)
> end_pfn_64: (unused)
> max_mapnr_64: 524288 (0x80000)
>
> data_offset: 22000
> block_size: 4096
> block_shift: 12
> bitmap: 2d05250
> bitmap_len: 131072
> max_mapnr: 524288 (0x80000)
> dumpable_bitmap: 7f454b94e010
> byte: 0
> bit: 0
> compressed_page: 2d026c0
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d55290
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d56290
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d57290
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d58290
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d59290
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d5a290
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d5b290
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d5c290
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d5d290
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d5e290
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d5f290
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d60290
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d61290
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d62290
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d63290
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: 2d64290
> pg_hit_count: 0
>
> page_cache_buf: 2d55290
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: 2d00210
> readmem: read_diskdump()
> crash: pv_init_ops exists: ARCH_PVOPS
> VMCOREINFO: NUMBER(phys_base): 899678208 -> 35a00000
> gdb vmlinux.debuginfo
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(328 -> 0)
> GETBUF(1500 -> 1)
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffff82131200, KVADDR, "page_offset_base", 8, (FOE|Q), d8b948>
> <read_diskdump: addr: ffffffff82131200 paddr: 37b31200 cnt: 8>
> read_diskdump: PAGE_EXCLUDED: paddr/pfn: 37b31200/37b31
> crash: page excluded: kernel virtual address: ffffffff82131200 type: "page_offset_base"
> [root at lxkmg-pag-ale test64]#
Hmmm, I think you should find better results with:
$ crash vmlinux vmcore (On the machine used for post-processing)
and
$ crash vmlinux /proc/kcore (On the original crashing system)
I think if that helps you reach the crash > prompt, we have a
potential issue which needs to be looked-at.
Can you please try to run the same and let me know the outcome?
Thanks,
Bhupesh
More information about the Crash-utility
mailing list