[Crash-utility] crash for live system fails to load - Ubuntu 16.10
Dave Anderson
anderson at redhat.com
Mon Apr 17 13:18:14 UTC 2017
----- Original Message -----
> Hi,
>
> I've been using crash for a while..
> Strangely, on Ubuntu (and Lubuntu) 16.10, I face the issue described below:
>
> $ cat /etc/issue
> Ubuntu 16.10 \n \l
>
> $ uname -r
> 4.8.0-46-generic
> $ cat /proc/version
> Linux version 4.8.0-46-generic (buildd at lcy01-16) (gcc version 6.2.0
> 20161005 (Ubuntu 6.2.0-5ubuntu12) ) #49-Ubuntu SMP Fri Mar 31 13:57:14
> UTC 2017
> $ cat /proc/version_signature
> Ubuntu 4.8.0-46.49-generic 4.8.17
> $
>
> So I download the dbgsym package for this kernel as:
> $ wget
> http://ddebs.ubuntu.com/pool/main/l/linux/linux-image-4.8.0-46-generic-dbgsym_4.8.0-46.49_amd64.ddeb
> ...
> Then extract the vmlinux image from the package.
>
> $ ls -l /usr/lib/debug/boot/vmlinux-4.8.0-46-generic
> -rw-r--r-- 1 root root 509M Mar 31 12:43
> /usr/lib/debug/boot/vmlinux-4.8.0-46-generic
> $
>
> Run crash for 'live system':
>
> $ sudo crash /usr/lib/debug/boot/vmlinux-4.8.0-46-generic /proc/kcore
> /boot/System.map-4.8.0-46-generic
> [sudo] password for kaiwan:
>
> crash 7.1.5
> Copyright (C) 2002-2016 Red Hat, Inc.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> --snip--
>
> crash: seek error: kernel virtual address: ffffffff81f5dcc0 type: "possible"
> WARNING: cannot read cpu_possible_map
> crash: seek error: kernel virtual address: ffffffff81f5dc40 type: "present"
> WARNING: cannot read cpu_present_map
> crash: seek error: kernel virtual address: ffffffff81f5dc80 type: "online"
> WARNING: cannot read cpu_online_map
> crash: seek error: kernel virtual address: ffffffff81f5dc00 type: "active"
> WARNING: cannot read cpu_active_map
> crash: seek error: kernel virtual address: ffffffff81e2a070 type:
> "pv_init_ops"
> crash: seek error: kernel virtual address: ffffffff82167130 type:
> "shadow_timekeeper xtime_sec"
> crash: seek error: kernel virtual address: ffffffff81e0d304 type:
> "init_uts_ns"
> crash: /boot/System.map-4.8.0-46-generic and /proc/kcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
> $
>
> It _does_ load up with the --minimal switch, but of course I'd like to
> run it with full capabilities.
>
>
> FYI, output with debug ON:
>
> $ sudo crash -d5 /usr/lib/debug/boot/vmlinux-4.8.0-46-generic
> /proc/kcore /boot/System.map-4.8.0-46-generic
>
> crash 7.1.5
> Copyright (C) 2002-2016 Red Hat, Inc.
> --snip--
>
> get_live_memory_source: /proc/kcore
> proc_kcore_data:
> flags: 0 (KCORE_LOCAL|KCORE_ELF64)
> segments: 17
> elf_header: 55f59ded06e0
> elf64: 55f59ded06e0
> load64: 55f59ded0758
> elf32: 0
> load32: 0
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 7fffff602000
> p_vaddr: ffffffffff600000
> p_paddr: 0
> p_filesz: 1000
> p_memsz: 1000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 7fff9da02000
> p_vaddr: ffffffff9da00000
> p_paddr: 0
> p_filesz: 1268000
> p_memsz: 1268000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 2922c0002000
> p_vaddr: ffffa922c0000000
> p_paddr: 0
> p_filesz: 1fffffffffff
> p_memsz: 1fffffffffff
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 7fffc0002000
> p_vaddr: ffffffffc0000000
> p_paddr: 0
> p_filesz: 3f000000
> p_memsz: 3f000000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a100003000
> p_vaddr: ffff96a100001000
> p_paddr: 0
> p_filesz: 57000
> p_memsz: 57000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 6a0000002000
> p_vaddr: ffffea0000000000
> p_paddr: 0
> p_filesz: 2000
> p_memsz: 2000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a10005b000
> p_vaddr: ffff96a100059000
> p_paddr: 0
> p_filesz: 33000
> p_memsz: 33000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a100102000
> p_vaddr: ffff96a100100000
> p_paddr: 0
> p_filesz: 77b5f000
> p_memsz: 77b5f000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 6a0000006000
> p_vaddr: ffffea0000004000
> p_paddr: 0
> p_filesz: 1dee000
> p_memsz: 1dee000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a177c63000
> p_vaddr: ffff96a177c61000
> p_paddr: 0
> p_filesz: 2fb000
> p_memsz: 2fb000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a177f5f000
> p_vaddr: ffff96a177f5d000
> p_paddr: 0
> p_filesz: 34af4000
> p_memsz: 34af4000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 6a0001dff000
> p_vaddr: ffffea0001dfd000
> p_paddr: 0
> p_filesz: d2d000
> p_memsz: d2d000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a1aca7e000
> p_vaddr: ffff96a1aca7c000
> p_paddr: 0
> p_filesz: 9903000
> p_memsz: 9903000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a1b8001000
> p_vaddr: ffff96a1b7fff000
> p_paddr: 0
> p_filesz: 1000
> p_memsz: 1000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 6a0002e01000
> p_vaddr: ffffea0002dff000
> p_paddr: 0
> p_filesz: 1000
> p_memsz: 1000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 16a200002000
> p_vaddr: ffff96a200000000
> p_paddr: 0
> p_filesz: 142800000
> p_memsz: 142800000
> p_align: 4096
>
> Elf64_Phdr:
> p_type: 1
> p_flags: 7
> p_offset: 6a0004002000
> p_vaddr: ffffea0004000000
> p_paddr: 0
> p_filesz: 50a0000
> p_memsz: 50a0000
> p_align: 4096
>
> readmem: read_proc_kcore() -> /proc/kcore
> crash: pv_init_ops exists: ARCH_PVOPS
> _text: ffffffff81000000 Kernel code: 1aaa00000 -> phys_base: 1a9a00000
>
> gdb /usr/lib/debug/boot/vmlinux-4.8.0-46-generic
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> <readmem: ffffffff81f5dcc0, KVADDR, "possible", 64, (ROE), 55f59c515060>
> <read_proc_kcore: addr: ffffffff81f5dcc0 paddr: 1ab95dcc0 cnt: 64>
> crash: seek error: kernel virtual address: ffffffff81f5dcc0 type: "possible"
> WARNING: cannot read cpu_possible_map
> <readmem: ffffffff81f5dc40, KVADDR, "present", 64, (ROE), 55f59c515060>
> <read_proc_kcore: addr: ffffffff81f5dc40 paddr: 1ab95dc40 cnt: 64>
> crash: seek error: kernel virtual address: ffffffff81f5dc40 type: "present"
> WARNING: cannot read cpu_present_map
> <readmem: ffffffff81f5dc80, KVADDR, "online", 64, (ROE), 55f59c515060>
> <read_proc_kcore: addr: ffffffff81f5dc80 paddr: 1ab95dc80 cnt: 64>
> crash: seek error: kernel virtual address: ffffffff81f5dc80 type: "online"
> WARNING: cannot read cpu_online_map
> <readmem: ffffffff81f5dc00, KVADDR, "active", 64, (ROE), 55f59c515060>
> <read_proc_kcore: addr: ffffffff81f5dc00 paddr: 1ab95dc00 cnt: 64>
> crash: seek error: kernel virtual address: ffffffff81f5dc00 type: "active"
> WARNING: cannot read cpu_active_map
> <readmem: ffffffff81e2a070, KVADDR, "pv_init_ops", 8, (ROE), 7fff3eafb4b0>
> <read_proc_kcore: addr: ffffffff81e2a070 paddr: 1ab82a070 cnt: 8>
> crash: seek error: kernel virtual address: ffffffff81e2a070 type:
> "pv_init_ops"
> <readmem: ffffffff82167130, KVADDR, "shadow_timekeeper xtime_sec", 8,
> (ROE), 7fff3eafb460>
> <read_proc_kcore: addr: ffffffff82167130 paddr: 1abb67130 cnt: 8>
> crash: seek error: kernel virtual address: ffffffff82167130 type:
> "shadow_timekeeper xtime_sec"
> xtime timespec.tv_sec: 8: Thu Jan 1 05:30:08 1970
> <readmem: ffffffff81e0d304, KVADDR, "init_uts_ns", 390, (ROE), 55f59c2b9e3c>
> <read_proc_kcore: addr: ffffffff81e0d304 paddr: 1ab80d304 cnt: 390>
> crash: seek error: kernel virtual address: ffffffff81e0d304 type:
> "init_uts_ns"
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> base kernel version: 0.0.0
> <readmem: ffffffff81a000c0, KVADDR, "accessible check", 8, (ROE|Q),
> 7fff3eaf87b0>
> <read_proc_kcore: addr: ffffffff81a000c0 paddr: 1ab4000c0 cnt: 8>
> crash: seek error: kernel virtual address: ffffffff81a000c0 type:
> "accessible check"
> crash: /boot/System.map-4.8.0-46-generic and /proc/kcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS] (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
> $
>
> The similar procedure worked just fine on [L]Ubuntu 15.10.
> Any help in solving this would be appreciated, TIA!
>
> -Kaiwan.
>
> Thanks & Regards,
> Kaiwan.
It's unlikely that crash version 7.1.5 will work with a 4.8-based kernel.
In fact, just the other day I received a report privately with the same situation,
where Ubuntu 16.10 wouldn't work with crash-7.1.5, and it was necessary
to upgrade the crash utility version.
Also, I don't understand why you are using System.map arguments? For
example, don't the symbol values in vmlinux-4.8.0-46-generic match
those in /boot/System.map-4.8.0-46-generic? If they match, then
you shouldn't need a System.map argument because it only serves to
restrict certains commands, such as those showing line numbers.
System.map arguments should only be used if the vmlinux file does
not exactly match the running kernel or dumpfile being analyzed.
Dave
More information about the Crash-utility
mailing list