[Crash-utility] crash can live analysis but vmcore can't.
Dave Anderson
anderson at redhat.com
Mon Mar 26 14:06:49 UTC 2012
Whenever a crash session fails with a "do not match" message
such as this:
crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
it is because the virtual addresses of the kernel symbols in the
vmlinux file do not match the virtual addresses of the kernel that
crashed.
When you run with "crash -d<number>" you will see the first few
attempts to gather kernel data (kernel_config_data, xtime, the
cpu_possible, cpu_present and cpu_online maps, the utsname structure,
and then the "linux_banner" string. The linux_banner string is what
you would see in /proc/version on the live system, for example:
# cat /proc/version
Linux version 2.6.32-220.el6.x86_64 (mockbuild at x86-004.build.bos.redhat.com) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Wed Nov 9 08:03:13 EST 2011
#
If the crash utility does not find the correct string at the address
specified in the vmlinux file, then it makes no sense to continue,
and so it quits with the "do not match" message.
Accordingly, one of your examples ends like this, where the
vmlinux file indicates that the linux_banner string is located
at kernel virtual address ffffffff804a7000. That address was
translated to physical address 4a7000, and then read from the
compressed kdump vmcore. But the linux_banner string read from
that address in the vmcore contains garbage:
...
<readmem: ffffffff804a7000, KVADDR, "readstring characters", 1499, (ROE|Q), 7fff35eeaa70>
<read_diskdump: addr: ffffffff804a7000 paddr: 4a7000 cnt: 1499>
read_diskdump: paddr/pfn: 4a7000/4a7 -> physical page is cached: 4a7000
linux_banner:
�x;�����
crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
...
I cannot answer for SUSE, but if I am not mistaken, they build a
debug kernel (i.e, built with -g) for their distrobution kernels
after-the-fact, and when that is done, the kernel virtual addresses
are not exactly the same. To work around that, you can pass the
System.map file of the actual crashed kernel -- which has the *correct"
kernel virtual addresses -- on the crash command line. And when
you did that, things apparently worked correctly.
In other words, this failed:
# crash vmlinux-2.6.27.19-5-default vmlinux-2.6.27.19-5-default.debug vmcore
But this worked:
# crash vmlinux-2.6.27.19-5-default System.map-2.6.27.19-5-default vmcore
And it worked because the crash utility takes the (correct) symbol values
found in the System.map file, and "patches" the symbol values that were
initially read from the vmlinux file(s).
Complicating matters a bit is the usage of a vmlinux file and its associated
vmlinux.debug file. In old RHEL3 kernels, Red Hat would split the vmlinux
file into its stripped version and a .debug version, the same way as is
done with user-space programs. But from RHEL4 onwards, the that was no longer
done, and there is a single vmlinux file that contains all the debuginfo
data, and which is the same kernel as the distribution kernel. In SLES kernels,
I am not sure whether that was true in the example above, or whether the .debug
kernel is the kernel that they build after-the-fact?
In any case, I note that there a several novell.com addresses that are cc'd
with this email, and there are many SLES users and Novell/Suse employees
on this list who can give you the specifics of how they do things.
Dave
----- Original Message -----
>
>
> on another machine,I use all kinds of crash command :
>
>
> Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #crash
> vmlinux-2.6.27.19-5-default.gz System.map-2.6.27.19-5-default vmcore
>
> crash 6.0.4
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> crash: /var/tmp/vmlinux-2.6.27.19-5-default.gz_RYx9JC: no debugging
> data available
>
> Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #gunzip vmlinux-2.
> 6.27.19-5-default.gz
>
> Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 # crash
> vmlinux-2.6.27.19-5-default vmlinux-2.6.27.19-5-default.debug vmcore
>
> crash 6.0.4
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> WARNING: could not find MAGIC_START!
> crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
> Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #cat README.txt
> Kernel crashdump
> ----------------
>
> Crash time : 2012-03-24 10:57 (+0000)
> Kernel version : 2.6.27.19-5-default
> Host : sles11
> Dump level : 0
> Dump format : compressed
>
> but I use another command is OK, what's wrong ?
>
> Vsles11sp1 /mnt/disk/crash/2012-03-24-18:57 #crash
> vmlinux-2.6.27.19-5-default System.map-2.6.27.19-5-default vmcore
>
> crash 6.0.4
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> SYSTEM MAP: System.map-2.6.27.19-5-default
> DEBUG KERNEL: vmlinux-2.6.27.19-5-default (2.6.27.19-5-default)
> DUMPFILE: vmcore
> CPUS: 2
> DATE: Sat Mar 24 18:57:31 2012
> UPTIME: 00:13:30
> LOAD AVERAGE: 0.27, 0.30, 0.28
> TASKS: 190
> NODENAME: sles11
> RELEASE: 2.6.27.19-5-default
> VERSION: #6 SMP Sat Nov 12 08:58:08 CST 2011
> MACHINE: x86_64 (2393 Mhz)
> MEMORY: 3.9 GB
> PANIC: "SysRq : Trigger a crashdump"
> PID: 4700
> COMMAND: "bash"
> TASK: ffff88007f8f6080 [THREAD_INFO: ffff88007f802000]
> CPU: 0
> STATE: TASK_RUNNING (SYSRQ)
>
> crash>
>
>
>
>
>
>
>
> From: hujun_hotmail
> Sent: Saturday, March 24, 2012 8:07 PM
> To: crash mailist
> Subject: [Crash-utility] crash can live analysis but vmcore can't.
>
>
> hello experts:
>
> My linux system : Suse SLES 11 , crash can live analysis, but can't
> open vmcore generated by kdump.
>
> why?
>
>
>
> sles11:/mnt/sda12/crash/2012-03-24-18:57 # crash
>
>
> crash 6.0.4
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> KERNEL: /boot/vmlinuz-2.6.27.19-5-default.vmlinux-debug
> DUMPFILE: /dev/mem
> CPUS: 2
> DATE: Sat Mar 24 19:10:45 2012
> UPTIME: 00:07:17
> LOAD AVERAGE: 0.16, 0.19, 0.12
> TASKS: 189
> NODENAME: sles11
> RELEASE: 2.6.27.19-5-default
> VERSION: #6 SMP Sat Nov 12 08:58:08 CST 2011
> MACHINE: x86_64 (2393 Mhz)
> MEMORY: 3.9 GB
> PID: 4590
> COMMAND: "crash"
> TASK: ffff8801230fa340 [THREAD_INFO: ffff880131c86000]
> CPU: 0
> STATE: TASK_RUNNING (ACTIVE)
>
> crash>
>
>
>
>
>
>
> sles11:/mnt/sda12/crash/2012-03-24-18:57 # crash
> vmlinux-2.6.27.19-5-default.gz vmlinux-2.6.27.19-5-default.debug
> vmcore
> crash 6.0.4
> Copyright (C) 2002-2012 Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006 IBM Corporation
> Copyright (C) 1999-2006 Hewlett-Packard Co
> Copyright (C) 2005, 2006 Fujitsu Limited
> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> Copyright (C) 2005 NEC Corporation
> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public
> License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions. Enter "help copying" to see the conditions.
> This program has absolutely no warranty. Enter "help warranty" for
> details.
>
> please wait... (uncompressing vmlinux-2.6.27.19-5-default.gz)
>
> compressed kdump: header->utsname.machine:
> compressed kdump: memory bitmap offset: 2000
> diskdump_data:
> filename: vmcore
> flags: 6 (KDUMP_CMPRS_LOCAL|ERROR_EXCLUDED)
> dfd: 4
> ofp: 0
> machine_type: 62 (EM_X86_64)
>
> header: dee020
> signature: "KDUMP "
> header_version: 1
> utsname:
> sysname:
> nodename:
> release:
> version:
> machine:
> domainname:
> timestamp:
> tv_sec: 0
> tv_usec: 0
> status: 0 ()
> block_size: 4096
> sub_hdr_size: 1
> bitmap_blocks: 80
> max_mapnr: 1294336
> total_ram_blocks: 0
> device_blocks: 0
> written_blocks: 0
> current_cpu: 0
> nr_cpus: 1
> tasks[nr_cpus]: 0
>
> sub_header: 0 (n/a)
>
> sub_header_kdump: def030
> phys_base: 0
> dump_level: 0 (0x0)
>
> data_offset: 52000
> block_size: 4096
> block_shift: 12
> bitmap: 7f812de69010
> bitmap_len: 327680
> dumpable_bitmap: 7f812de18010
> byte: 0
> bit: 0
> compressed_page: e00a40
> curbufptr: 0
>
> page_cache_hdr[0]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df0a30
> pg_hit_count: 0
> page_cache_hdr[1]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df1a30
> pg_hit_count: 0
> page_cache_hdr[2]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df2a30
> pg_hit_count: 0
> page_cache_hdr[3]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df3a30
> pg_hit_count: 0
> page_cache_hdr[4]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df4a30
> pg_hit_count: 0
> page_cache_hdr[5]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df5a30
> pg_hit_count: 0
> page_cache_hdr[6]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df6a30
> pg_hit_count: 0
> page_cache_hdr[7]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df7a30
> pg_hit_count: 0
> page_cache_hdr[8]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df8a30
> pg_hit_count: 0
> page_cache_hdr[9]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: df9a30
> pg_hit_count: 0
> page_cache_hdr[10]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfaa30
> pg_hit_count: 0
> page_cache_hdr[11]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfba30
> pg_hit_count: 0
> page_cache_hdr[12]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfca30
> pg_hit_count: 0
> page_cache_hdr[13]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfda30
> pg_hit_count: 0
> page_cache_hdr[14]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dfea30
> pg_hit_count: 0
> page_cache_hdr[15]:
> pg_flags: 0 ()
> pg_addr: 0
> pg_bufptr: dffa30
> pg_hit_count: 0
>
> page_cache_buf: df0a30
> evict_index: 0
> evictions: 0
> accesses: 0
> cached_reads: 0
> valid_pages: df0040
> readmem: read_diskdump()
> [?1034hcompressed kdump: phys_base: 0
> gdb vmlinux-2.6.27.19-5-default.debug
> GNU gdb (GDB) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html >
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
> GETBUF(248 -> 0)
> GETBUF(1500 -> 1)
>
> please wait... (patching 35285 gdb minimal_symbol values)
>
> FREEBUF(1)
> FREEBUF(0)
> <readmem: ffffffff804b8210, KVADDR, "kernel_config_data", 32768,
> (ROE), 17e92b0>
> <read_diskdump: addr: ffffffff804b8210 paddr: 4b8210 cnt: 3568>
> read_diskdump: paddr/pfn: 4b8210/4b8 -> cache physical page: 4b8000
> <read_diskdump: addr: ffffffff804b9000 paddr: 4b9000 cnt: 4096>
> read_diskdump: paddr/pfn: 4b9000/4b9 -> cache physical page: 4b9000
> <read_diskdump: addr: ffffffff804ba000 paddr: 4ba000 cnt: 4096>
> read_diskdump: paddr/pfn: 4ba000/4ba -> cache physical page: 4ba000
> <read_diskdump: addr: ffffffff804bb000 paddr: 4bb000 cnt: 4096>
> read_diskdump: paddr/pfn: 4bb000/4bb -> cache physical page: 4bb000
> <read_diskdump: addr: ffffffff804bc000 paddr: 4bc000 cnt: 4096>
> read_diskdump: paddr/pfn: 4bc000/4bc -> cache physical page: 4bc000
> <read_diskdump: addr: ffffffff804bd000 paddr: 4bd000 cnt: 4096>
> read_diskdump: paddr/pfn: 4bd000/4bd -> cache physical page: 4bd000
> <read_diskdump: addr: ffffffff804be000 paddr: 4be000 cnt: 4096>
> read_diskdump: paddr/pfn: 4be000/4be -> cache physical page: 4be000
> <read_diskdump: addr: ffffffff804bf000 paddr: 4bf000 cnt: 4096>
> read_diskdump: paddr/pfn: 4bf000/4bf -> cache physical page: 4bf000
> <read_diskdump: addr: ffffffff804c0000 paddr: 4c0000 cnt: 528>
> read_diskdump: paddr/pfn: 4c0000/4c0 -> cache physical page: 4c0000
> WARNING: could not find MAGIC_START!
> GETBUF(248 -> 0)
> FREEBUF(0)
> GETBUF(64 -> 0)
> <readmem: ffffffff80a59910, KVADDR, "possible", 64, (ROE), da00e0>
> <read_diskdump: addr: ffffffff80a59910 paddr: a59910 cnt: 64>
> read_diskdump: paddr/pfn: a59910/a59 -> cache physical page: a59000
> cpu_possible_map:
> <readmem: ffffffff8095d950, KVADDR, "present", 64, (ROE), da00e0>
> <read_diskdump: addr: ffffffff8095d950 paddr: 95d950 cnt: 64>
> read_diskdump: paddr/pfn: 95d950/95d -> cache physical page: 95d000
> cpu_present_map: 2 3 6 7 10 11 14 15 18 19 22 23 26 27 30 31 34 35 38
> 39 42 43 46 47 50 51 54 55 58 59 62 63 66 67 70 71 74 75 78 79 82 83
> 86 87 90 91 94 95 98 99 102 103 106 107 110 111 114 115 118 119 122
> 123 126 127 130 131 134 135 138 139 142 143 146 147 150 151 154 155
> 158 159 162 163 166 167 170 171 174 175 178 179 182 183 186 187 190
> 191 194 195 198 199 202 203 206 207 210 211 214 215 218 219 222 223
> 226 227 230 231 234 235 238 239 242 243 246 247 250 251 254 255 258
> 259 262 263 266 267 270 271 274 275 278 279 282 283 286 287 290 291
> 294 295 298 299 302 303 306 307 310 311 314 315 318 319 322 323 326
> 327 330 331 334 335 338 339 342 343 346 347 350 351 354 355 358 359
> 362 363 366 367 370 371 374 375 378 379 382 383 386 387 390 391 394
> 395 398 399 402 403 406 407 410 411 414 415 418 419 422 423 426 427
> 430 431 434 435 438 439 442 443 446 447 450 451 454 455 458 459 462
> 463 466 467 470 471 474 475 478 479 482 483 486 487 490 491 494 495
> 498 499 502 503 506 507 510 511
> <readmem: ffffffff808cbe90, KVADDR, "online", 64, (ROE), da00e0>
> <read_diskdump: addr: ffffffff808cbe90 paddr: 8cbe90 cnt: 64>
> read_diskdump: paddr/pfn: 8cbe90/8cb -> cache physical page: 8cb000
> cpu_online_map:
> FREEBUF(0)
> <readmem: ffffffff80abaf00, KVADDR, "xtime", 16, (FOE), c5e430>
> <read_diskdump: addr: ffffffff80abaf00 paddr: abaf00 cnt: 16>
> read_diskdump: paddr/pfn: abaf00/aba -> cache physical page: aba000
> <readmem: ffffffff806dc894, KVADDR, "init_uts_ns", 390, (ROE),
> c5ea1c>
> <read_diskdump: addr: ffffffff806dc894 paddr: 6dc894 cnt: 390>
> read_diskdump: paddr/pfn: 6dc894/6dc -> cache physical page: 6dc000
> base kernel version: 0.0.0
> <readmem: ffffffff804a7000, KVADDR, "accessible check", 8, (ROE|Q),
> 7fff35eeba88>
> <read_diskdump: addr: ffffffff804a7000 paddr: 4a7000 cnt: 8>
> read_diskdump: paddr/pfn: 4a7000/4a7 -> cache physical page: 4a7000
> <readmem: ffffffff804a7000, KVADDR, "readstring characters", 1499,
> (ROE|Q), 7fff35eeaa70>
> <read_diskdump: addr: ffffffff804a7000 paddr: 4a7000 cnt: 1499>
> read_diskdump: paddr/pfn: 4a7000/4a7 -> physical page is cached:
> 4a7000
> linux_banner:
> 紉;���
> crash: vmlinux-2.6.27.19-5-default.debug and vmcore do not match!
>
> Usage:
>
> crash [OPTION]... NAMELIST MEMORY-IMAGE (dumpfile form)
> crash [OPTION]... [NAMELIST] (live system form)
>
> Enter "crash -h" for details.
>
>
>
>
>
>
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
> --
> Crash-utility mailing list
> Crash-utility at redhat.com
> https://www.redhat.com/mailman/listinfo/crash-utility
>
More information about the Crash-utility
mailing list